There's a variety of other serializations for RDF. The most popular seems to be Turtle and RDF/JSON (from what I can see, there's lots of regret about the XML serialization, though Omeka S ships with the XML versions). JSON-LD is also used for expressing RDF data, but I'm not sure how much it is used for defining vocabularies/ontologies (in theory it's possible, just not sure if it's done in practice.
Omeka S uses the EasyRDF library to do our parsing and importing, so anything it can read, we should be able to import. Thus, for a user-defined vocabulary (or ontology, most of the time the technical distinctions don't matter), you just need to get it serialized in something EasyRDF can read. One popular tool for writing vocabularies is Protege. It can be a bit daunting at first, but once you ignore the features you don't need it's a good tool for building and exporting vocabularies.
You would just build the user-defined vocab however you want, then Omeka S can import it.