Adding a new file parser

Xj3D is a very flexible toolkit. There are many things that you can do with the code. One question that we get asked quite a lot is - how do I add a different parser to the system? For example, you wish to try a different file format (eg MPEG4 BIFs or VRML1.0). This is a simple outline on what you would need to do.

Intro

Xj3D runs a very componentised architecture. Each piece of the process is placed in its separate box. Therefore, we put the parser in one box, the scene builder in another box, rendering code in another etc. For file parsing, we use JavaCC from WebGain.

In the current architecture of the code, we run two separate parsing systems in our custom parser for the UTF8 (VRML Classic) syntax and an XML parser for that encoding. Although our implementation falls somewhat below the goals set out in this document (our XML generates a DOM and then feeds the DOM to a converter), this is the way that any extension should be applied to the system.

What Xj3D does now

For the UTF8 (VRML Classic) file format, we use JavaCC. JavaCC does our lexical parsing based on a BNF grammar description from the specification. Most specifications will document some form of grammar for you. Note that at this stage it is pure lexical parsing, we don't check the validity of adding one node as a child of another.

Feeding the output of the Grammar to something that does the scene building is the job of an API we call SAV - Simple API for VRML processing. It's very much like SAX is for DOM. As you are parsing the file and getting the details back from JavaCC you fire these SAV events off to the "listeners". It is quite hard to describe in a concise way, but have a look at the VRML97 version of the JavaCC grammar that we use ($PROJECT_ROOT/src/javacc/vrml97). These events are fed to a whatever listener is registered and that is as far as the parsing code goes. Any structural issues, such as protos, checking nodes against the declared profile or whatever is handled elsewhere in the codebase.

Implementing your own parser

Before you start, there are a number of prerequisites that will need to be filled. Most importantly - do you have a definition of the file format? Strange as this may sound, many heavily used file formats do not have any documentation.

Writing the parser

The next step is deciding how you are going to parse the file. If the file format is trivial, then you should be able to custom code a parser fairly trivially using classes like StreamTokenizer. For more complex grammars, we suggest you use JavaCC as it can handle some very complex structures quite easily.

Once you have decided on your parsing technology, start implementing the grammar. Write a simple parser that just reads in the values, and prints them out on the standard output. Once that works you will need to look into the issues of matching your file format with the VRML "system". The internals of the Xj3D toolkit are built to match the model that VRML defines for a scene graph. For many file formats, you will have quite some differences between the VRML model and your model. Those issues you will need to work out before you start to implement the next step.

Information from the parser is passed to the rest of the Xj3D toolkit through the SAV interfaces. These interfaces are designed to express VRML concepts, not generalised 3D graphics concepts. It is most likely that your file format will not generate all of the possible SAV events. In other cases, your file format may express information that is entirely foreign to the VRML model so your parser will have to do some sort of translation. Remember though, that the SAV interface is a serialised API - the code at the other end is expecting events to be a serial stream of information that has a relatively strict pattern that happens to conform to a VRML file. It will not be able to handle random events that are not logical in terms of a VRML file. If you do sent events that don't fit the model, expect to have the rest of the code issue errors. While you are fine to buffer up information as you parse it and then issue the events once you have determined the correct translation, make sure that you issue them in the correct order.

Dealing with Errors

As with any production application, you must be able to deal with bad input as well as good input. Simply crashing on bad input is not sufficient. There are many levels of error handling within Xj3D and your parser code is going to have to deal with this. Remember that one of the primary purposes of Xj3D is to verify the new specification work and as such it is much less tolerant of badly formated/structured information. We do a lot of internal consistency checks and throw more (deliberate) exceptions than you would expect from a normal commercial browser implementation. Expect to have to deal with errors coming from the internal implementation.

All of the SAV methods throw one or more exceptions. Because it is your code making calls to an implementor of the SAV interfaces, you are going to have to deal with the exceptions they generate. All methods in the SAV interfaces throw exceptions that are based on either VRMLException or SAVException. The former is used to indicate VRML structural problems such as a field not being valid for that particular node, or that the value is out of range. The latter is used for syntax issues such as a node declaration being generated at the wrong time or a specific feature asked for by the parser is not available.

How you treat the individual exceptions is up to you. However, you should note that SAV provides an interface for you to pass on error information, through the ErrorHandler interface. You can pass exceptions here and that handler will display them or whatever. If the handler thinks that the issue is bad it enough, it may issue a further exception which you are then free to allow it to stop the parsing process.

Using the Parser with other code facilities

When you are happy with the parser, your next step will be to use it in an integrated code solution. Within the structure of the Xj3D toolkit, there is no formal structure for adding new capabilities like this. We assume that you are going to assemble the parts that you need together. So, if you wanted to have a Java3D loader for your new file format, you can't just tweak a few config parameters and expect that loader to automatically pick up your new parser implementation. To do this, you will need to write your own handling code that collects all of the pieces together to form a complete rendering solution for your application.

There are many different ways that your parser can be integrated into the system. Rather than telling you about all of them, have a look at the following classes for examples on how to use the parser in these different ways:

DIYbrowser - Examples area. Shows how to construct a complete browser application using the toolkit.
org.web3d.j3d.loaders.VRML97Loader. How to construct a Java3D loader to load your file format.
org.web3d.j3d.browser.BrowserJPanel A panel that implements a VRML browser functionality. Also happens to be the core of the EAI code for createComponent().