Closure XML Parser

An XML parser written in Common Lisp.

Closure XML was written by Gilbert Baumann (unk6 at rz.uni-karlsruhe.de) as part of the Closure web browser.
Contributions to the parser by

• Henrik Motakef (hmot at henrik-motakef.de)
  (SAX layer; namespace support)
• David Lichteblau at knowledgeTools (david at knowledgetools.de)
  (conversion into an independent package; DOM bug fixing; validation)

Mailing list cxml-devel is hosted on common-lisp.net.

Download

Get a tarball.

David's tla archive is at http://www.common-lisp.net/project/cxml/david@knowledgetools.de--cxml/. (Brief tla usage instructions: Unpack the cxml tarball.  Enter tla register-archive URL to turn it into a working copy.  tla update is similar to cvs up.)

Contents

• Recent changes
• CXML modules
• Installation
• Tests
• To do
• Using CXML
  • Parsing and validating
  • Serialization
  • Miscellaneous utility functions
  • XMLS compatibility
  • Dealing with rods
• SAX interface
• DOM notes

• RUNES, a portable implementation of Unicode strings.
• CXML, a namespace-aware validating SAX parser implementing the XML 1.0 specification.
• DOM, an implementation of the DOM Level 1 Core interfaces.

ASDF is used for compilation. The following instructions assume that ASDF has already been loaded.

Configuration (optional). CXML has full Unicode code support -- even on Lisps without Unicode strings. On non-unicode aware Lisps, DOMString is implemented as an array of character codes. If your Lisp supports 16 bit characters natively, you can enable feature RUNE-IS-CHARACTER to select an alternative DOMString implementatation, which uses real characters instead of characters codes.

* (pushnew :rune-is-character *features*)

Compiling and loading CXML. Register the .asd file, e.g. by symlinking it:

$ ln -sf `pwd`/cxms.asd /path/to/your/registry

Then compile CXML using:

* (asdf:operate 'asdf:load-op :cxml)

$ export CVSROOT=:pserver:anonymous@dev.w3.org:/sources/public
$ cvs login    # password is "anonymous"
$ cvs co 2001/XML-Test-Suite/xmlconf
$ cvs co 2001/DOM-Test-Suite

* (xmlconf:run-all-tests "/path/to/2001/XML-Test-Suite/xmlconf/")
0/556 tests failed; 1606 tests were skipped
* (domtest:run-all-tests "/path/to/2001/DOM-Test-Suite/")
0/450 tests failed; 71 tests were skipped

$ cd 2001/XML-Test-Suite/xmlconf/
xmltest$ patch -p0 -R </path/to/cxml/test/xmlconf-base.diff

• David's changes might have affected performance.  Some benchmarking needs to be done here. (The actual parser seems to be faster than xmls -- good enough for me.)
• DOM in general is pretty heavyweight.  There is/was a "simple-dom" which should be faster.  This might be worth reviving.
• For those who don't like DOM at all, it would be a very simple exercise to write a SAX handler for "Lisp-XML" output instead of DOM. (done)
• The serializer supports only Canonical XML right now.  In the future we want support for: Including doctype declarations in the output (done), ordinary output with less character reference noise (done), optional indentation (done), user-specified encoding, etc.
• There are still thread-safety issues. (fixed)
• Validation! (done)
• Upgrade to DOM Level 2 for complete namespace support.
• Unless rune-is-character is enabled, rod hashing currently uses equalp hash tables, which can be very slow. (See %make-rod-hash-table.)

(Compare also with Gilbert Baumann's older TODO list in xml-parse.lisp.)

• pathname -- a Common Lisp pathname
• stream -- a Common Lisp stream with element-type (unsigned-byte 8)

(cxml:parse-file "test.xml" (dom:make-dom-builder))

• t or 1: Canonical XML
• 2: Second Canonical Form
• NIL: Use a more readable non-canonical representation.

With an indentation level, pretty-print the XML by inserting additional whitespace.  Note that indentation changes the document model and should only be used if whitespace does not matter to the application.

unparse-document-to-octets returns an (unsigned-byte 8) array, whereas unparse-document writes characters.  unparse-document is useful together with with-output-to-string.  However, note that the resulting document in both cases is UTF-8 encoded, so the characters written by unparse-document are really UTF-8 bytes encoded as characters.

These function provide the low-level mechanism used by the DOM serialization functions. To serialize a document without building its DOM tree first, create a sink handle and call SAX functions on that handle. sax:end-document returns the serialized form of the document described by the SAX events.

Example:

(with-xml-output (make-octet-stream-sink stream :indentation 2 :canonical nil)
  (with-element "foo"
    (attribute "xyz" "abc")
    (with-element "bar"
      (attribute "blub" "bla"))
    (text "Hi there.")))

Prints this to stream, which must be an (unsigned-byte 8) stream:

<foo xyz="abc">
  <bar blub="bla"></bar>
  Hi there.
</foo>

(Note that these functions accept both strings and rods, so we could write "foo" instead of #"foo" above.)

(let ((d (parse-file "~/test.xml" (dom:make-dom-builder)))
      (x (parse-dtd-file "~/test.dtd")))
  (dom:map-document (cxml:make-validator x #"foo") d))

Like other XML parsers written in Lisp, CXML can work with documents represented as list structures. The specific model implemented by cxml is compatible with the xmls parser. Xmls list structures are a simpler and faster alternative to full DOM document trees. They also serve as an example showing how to implement user-defined document models as an independent layer over the the base parser (c.f. xmls/xmls-compat.lisp in the cxml distribution). However, note that the list structures do not include all information available in DOM documents and are sometimes more difficult to work wth since many DOM functions cannot be implemented on them.

(cxml:parse-file "test.xml" (cxml-xmls:make-xmls-builder))

Use this function to serialize XMLS data. For example, we could define a replacement for xmls:write-xml like this:

(defun write-xml (stream node &key indent)
  (let ((sink (cxml:make-character-stream-sink
               stream :canonical nil :indentation indent)))
    (cxml-xmls:map-node sink node)))

fixme: namespace support.

CL-USER(14): (cxml:parse-string "<test/>" (cxml-xmls:make-xmls-builder))
(#(116 101 115 116) NIL)

CL-USER(17): (parse-string "<test/>" (cxml:make-recoder (cxml-xmls:make-xmls-builder)))
("test" NIL)

CXML implements the DOM Level 1 Core interfaces.  Explaining DOM is better left to the specification, so please refer to the official W3C documents for DOM.

However, there is no "standard" DOM mapping for Lisp.  DOM is specified in CORBA IDL, but it refrains from using object-oriented IDL features, allowing for a much more natural Lisp implemenation than the the ordinary IDL/Lisp mapping would.

Differences between CXML's DOM and the direct IDL/Lisp mapping:

• DOM function names are symbols in the DOM package (not the OP package).
• DOM functions have proper required arguments, not a huge &rest lambda list.
• Although most IDL interfaces are implemented as CLOS classes by CXML, the Lisp types of DOM objects is not documented and cannot be relied upon.  A node's type can be determined using dom:node-type instead.
• DOMString is mapped to rod, which is either an (unsigned-byte 16) array type or a string type.
• The IDL/Lisp mapping maps CORBA enums to Lisp keywords.  Unfortunately, the DOM IDL does not use enums.  Instead, both exception types and node types are defined integer constants.  CXML chooses to ignore this definition and uses keywords instead.
• DOM uses StudlyCaps.  Lisp programmers don't.  We insert #\- before every upper case letter preceded by a lower case letter and before every upper case letter which is followed by a lower case letter, but preceded by a capital letter.  This algorithms leads to the natural Lisp spelling of DOM function names.
• Implementation note: DOM's NodeList does not necessarily map to a native "sequence" type.  (For example, node lists are objects in Java, not arrays.)  NodeList is specified to reflect changes done after a node list was created, so node lists cannot be Lisp lists.  (A node list could be implemented as a CLOS object pointing to said list though.)  Instead, CXML currently implements node lists as adjustable vectors.  Note that code which relies on this implementation and uses Lisp sequence functions instead of sticking to dom:item and dom:length is not portable.  As a compromise, you can use our extensions dom:map-node-list or dom:do-node-list, which can be implemented portably.

Example:

XML(97): (dom:node-type
          (dom:document-element
           (cxml:parse-file "~/test.xml" (dom:make-dom-builder))))
:ELEMENT