Closure XML Parser

An XML parser written in Common Lisp.

Closure XML was written by Gilbert Baumann (unk6 at rz.uni-karlsruhe.de) as part of the Closure web browser.
Contributions to the parser by

• Henrik Motakef (hmot at henrik-motakef.de)
  (SAX layer; namespace support)
• David Lichteblau for knowledgeTools (conversion into an independent package; DOM bug fixing; validation) and headcraft (most of the october 2004 release).

Send bug reports to david@lichteblau.com.

Download

Get a tarball.

David's tla archive is at http://www.common-lisp.net/project/cxml/david@knowledgetools.de--cxml/. (Brief tla usage instructions: Unpack the cxml tarball.  Enter tla register-archive URL to turn it into a working copy.  tla update is similar to cvs up.) Note that this used to be www.common-lisp.net and is now just common-lisp.net.

Contents

• Recent Changes
• CXML Modules
• Installation
• Tests
• To Do
• Using CXML
  • Quick-Start Example
  • Parsing and Validating
  • Serialization
  • Miscellaneous Utility Functions
  • XMLS Compatibility
  • Dealing with Rods
  • Caching of DTD Objects
  • XML Catalogs
• SAX Interface
• DOM Notes

• RUNES, a portable implementation of Unicode strings.
• CXML, a namespace-aware validating SAX parser implementing the XML 1.0 specification.
• DOM, an implementation of the DOM Level 1 Core interfaces.

ASDF is used for compilation. The following instructions assume that ASDF has already been loaded.

Prerequisites. CXML needs the puri library.

Compiling and loading CXML. Register the .asd file, e.g. by symlinking it:

$ ln -sf `pwd`/cxml.asd /path/to/your/registry/

Then compile CXML using:

* (asdf:operate 'asdf:load-op :cxml)

You can then try the quick-start example.

$ export CVSROOT=:pserver:anonymous@dev.w3.org:/sources/public
$ cvs login    # password is "anonymous"
$ cvs co 2001/XML-Test-Suite/xmlconf
$ cvs co 2001/DOM-Test-Suite

* (xmlconf:run-all-tests "/path/to/2001/XML-Test-Suite/xmlconf/")
0/556 tests failed; 1606 tests were skipped
* (domtest:run-all-tests "/path/to/2001/DOM-Test-Suite/")
0/450 tests failed; 71 tests were skipped

$ cd 2001/XML-Test-Suite/xmlconf/
xmltest$ patch -p0 -R </path/to/cxml/test/xmlconf-base.diff

• David's changes might have affected performance.  Some benchmarking needs to be done here. (The actual parser seems to be faster than xmls -- good enough for me.)
• DOM in general is pretty heavyweight.  There is/was a "simple-dom" which should be faster.  This might be worth reviving.
• For those who don't like DOM at all, it would be a very simple exercise to write a SAX handler for "Lisp-XML" output instead of DOM. (done)
• The serializer supports only Canonical XML right now.  In the future we want support for: Including doctype declarations in the output (done), ordinary output with less character reference noise (done), optional indentation (done), user-specified encoding, etc.
• There are still thread-safety issues. (fixed)
• Validation! (done)
• Upgrade to DOM Level 2 for complete namespace support.
• Unless rune-is-character is enabled, rod hashing currently uses equalp hash tables, which can be very slow. (See %make-rod-hash-table.)

(Compare also with Gilbert Baumann's older TODO list in xml-parse.lisp.)

Make sure to install and load cxml first.

Create a test file called example.xml:

* (with-open-file (s "example.xml" :direction :output)
    (write-string "<test a='b'><child/></test>" s))

Parse example.xml into a DOM tree (read more):

* (cxml:parse-file "example.xml" (dom:make-dom-builder))
#<DOM-IMPL::DOCUMENT @ #x72206172>
;; save result for later:
* (defparameter *example* *)
*EXAMPLE*

Inspect the DOM tree (read more):

* (dom:document-element *example*)
#<DOM-IMPL::ELEMENT test @ #x722b6ba2>
* (dom:tag-name (dom:document-element *example*))
"test"
* (dom:child-nodes (dom:document-element *example*))
#(#<DOM-IMPL::ELEMENT child @ #x722b6d8a>)
* (dom:get-attribute (dom:document-element *example*) "a")
"b"

Serialize the DOM document back into a stream (read more):

(cxml:unparse-document *example* *standard-output*)
<test a="b"><child></child></test>

As an alternative to DOM, parse into xmls-compatible list structure (read more):

* (cxml:parse-file "example.xml" (cxml-xmls:make-xmls-builder))
("test" (("a" "b")) ("child" NIL))

• pathname -- a Common Lisp pathname
• stream -- a Common Lisp stream with element-type (unsigned-byte 8)

(cxml:parse-file "test.xml" (dom:make-dom-builder))

• t or 1: Canonical XML
• 2: Second Canonical Form
• NIL: Use a more readable non-canonical representation.

With an indentation level, pretty-print the XML by inserting additional whitespace.  Note that indentation changes the document model and should only be used if whitespace does not matter to the application.

unparse-document-to-octets returns an (unsigned-byte 8) array, whereas unparse-document writes characters.  unparse-document is useful together with with-output-to-string.  However, note that the resulting document in both cases is UTF-8 encoded, so the characters written by unparse-document are really UTF-8 bytes encoded as characters.

These function provide the low-level mechanism used by the DOM serialization functions. To serialize a document without building its DOM tree first, create a sink handle and call SAX functions on that handle. sax:end-document returns the serialized form of the document described by the SAX events.

Example:

(with-xml-output (make-octet-stream-sink stream :indentation 2 :canonical nil)
  (with-element "foo"
    (attribute "xyz" "abc")
    (with-element "bar"
      (attribute "blub" "bla"))
    (text "Hi there.")))

Prints this to stream, which must be an (unsigned-byte 8) stream:

<foo xyz="abc">
  <bar blub="bla"></bar>
  Hi there.
</foo>

(Note that these functions accept both strings and rods, so we could write "foo" instead of #"foo" above.)

xhtmlgen is included as contrib/xhtmlgen.lisp in the cxml distribution. Example:

(let ((sink (cxml:make-character-stream-sink *standard-output*)))
  (sax:start-document sink)
  (xhtml-generator:write-doctype sink)
  (xhtml-generator:with-html sink
    (:html
     (:head
      (:title "Titel"))
     (:body
      ((:p "style" "font-weight: bold")
       "Inhalt")
      (:ul
       (:li "Eins")
       (:li "Zwei")
       (:li "Drei")))))
  (sax:end-document sink))

(let ((d (parse-file "~/test.xml" (dom:make-dom-builder)))
      (x (parse-dtd-file "~/test.dtd")))
  (dom:map-document (cxml:make-validator x #"foo") d))

Like other XML parsers written in Lisp, CXML can work with documents represented as list structures. The specific model implemented by cxml is compatible with the xmls parser. Xmls list structures are a simpler and faster alternative to full DOM document trees. They also serve as an example showing how to implement user-defined document models as an independent layer over the the base parser (c.f. xml/xmls-compat.lisp in the cxml distribution). However, note that the list structures do not include all information available in DOM documents and are sometimes more difficult to work wth since many DOM functions cannot be implemented on them.

(cxml:parse-file "test.xml" (cxml-xmls:make-xmls-builder))

Use this function to serialize XMLS data. For example, we could define a replacement for xmls:write-xml like this:

(defun write-xml (stream node &key indent)
  (let ((sink (cxml:make-character-stream-sink
               stream :canonical nil :indentation indent)))
    (cxml-xmls:map-node sink node)))

The node list's car can also be a cons of local name and namespace prefix ns. fixme: It is unclear to me how namespaces are meant to work in xmls, since xmls documentation differs from how xmls actually works in current releases. Usually applications need to know both the namespace prefix and the namespace URI. We currently follow the xmls implementation and use the namespace prefix instead of following its documentation which shows the URI. We do not follow xmls in munging xmlns attribute values. Attributes themselves have namespaces and it is not clear to me how that works in xmls.

CL-USER(14): (cxml:parse-string "<test/>" (cxml-xmls:make-xmls-builder))
(#(116 101 115 116) NIL)

CL-USER(17): (parse-string "<test/>" (cxml:make-recoder (cxml-xmls:make-xmls-builder)))
("test" NIL)

• User-defined entity resolvers can be used to open entities using arbitrary protocols. For example, an entity resolver could handle all System-IDs with the http scheme using some HTTP library. Refer to the description of the entity-resolver keyword argument to parser functions (see cxml:parse-file) to more information on entity resolvers.
• XML Catalogs are (local) tables in XML syntax which map External IDs to alternative System IDs. If, say, the xhtml DTD is present in the local file system and the local copy has been registered with the XML catalog, CXML will use the local copy of the DTD instead of trying to open the version available using HTTP.

This section describes XML Catalogs, the second solution. CXML implements Oasis XML Catalogs.

Example:

* (setf cxml:*catalog* nil)
* (cxml:parse-file "test.xhtml" nil)
=> Error: URI scheme :HTTP not supported

* (setf cxml:*catalog* (cxml:make-catalog))
* (cxml:parse-file "test.xhtml" nil)
;; no error!
NIL

Note that parsed catalog files are cached in the catalog object. Catalog files cached do not expire automatically. To ensure that all catalog files are parsed again, create a new catalog object.

CXML implements the DOM Level 1 Core interfaces.  Explaining DOM is better left to the specification, so please refer to the official W3C documents for DOM.

However, there is no "standard" DOM mapping for Lisp.  DOM is specified in CORBA IDL, but it refrains from using object-oriented IDL features, allowing for a much more natural Lisp implemenation than the the ordinary IDL/Lisp mapping would.

Differences between CXML's DOM and the direct IDL/Lisp mapping:

• DOM function names are symbols in the DOM package (not the OP package).
• DOM functions have proper required arguments, not a huge &rest lambda list.
• Although most IDL interfaces are implemented as CLOS classes by CXML, the Lisp types of DOM objects is not documented and cannot be relied upon.  A node's type can be determined using dom:node-type instead.
• DOMString is mapped to rod, which is either an (unsigned-byte 16) array type or a string type.
• The IDL/Lisp mapping maps CORBA enums to Lisp keywords.  Unfortunately, the DOM IDL does not use enums.  Instead, both exception types and node types are defined integer constants.  CXML chooses to ignore this definition and uses keywords instead.
• DOM uses StudlyCaps.  Lisp programmers don't.  We insert #\- before every upper case letter preceded by a lower case letter and before every upper case letter which is followed by a lower case letter, but preceded by a capital letter.  This algorithms leads to the natural Lisp spelling of DOM function names.
• Implementation note: DOM's NodeList does not necessarily map to a native "sequence" type.  (For example, node lists are objects in Java, not arrays.)  NodeList is specified to reflect changes done after a node list was created, so node lists cannot be Lisp lists.  (A node list could be implemented as a CLOS object pointing to said list though.)  Instead, CXML currently implements node lists as adjustable vectors.  Note that code which relies on this implementation and uses Lisp sequence functions instead of sticking to dom:item and dom:length is not portable.  As a compromise, you can use our extensions dom:map-node-list or dom:do-node-list, which can be implemented portably.

Example:

XML(97): (dom:node-type
          (dom:document-element
           (cxml:parse-file "~/test.xml" (dom:make-dom-builder))))
:ELEMENT