SAX parsing and serialization
This chapter describes CXML's SAX-like parser interface.
The SAX layer is an important concept in CXML that users will
encounter in various situations:
To parse into DOM, use the SAX parser as described below with
a DOM builder as the SAX handler. (Refer to make-dom-builder for information about
Serialization is done using SAX, too. SAX handlers that
process and consume events without sending them to another
handler are called sinks in CXML. Serialization sinks
write XML output for the events they receive. For example, to
serialize DOM, use map-document to turn the DOM
document into SAX events together with a sink for
SAX handlers can be chained together. Various SAX handlers
are offered that can be used in this way, transforming SAX
events before handing them to the next handler. This includes
handlers for whitespace removal, namespace
normalization, and rod-to-string recoding.
However, SAX events are easier to generate than to process. That
is why CXML offers Klacks, a "pull-based" API in addition to SAX.
Klacks events are generally easier to process than to generate.
Please refer to the Klacks documentation
Parsing and Validating
Old-style convenience functions:
Function CXML:PARSE-FILE (pathname handler &key ...)
Same as cxml:parse with a pathname argument.
(But note that cxml:parse-file interprets string
arguments as namestrings, while cxml:parse expects
literal XML documents.)
Function CXML:PARSE-STREAM (stream handler &key ...)
Same as cxml:parse with a stream argument.
Function CXML:PARSE-OCTETS (octets handler &key ...)
Same as cxml:parse with an octet vector argument.
Function CXML:PARSE-ROD (rod handler &key ...)
Same as cxml:parse with a string argument.
New all-in-one parser interface:
Function CXML:PARSE (input handler &key ...)
Parse an XML document, where input is a string, pathname, octet
vector, or stream.
Return values from this function depend on the SAX handler used.
input -- one of:
pathname -- a Common Lisp pathname.
Open the file specified by the pathname and parse its
contents as an XML document.
stream -- a Common Lisp stream with element-type
octets -- an (unsigned-byte 8) array.
The array is parsed directly, and interpreted according to the
encoding it specifies.
string/rod -- a rod (or string on
Parses an XML document from the input string that has already
undergone external-format decoding.
stream -- a Common Lisp stream with element-type
octets -- an (unsigned-byte 8) array
handler -- a SAX handler
Common keyword arguments:
validate -- A boolean. Defaults to
nil. If true, parse in validating mode, i.e. assert that
the document contains a DOCTYPE declaration and conforms to the
dtd -- unless nil, an extid instance
specifying the external subset to load. This options overrides
the extid specified in the document type declaration, if any.
See below for make-extid. This option is useful
for verification purposes together with the root
and disallow-internal-subset arguments.
root -- the expected root element
name, or nil (the default).
entity-resolver -- nil or a function of two
arguments which is invoked for every entity referenced by the
document with the entity's Public ID (a rod) and System ID (an
URI object) as arguments. The function may either return
nil, CXML will then try to resolve the entity as usual.
Alternatively it may return a Common Lisp stream specialized on
(unsigned-byte 8) which will be used instead. (It may
also signal an error, of course, which can be useful to prohibit
parsed XML documents from including arbitrary files readable by
disallow-internal-subset -- a boolean. If true, signal
an error if the document contains an internal subset.
recode -- a boolean. (Ignored on Lisps with Unicode
support.) Recode rods to UTF-8 strings. Defaults to true.
Make sure to use utf8-dom:make-dom-builder if this
option is enabled and rune-dom:make-dom-builder
Note: parse-rod assumes that the input has already been
decoded into Unicode runes and ignores the encoding
specified in the XML declaration, if any.
Function CXML:PARSE-EMPTY-DOCUMENT (uri qname handler &key public-id system-id entity-resolver recode)
Simulate parsing a document with a document element qname
having no attributes except for an optional namespace
declaration to uri. If an external ID is specified
(system-id, public-id), find, parse, and report
this DTD as if with parse-file, using the specified
Function CXML:PARSE-DTD-FILE (pathname)
Function CXML:PARSE-DTD-STREAM (stream)
from a stand-alone file and return an object representing the DTD,
suitable as an argument to validate.
pathname -- a Common Lisp pathname
stream -- a Common Lisp stream with element-type
Function CXML:MAKE-EXTID (publicid systemid)
Create an object representing the External ID composed
of the specified Public ID, a rod or nil, and System ID
(an URI object).
Condition class CXML:XML-PARSE-ERROR ()
Superclass of all conditions signalled by the CXML parser.
Condition class CXML:WELL-FORMEDNESS-VIOLATION (cxml:xml-parse-error)
This condition is signalled for all well-formedness violations.
(Note that, when parsing document that is not well-formed in validating
mode, the parser might encounter validity errors before detecting
well-formedness problems, so also be prepared for validity-error
in that situation.)
Condition class CXML:VALIDITY-ERROR (cxml:xml-parse-error)
Reports the violation of a validity constraint.
Serialization is performed using sink objects. There are
different kinds of sinks for output to lisp streams and vectors in
Technically, sinks are SAX handlers that write XML output for SAX
events sent to them. In practise, user code would normally not
generate those SAX events manually, and instead use a function
like dom:map-document or xmls-compat:map-node to serialize an
In addition to map-document, cxml has a set of
convenience macros for serialization (see below for
with-xml-output, with-element, etc).
Function CXML:MAKE-OCTET-VECTOR-SINK (&rest keys) => sink
Function CXML:MAKE-OCTET-STREAM-SINK (stream &rest keys) => sink
Function CXML:MAKE-ROD-SINK (&rest keys) => sink
Only on Lisps with Unicode support:
Function CXML:MAKE-STRING-SINK -- alias for cxml:make-rod-sink
Function CXML:MAKE-CHARACTER-STREAM-SINK (stream &rest keys) => sink
Only on Lisps without Unicode support:
Function CXML:MAKE-STRING-SINK/UTF8 (&rest keys) => sink
Function CXML:MAKE-CHARACTER-STREAM-SINK/UTF8 (stream &rest keys) => sink
Return a SAX serialization handle.
The -octet- functions write the document encoded into
make-octet-stream-sink works with Lisp streams of
element-type (unsigned-byte 8).
make-octet-vector-sink returns a vector of
make-character-stream-sink works with character
streams. It serializes the document into characters without
encoding it into an external format. When using these
functions, take care to avoid encoding the result into
an incorrect external format. (Note that characters undergo
external format conversion when written to a character stream.
If the document's XML declaration specifies an encoding, make
sure to specify this encoding as the external format if and when
writing the serialized document to a character stream. If the
document does not specify an encoding, either UTF-8 or UTF-16
must be used.) This function is available only on Lisps with
make-rod-sink serializes the document into a vector of
runes without encoding it into an external format.
(On Lisp with unicode support, the result will be a string;
otherwise, a vector of character codes will be returned.)
The warnings given for make-character-stream-sink
apply to this function as well.
The /utf8 functions write the document encoded into
characters representing a UTF-8 encoding.
When using these functions, take care to avoid encoding the
result into an external format for a second time. (Note
that characters undergo external format conversion when written
to a character stream. Since these functions already perform
external format conversion, make sure to specify an external
format that does "nothing" if and when writing the serialized document
to a character stream. ISO-8859-1 external formats usually
achieve the desired effect.)
make-character-stream-sink/utf8 works with character streams.
make-string-sink/utf8 returns a string.
These functions are available only on Lisps without unicode support.
canonical -- canonical form, one of NIL, T, 1, 2
indentation -- indentation level. An integer or nil.
encoding -- the character encoding to use. A string or
keyword. nil is also allowed and means UTF-8.
omit-xml-declaration-p -- Boolean. Don't write an XML
The following canonical values are allowed:
An internal subset will be included in the result regardless of
the canonical setting. It is the responsibility of the
caller to not report an internal subset for
canonical <= 1, or only notations as required for
canonical = 2. For example, the
include-doctype argument to dom:map-document
should be set to nil for the former behaviour and
:canonical-notations for the latter.
With an indentation level, pretty-print the XML by
inserting additional whitespace. Note that indentation
changes the document model and should only be used if whitespace
does not matter to the application.
Macro CXML:WITH-XML-OUTPUT (sink &body body) => sink-specific result
Macro CXML:WITH-NAMESPACE ((prefix uri) &body body) => result
Macro CXML:WITH-ELEMENT (qname &body body) => result
Macro CXML:WITH-ELEMENT* ((prefix lname) &body body) => result
Function CXML:ATTRIBUTE (qname value) => value
Generic Function CXML:UNPARSE-ATTRIBUTE (value) => string
Function CXML:ATTRIBUTE* (prefix lname value) => value
Function CXML:TEXT (data) => data
Function CXML:CDATA (data) => data
Function CXML:doctype (name public-id system-id &optional internal-subset)
Convenience syntax for event-based serialization.
(with-xml-output (make-octet-stream-sink stream :indentation 2 :canonical nil)
(attribute "xyz" "abc")
(attribute "blub" "bla"))
(text "Hi there.")))
Prints this to stream:
Macro XHTML-GENERATOR:WITH-XHTML (sink &rest forms)
Macro XHTML-GENERATOR:WRITE-DOCTYPE (sink)
Macro with-xhtml is a modified version of
Franz' htmlgen works as a SAX driver for XHTML.
It aims to be a plug-in replacement for the html macro.
xhtmlgen is included as contrib/xhtmlgen.lisp in
the cxml distribution. Example:
(let ((sink (cxml:make-character-stream-sink *standard-output*)))
((:p "style" "font-weight: bold")
Miscellaneous SAX handlers
Function CXML:MAKE-VALIDATOR (dtd root)
Create a SAX handler which validates against a DTD instance.
The document's root element must be named root.
Used with dom:map-document, this validates a document
object as if by re-reading it with a validating parser, except
that declarations recorded in the document instance are completely
(let ((d (parse-file "~/test.xml" (cxml-dom:make-dom-builder)))
(x (parse-dtd-file "~/test.dtd")))
(dom:map-document (cxml:make-validator x #"foo") d))
Class CXML:BROADCAST-HANDLER ()
Function CXML:MAKE-BROADCAST-HANDLER (&rest handlers)
broadcast-handler is a SAX handler which passes every event it
receives on to each of several chained handlers, somewhat similar
to the way a broadcast-stream works.
You can subclass broadcast-stream to modify the events
before they are being passed on. Define methods on your handler
class for the events to be modified. All other events will pass
through to the chained handlers unmodified.
Broadcast handler functions return the result of calling the event
function on the last handler in the list. In particular,
the overall result from sax:end-document will be ignored
for all other handlers.
Class CXML:SAX-PROXY (broadcast-handler)
sax-proxy is a subclass of broadcast-handler
which sends events to exactly one chained handler. This class is
still included for compatibility with older versions of
CXML which did not include the more
general broadcast-handler yet, but has been retrofitted
as a subclass of the latter.
Accessor CXML:MAKE-NAMESPACE-NORMALIZER (next-handler)
Return a SAX handler that performs DOM
3-style namespace normalization on attribute lists in
start-element events before passing them on the next
Function CXML:MAKE-WHITESPACE-NORMALIZER (chained-handler &optional dtd)
Return a SAX handler which removes whitespace from elements that
have element content and have not been declared to
preserve space using an xml:space attribute.
<!DOCTYPE test [
<!ELEMENT test (foo,bar*)>
<!ATTLIST test a CDATA #IMPLIED>
<!ELEMENT foo #PCDATA>
<!ELEMENT bar (foo?)>
<!ATTLIST bar xml:space (default|preserve) "default">
<bar xml:space="preserve"> </bar>
<test a="b"><foo> </foo><bar></bar><bar xml:space="preserve"> </bar></test>
Recoders are a mechanism used by CXML internally on Lisp implementations
without Unicode support to recode UTF-16 vectors (rods) of
integers (runes) into UTF-8 strings.
User code does not usually need to deal with recoders in current
versions of CXML.
Function CXML:MAKE-RECODER (chained-handler recoder-fn)
Return a SAX handler which passes all events on to
chained-handler after converting all strings and rods
using recoder-fn, a function of one argument.
Caching of DTD Objects
To avoid spending time parsing the same DTD over and over again,
CXML can cache DTD objects. The parser consults
cxml:*dtd-cache* whenever it is looking for an external
subset in a document which does not have an internal subset and
uses the cached DTD instance if one is present in the cache for
the System ID in question.
Note that DTDs do not expire from the cache automatically.
(Future versions of CXML might introduce automatic checks for
The DTD cache object consulted by the parser when it needs a DTD.
Function CXML:MAKE-DTD-CACHE ()
Return a new, empty DTD cache object.
If true, instructs the parser to enter all DTDs that could have
been cached into *dtd-cache* if they were not cached
already. Defaults to nil.
Reader CXML:GETDTD (uri dtd-cache)
Return a cached instance of the DTD at uri, if present in
the cache, or nil.
Writer CXML:GETDTD (uri dtd-cache)
Enter a new value for uri into dtd-cache.
Function CXML:REMDTD (uri dtd-cache)
Ensure that no DTD is recorded for uri in the cache and
return true if such a DTD was present.
Function CXML:CLEAR-DTD-CACHE (dtd-cache)
Remove all entries from dtd-cache.
Class SAX:SAX-PARSER ()
A class providing location information through an
implementation-specific subclass. Parsers will use
sax:register-sax-parser to pass their parser instance to
the handler. The easiest way to receive sax parsers instances is
to inherit from sax-parser-mixin when defining a sax handler.
Class SAX:SAX-PARSER-MIXIN ()
A mixin for sax handler classes that records the sax handler
object for use with the following functions. Trampoline methods
are provided that allow those functions to be called directly on
Function SAX:SAX-HANDLER (sax-handler-mixin) => sax-handler
Return the sax-parser instance recorded by this handler, or NIL.
Function SAX:LINE-NUMBER (sax-parser)
Return an approximation of the current line number, or NIL.
Function SAX:COLUMN-NUMBER (sax-parser)
Return an approximation of the current column number, or NIL.
Function SAX:SYSTEM-ID (sax-parser)
Return the URI of the document being parsed. This is either the
main document, or the entity's system ID while contents of a parsed
general external entity are being processed.
Function SAX:XML-BASE (sax-parser)
Return the [Base URI] of the current element. This URI can differ from
the value returned by sax:system-id if xml:base
attributes are present.
External entities (for example, DTDs) are referred to using their
Public and System IDs. Usually the System ID, a URI, is used to
locate the entity. CXML itself handles only file://-URIs, but
many System IDs in practical use are http://-URIs. There are two
different mechanims applications can use to allow CXML to locate
entities using arbitrary Public ID or System ID:
User-defined entity resolvers can be used to open entities using
arbitrary protocols. For example, an entity resolver could
handle all System-IDs with the http scheme using some
HTTP library. Refer to the description of the
entity-resolver keyword argument to parser functions (see cxml:parse-file) to more
information on entity resolvers.
XML Catalogs are (local) tables in XML syntax which map External
IDs to alternative System IDs. If, say, the xhtml DTD is
present in the local file system and the local copy has been
registered with the XML catalog, CXML will use the local copy of
the DTD instead of trying to open the version available using HTTP.
This section describes XML Catalogs, the second solution. CXML
The XML Catalog object consulted by the parser before trying to
open an entity. Initially nil.
The default "prefer" mode from the Catalog specification, one
of :public or :system. Defaults
Function CXML:MAKE-CATALOG (&optional uris)
Return a catalog object for the catalog files specified.
Function CXML:RESOLVE-URI (uri catalog)
Look up uri in catalog and return the
resulting URI, or nil if no match was found.
Function CXML:RESOLVE-EXTID (publicid systemid catalog)
Look up the External ID (publicid, systemid)
in catalog and return the resulting URI, or nil
if no match was found.
* (setf cxml:*catalog* nil)
* (cxml:parse-file "test.xhtml" nil)
=> Error: URI scheme :HTTP not supported
* (setf cxml:*catalog* (cxml:make-catalog))
* (cxml:parse-file "test.xhtml" nil)
;; no error!
Note that parsed catalog files are cached in the catalog object.
Catalog files cached do not expire automatically. To ensure that
all catalog files are parsed again, create a new catalog object.
A SAX handler is an arbitrary objects that implements some of the
generic functions in the SAX package. Note that no default
handler class is necessary, because all generic functions have default
methods which do nothing. SAX functions are:
Function SAX:START-DOCUMENT (handler)
Function SAX:END-DOCUMENT (handler)
Function SAX:START-ELEMENT (handler namespace-uri local-name qname attributes)
Function SAX:END-ELEMENT (handler namespace-uri local-name qname)
Function SAX:START-PREFIX-MAPPING (handler prefix uri)
Function SAX:END-PREFIX-MAPPING (handler prefix)
Function SAX:PROCESSING-INSTRUCTION (handler target data)
Function SAX:COMMENT (handler data)
Function SAX:START-CDATA (handler)
Function SAX:END-CDATA (handler)
Function SAX:CHARACTERS (handler data)
Function SAX:START-DTD (handler name public-id system-id)
Function SAX:END-DTD (handler)
Function SAX:START-INTERNAL-SUBSET (handler)
Function SAX:END-INTERNAL-SUBSET (handler)
Function SAX:UNPARSED-ENTITY-DECLARATION (handler name public-id system-id notation-name)
Function SAX:EXTERNAL-ENTITY-DECLARATION (handler kind name public-id system-id)
Function SAX:INTERNAL-ENTITY-DECLARATION (handler kind name value)
Function SAX:NOTATION-DECLARATION (handler name public-id system-id)
Function SAX:ELEMENT-DECLARATION (handler name model)
Function SAX:ATTRIBUTE-DECLARATION (handler ename aname type default)
Accessor SAX:ATTRIBUTE-PREFIX (attribute)
Accessor SAX:ATTRIBUTE-NAMESPACE-URI (attribute)
Accessor SAX:ATTRIBUTE-LOCAL-NAME (attribute)
Accessor SAX:ATTRIBUTE-QNAME (attribute)
Accessor SAX:ATTRIBUTE-SPECIFIED-P (attribute)
Accessor SAX:ATTRIBUTE-VALUE (attribute)
Function SAX:FIND-ATTRIBUTE (qname attributes)
Function SAX:FIND-ATTRIBUTE-NS (uri lname attributes)
The entity declaration methods are similar to Java SAX
definitions, but parameter entities are distinguished from
general entities not by a % prefix to the name, but by
the kind argument, either :parameter or
The arguments to sax:element-declaration and
sax:attribute-declaration differ significantly from their
fixme: For more information on these functions refer to the docstrings.