Parent Directory | Revision Log
|Links to HEAD:||(view) (annotate)|
|Links to unicode-string-buffer-impl-branch:||(view) (annotate)|
o Update docstring for OCTETS-TO-STRING to match the implementation. o Use AREF instead of BREF in EF-OCTETS-TO-STRING.
Merge changes from unicode-string-buffer-branch.
In OCTETS-TO-STRING, correct the computation of the end of the string. (It caused an error if the string were not given.)
extfmts.lisp: o Allow caller to specify a state so octets can be converted in batches with the appropriate state between them. stream.lisp: o Call OCTETS-TO-STRING with the appropriate state from the stream. o Better declarations in FAST-READ-CHAR-STRING-REFILL.
o Use bref instead of aref in EF-OCTETS-TO-STRING. (Generated code is smaller and better. Probably some issue with register allocation and vop selection.) o Use SAFETY 0 in EF-OCTETS-TO-STRING, depending on OCTETS-TO-STRING to do the correct things. With these changes, utf16 is now the same speed is 20a, and utf8 is much faster than 20a.
Declare some variables to get rid of some compiler notes.
Merge changes to OCTETS-TO-STRING from unicode-string-buffer-branch.
Merge changes from the release-20a branch.
Merge changes from 20a branch.
Merge changes from 20a-pre1 (tag release-20a-pre1) to trunk.
Add support for flushing out any state in an external format when closing an output stream. This causes things like (with-open-file (s "foo" :direction :output :external-format :utf-8) (write-char #\u+d800 s)) to output the replacement character instead of creating an empty file. code/extfmts.lisp: o Add new slot to efx structure to hold the function to flush the state in an external format. o Add accessor for the flush-state slot. o Update DEFINE-EXTERNAL-FORMAT to allow specifying the flush function. o Add macro to call the flush-state function. o Added +EF-FLUSH+ o Use vm::defenum to name the constants instead of the hand-written values. o Export +REPLACEMENT-CHARACTER-CODE+ o Document the slots in an efx stucture. code/fd-stream.lisp: o Add ef-flush def-ef-macro to flush the state of an external format when closing an output file. If ef-flush-state is NIL, we just call EF-COUT to send out the replacement character. Otherwise, the flush-state function is called to handle it. o When closing an output character stream, call ef-flush to flush any state before flushing the buffers of the stream. o Document the unicode slots in an fd-stream. code/exports.lisp: o Export +REPLACEMENT-CHARACTER-CODE+
Illegal surrogate sequences (leading surrogate without trailing surrogate or a lone trailing surrogate) get replaced with the replacement character.
o Put some comments back in. o Put back some unicode/unicode-bootstrap conditionals.
Compile and load the external format code.
Fixes from Paul Foley: o Standard streams no longer change formats when *default-external-format* changes. Use stream:set-system-external-format instead, or (setf external-format). o char-to-octets properly handles surrogate characters being written. o Makes simple-streams work again. This change needs to be cross-compiled. 2009-07 binaries work for cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp cross-compile script.
code/extfmts.lisp: o Move the +ss-ef-foo+ constants to here from strategy.lisp, and update them so they don't overlap with existing +ef-foo+ constants. o Update +ef-max+ accordingly. pcl/simple-streams/impl.lisp: o Use +ss-ef-str+ instead of +ef-str+ in simple-stream-strlen. pcl/simple-streams/strategy.lisp: o Comment out +ss-ef-foo+ constants. o Use +ef-max+ instead of +ss-ef-max+, which is no longer defined. o Fix bugs in %dc-write-chars-fn: - Use ef variable - Need to call flush-out-buffer, not flush-buffer for dual-channel streams.
Update STRING-ENCODE and STRING-DECODE to handle surrogate pairs correctly. Previously, each surrogate was converted individually. This is wrong; they should be treated as a single codepoint that is converted.
Fix FILE-STRING-LENGTH to handle unicode streams. From Paul.
Fix bug introduced when converting tables from hashes to ntries. From Paul Foley. This makes mac-roman and derived external formats work once again.
Merge Unicode work to trunk. From label unicode-utf16-extfmt-2009-06-11.
Update from Paul: I've moved some slots out of external-format so they can be shared between external-formats that are identical in all but some variables. Also fixed a bug in octets-to-string that made it return one character short, and used char-code-limit instead of #x100 to determine when octets-to-char returns a "?", so now it'll work without change on 8 or 16 bit lisps.
Create a new search-list "ext-formats" that is initialized to "library:ext-formats/". This makes it easier to add new directories where external formats can be found. The previous use made it difficult because the formats had to be in the subdirectory ext-formats. save.lisp: o Create and initialize new search-list. extfmts.lisp: o Use the new search-list instead of "library:ext-formats/".
New external format stuff from Paul. bootfiles/19e/boot-2008-06-1.lisp: o Use this bootfile to compile the change in external-format structure. Just needed to get rid of a restart when compiling pcl. code/exports.lisp: o Renames ENCODE-STRING to STRING-ENCODE. Similarly for DECODE-STRING. code/extfmts.lisp: pcl/simple-streams/impl.lisp: pcl/simple-streams/strategy.lisp: pcl/simple-streams/external-formats/iso8859-1.lisp: pcl/simple-streams/external-formats/utf-8.lisp: pcl/simple-streams/external-formats/void.lisp: o Updated for new external format. I think the main change is not having to do a funcall for each character. pcl/simple-streams/external-formats/aliases o New file describing different names for external formats. pcl/simple-streams/external-formats/crlf.lisp: o New file for composing external format for CR/LF pcl/simple-streams/external-formats/utf-16-be.lisp: pcl/simple-streams/external-formats/utf-16-le.lisp: o New files supporting UTF-16 BE and LE formats. tools/make-main-dist.sh: o Copy over the new files and the aliases file too.
Update from Paul Foley. o Disable package errors when loading up external formats. o A minor patch allowing string-to-octets and vice versa to write into a preallocated array (though they might still allocate a bigger one if necessary), o Fix up any confusion between simple-base-string and simple-string so that nothing breaks when/if they're not the same.
Import Paul Foley's external-formats support. New files: o code/extfmts.lisp o pcl/simple-streams/external-formats/iso8859-1.lisp o pcl/simple-streams/external-formats/void.lisp code/exports.lisp: o Export the new symbols STRING-TO-OCTETS, OCTETS-TO-STRING, *DEFAULT-EXTERNAL-FORMAT*, ENCODE-STRING, and DECODE-STRING from the STREAM package o Make the symbols in the EXT package too. pcl/simple-streams/internal.lisp: o Move the implementation of STRING-TO-OCTETS and friends to a new file (extfmts.lisp). pcl/simple-streams/external-formats/utf-8.lisp: o New implementation. tools/make-main-dist.sh: o Create new target directory to hold external formats o Copy all of the external formats to the new directory. tools/pclcom.lisp: o Compile new code tools/worldcom.lisp: o Compile code/extfmts.lisp tools/worldload.lisp: o Load code/extfmts.lisp
This form allows you to request diffs between any two revisions of this file. For each of the two "sides" of the diff, select a symbolic revision name using the selection box, or choose 'Use Text Field' and enter a numeric revision.
|Powered by ViewVC 1.1.5|