- Nov 20, 2012
-
-
Raymond Toy authored
-
- Apr 25, 2012
-
-
Raymond Toy authored
it returns.
-
- Jan 12, 2012
-
-
Raymond Toy authored
code/extfmts.lisp: * Move the utf-8 implementation to this file * Update %find-external-format to return quickly for :utf-8. code/fd-stream-comp.lisp: * Precompile utf-8 functions. pcl/simple-streams/external-formats/utf-8.lisp: * Add note that this is only used for reference now because it's in the core.
-
- Nov 04, 2011
-
-
Raymond Toy authored
-
Raymond Toy authored
the number of characters converted when a buffer is given. If the buffer is not large enough, not all characters are converted. This is an incompatible change from the previous version. Suggested by Helmut Eller, cmucl-imp, 2011-10-29.
-
- Sep 25, 2011
-
-
Raymond Toy authored
entries with just the file path, removing the revision number, date, author and state. The actual information is now computed during compilation and stored in the fasl itself. (See ticket:48.)
-
- Nov 30, 2010
-
-
rtoy authored
Foley.) o While we're here, canonicalize *FILENAME-ENCODING* using the actual external format name.
-
- Sep 08, 2010
-
-
rtoy authored
happens.
-
- Sep 06, 2010
-
-
rtoy authored
-
- Sep 04, 2010
-
-
rtoy authored
-
- Aug 15, 2010
-
-
rtoy authored
the octets consumed for each character. This array is used to figure out the file position. Some tests comparing this scheme indicates a very small slowdown of about 1%, so this seems not to hurt. Use a cross-compile using the 2010-07 snapshot to build this. (Same procedure as used to build the 20b-pre1 release.) struct.lisp: o Add new slot OCTET-COUNT to LISP-STREAM to hold the array of octets per character. extfmts.lisp: o Add OCTETS-TO-STRING-COUNTED, which is like OCTETS-TO-STRING, except we need an array in which to store the number of octets consumed for each character processed. fd-stream.lisp: o Create the octet-count array creating the lisp stream string buffer. o In FD-STREAM-FILE-POSITION, use the octet count to count the number of octets that have been read but not yet returned to the user. stream.lisp: o Use OCTETS-TO-STRING-COUNTED instead of OCTETS-TO-STRING so we keep track of octet length of each character processed.
-
- Aug 14, 2010
-
-
rtoy authored
ascii.lisp: iso8859-1.lisp: iso8859-2.lisp: mac-roman.lisp: utf-16.lisp: utf-32-be.lisp: utf-32-le.lisp: utf-32.lisp: utf-8.lisp: o Inhibit warnings about funcalls to error (fdefinition of symbols). I'm tired of seeing the warnings. utf-16-be.lisp: utf-16-le.lisp: o Inhibit warnings about funcalls to error (fdefinition of symbols). I'm tired of seeing the warnings. o Fix bug in FLUSH-STATE: need to call the OUT function, not the ,OUTPUT function!
-
- Aug 04, 2010
-
-
rtoy authored
compiler noise for something we don't really care if it's slow.
-
- Jul 12, 2010
-
-
rtoy authored
little information about the format. Provide a means to get a list of external formats and to display the documentation. bootfiles/20a/boot-2010-07-1.lisp: o Use this bootstrap file when doing a normal build. code/exports.lisp: o New functions: - Add LIST-ALL-EXTERNAL-FORMATS to list all available external formats and their corresponding aliases. - Add DESCRIBE-EXTERNAL-FORMAT to print some information about the given format. o Add docuemntation slot to defstruct EXTERNAL-FORMAT. o Change DEFINE-EXTERNAL-FORMAT macro. Adds :DOCUMENTATION keyword to specify the documentation. Add :BASE keyword indicate that the external format is based on another format. (Previously, this wasn't needed, but is somewhat incompatible with adding a documentation string.) o Change DEFINE-COMPOSING-EXTERNAL-FORMAT to include :documentation keyword to specify the documentation for the format. o Minor reindentation of some docstrings. o Make sure documentation strings for external format are marked for translation; wrap other strings with intl:gettext to explicitly mark them for translations. o Add docstring for VOID and ISO8859-1 external formats. code/exports.lisp: o Export the new symbols LIST-ALL-EXTERNAL-FORMATS and DESCRIBE-EXTERNAL-FORMAT. Import into EXTENSIONS package. docs/cmu-user/unicode.tex: o Update docs to include LIST-ALL-EXTERNAL-FORMATS and DESCRIBE-EXTERNAL-FORMAT. o Update docs for DEFINE-EXTERNAL-FORMAT and DEFINE-COMPOSING-EXTERNAL-FORMAT to match implementation. general-info/release-20b.txt: o Update external-formats/*.lisp: o Update with docstrings. o Add :BASE keyword where needed.
-
- Jul 10, 2010
-
-
rtoy authored
o Add a simple function it list all external formats. o Add some docstrings. o Correctly indent some s-exps. exports.lisp: o Update package definitions to export new LIST-ALL-EXTERNAL-FORMATS.
-
- Jul 05, 2010
-
-
rtoy authored
args. o Generate different error messages for surrogate code points and code points that are too large.
-
rtoy authored
o Revert previous incompatible change to STRING-DECODE and STRING-ENCODE. Change the keyword parameters back to optional parameters, and make the error parameter the last one. fd-stream.lisp: o Update use of STRING-ENCODE.
-
- Jul 03, 2010
-
-
rtoy authored
o Update comments for the various slots in DEFINE-EXTERNAL-FORMAT. fd-stream.lisp: o Declare the types for the CHAR-TO-OCTETS-ERROR and OCTETS-TO-CHAR-ERROR slots in FD-STREAM. o Update docstrings for MAKE-FD-STREAM and OPEN for :DECODING-ERROR and :ENCODING-ERROR parameters.
-
rtoy authored
o Add error parameter to flush-state in external format definition so we can handle errors when flushing the state to a stream. o Add optional error parameter to flush-state macro. code/fd-stream.lisp: o For the case where an external format has flush method, EF-FLUSH was not calling it correctly. Update so the output function actuall works. o Add error handler to call to flush-state. o For the case where an external format does not have a flush method, output the state value instead of a replacement character so the external format can handle any errors.
-
- Jul 02, 2010
-
-
rtoy authored
o Pass the error handler on for composed external formats. code/fd-stream.lisp: o Forgot to pass the error-handler to char-to-octets in EF-COUT. o In MAKE-FD-STREAM slightly change handling of encoding-error and decoding-error: - If :encoding-error is a character, then the external format will use that character whenever an encoding error happens instead of the default (internally specified by the external format). - If :decoding-error is a character, then the external format will use that character whenever an encoding error happens instead of the default (internally specified by the external format). If :decoding-error is T, then a cerror is signaled; if continued, the Unicode replacement character (#\U+FFFD) is used. o Fix bug in OPEN: The :decoding-error and :encoding-error keyword parameter was placed in the &aux section by mistake.
-
rtoy authored
o The optional error parameter doesn't need to be optional in DEFINE-EXTERNAL-FORMAT, EF-STRING-TO-OCTETS, EF-OCTETS-TO-STRING, EF-ENCODE and EF-DECODE. code/fd-stream.lisp: o Update comments for char-to-octets-error and octets-to-char-error. o Forgot to pass the error handler to char-to-octets in EF-SOUT and EF-STRLEN.
-
rtoy authored
code/extfmts.lisp o Call the error handler for iso8859-1 output. o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external format with the error argument. o In OCTETS-TO-CHAR - Call OCTETS-TO-CODEPOINT with the error handler. - For all of the error conditions, call the error handler if defined. o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can handle errors. Call CHAR-TO-OCTETS with the error handler. o Add error parameter to STRING-TO-OCTETS and use it. o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can handle errors. Call OCTETS-TO-CHAR with the error handler. o Add error parameter to OCTETS-TO-STRING and use it. o In STRING-ENCODE and STRING-DECODE, call the ef function with the error handler. o Change STRING-ENCODE to use keyword args instead of optional args. Add error parameter and use it. code/fd-stream-extfmt.lisp: o Tell OCTETS-TO-STRING about the error handler stored in the fd-stream. code/fd-stream.lisp: o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and ENCODING-ERROR keyword arguments for specifying how to handle decoding and encoding errors in external formats. code/stream.lisp: o Make sure the error handler is called in FAST-READ-CHAR-STRING-REFILL. pcl/simple-streams/external-formats/utf-8.lisp: o Initial cut at calling the error handler for the various possible invalid octet streams for a utf-8 encoding.
-
- Jun 30, 2010
-
-
rtoy authored
o All external formats need an extra required argument for the error handler. o Add optional error parameter to OCTETS-TO-CODEPOINT, CODEPOINT-TO-OCTETS, OCTETS-TO-CHAR, and CHAR-TO-OCTETS.
-
- Apr 20, 2010
-
-
rtoy authored
may get confused with source locations if the reader macros are installed.
-
- Apr 19, 2010
-
-
rtoy authored
-
- Mar 19, 2010
-
-
rtoy authored
boot-2010-02-1 as the bootstrap file. You should probably also use the new -P option for build.sh to generate and update the po files while building.
-
- Mar 12, 2010
-
-
rtoy authored
ensure that we can compile the external format correctly no matter what the user might have done to readtable and other variables. This supersedes the previous change that just bound *readtable* to the standard read table.
-
- Mar 08, 2010
-
-
rtoy authored
format in case the user has set a non-standard readtable that can't process the external format.
-
- Oct 18, 2009
-
-
rtoy authored
faster reads on external-formats. This is done by adding an additional buffer to streams so we can convert the entire in-buffer into characters all at once. To build this change, you need to do a cross-compile using boot-2009-10-1-cross.lisp. Using that build, do a normal build with these sources. For a non-unicode build use boot-2009-10-01.lisp with a 20a non-unicode build. code/extfmts.lisp: o Add another slot to the extfmts for copying the state. o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the necesssary changes for fast formats. This is incompatible with the previous version because the string is not grown if needed. code/fd-stream-extfmt.lisp: o Set *enable-stream-buffer-p* to T so we have fast streams. code/fd-stream.lisp: o Add new slots to support fast strams. o In SET-ROUTINES, initialize the new slots appropriately. o Update UNREAD-CHAR to be able to back up in the string buffer to unread. o Add implementation to copy the state of an external format. code/stream.lisp: o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats even if we've already converted the buffer with a different format. We reconvert the buffer with the old format until we reach the current character. Then the remaining octets are converted using the new format and stored in the string buffer. o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like FAST-READ-CHAR-REFILL does for the octet in-buffer. code/struct.lisp: o Add new slots to hold the string buffer, the current index, and length. These are needed for the fast formats. code/sysmacs.lisp: o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and FAST-READ-CHAR to support the string buffer. code/string.lisp: o Microoptimization of SURROGATEP to reduce the number of branchs. general-info/release-20b.txt: o Update with these changes pcl/simple-streams/external-formats/utf-16-be.lisp: pcl/simple-streams/external-formats/utf-16-le.lisp: pcl/simple-streams/external-formats/utf-16.lisp: o These formats actually have state, so update them to take a handle an initial state. These are needed if the string buffer ends with a leading surrogate and the next string buffer starts with a trailing surrogate. The conversion needs to combine the surrogates together.
-
- Oct 02, 2009
-
-
rtoy authored
-
- Sep 19, 2009
-
-
rtoy authored
-
- Sep 17, 2009
-
-
rtoy authored
-
- Sep 09, 2009
-
-
rtoy authored
-
- Aug 26, 2009
-
-
rtoy authored
closing an output stream. This causes things like (with-open-file (s "foo" :direction :output :external-format :utf-8) (write-char #\u+d800 s)) to output the replacement character instead of creating an empty file. code/extfmts.lisp: o Add new slot to efx structure to hold the function to flush the state in an external format. o Add accessor for the flush-state slot. o Update DEFINE-EXTERNAL-FORMAT to allow specifying the flush function. o Add macro to call the flush-state function. o Added +EF-FLUSH+ o Use vm::defenum to name the constants instead of the hand-written values. o Export +REPLACEMENT-CHARACTER-CODE+ o Document the slots in an efx stucture. code/fd-stream.lisp: o Add ef-flush def-ef-macro to flush the state of an external format when closing an output file. If ef-flush-state is NIL, we just call EF-COUT to send out the replacement character. Otherwise, the flush-state function is called to handle it. o When closing an output character stream, call ef-flush to flush any state before flushing the buffers of the stream. o Document the unicode slots in an fd-stream. code/exports.lisp: o Export +REPLACEMENT-CHARACTER-CODE+
-
- Aug 13, 2009
-
-
rtoy authored
surrogate or a lone trailing surrogate) get replaced with the replacement character.
-
- Aug 11, 2009
-
-
rtoy authored
o Put back some unicode/unicode-bootstrap conditionals.
-
- Aug 10, 2009
-
-
rtoy authored
-
rtoy authored
o Standard streams no longer change formats when *default-external-format* changes. Use stream:set-system-external-format instead, or (setf external-format). o char-to-octets properly handles surrogate characters being written. o Makes simple-streams work again. This change needs to be cross-compiled. 2009-07 binaries work for cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp cross-compile script.
-
- Jul 23, 2009
-
-
rtoy authored
o Move the +ss-ef-foo+ constants to here from strategy.lisp, and update them so they don't overlap with existing +ef-foo+ constants. o Update +ef-max+ accordingly. pcl/simple-streams/impl.lisp: o Use +ss-ef-str+ instead of +ef-str+ in simple-stream-strlen. pcl/simple-streams/strategy.lisp: o Comment out +ss-ef-foo+ constants. o Use +ef-max+ instead of +ss-ef-max+, which is no longer defined. o Fix bugs in %dc-write-chars-fn: - Use ef variable - Need to call flush-out-buffer, not flush-buffer for dual-channel streams.
-
- Jun 25, 2009
-
-
rtoy authored
correctly. Previously, each surrogate was converted individually. This is wrong; they should be treated as a single codepoint that is converted.
-