Parent Directory | Revision Log
|Links to HEAD:||(view) (annotate)|
Make read-char and read-byte signal errors when given the wrong kind of streams. This is a change from current 20a and 20b behavior which didn't signal errors, but matches the behavior for releases 19f and earlier. But allow them to work on binary-text-streams. This is the same behavior as before for binary-text-stream streams. However, read-sequence no longer allows reading from streams into arbitrary objects, unless the stream is a binary-text-stream stream. code/fd-stream-extfmt.lisp: o In %SET-FD-STREAM-EXTERNAL-FORMAT, only update fd-stream-in/fd-stream-out if we have a character or binary-text-stream stream. o Don't update the fd-stream-string-buffer or lisp-stream-in-buffer if we have a binary-text-stream because that will mess up how fast-read-char and fast-read-byte dispatch to do the right thing for binary-text-stream streams. code/fd-stream.lisp: o Set the fd-stream-in and fd-stream-bin slots appropriately depending on whether we have a character, binary, or binary-text-stream stream. o Only create the lisp-stream-in-buffer if we do NOT have a binary-text-stream. (Binary streams didn't use the lisp-stream-buffer previously, so no change there. Character streams use the lisp-stream-buffer and/or lisp-string-buffer.) o Set the fd-stream-flags appropriately for the kind of stream this is. Checking a fixnum is faster than checking the type of a stream. code/struct.lisp: o Add FLAGS slot to LISP-STREAM so we can tell what kind of stream (character, binary, binary-text-stream) we have. code/sysmacs.lisp: o Change FAST-READ-CHAR so that if we have a have a binary or binary-text-stream stream, we dispatch to the fast-read-char methods to do the right thing, including signaling an error for the wrong kind of stream. o Change FAST-READ-BYTE so that if we do not have a binary stream, we dispatch to the fast-read-char method to do the right thing. compiler/dump.lisp: o With the above changes, we can no longer write characters to a binary stream, like a FASL file. Make the fasl file a binary-text-stream so that we can. (Alternatively, we could create the FASL header as a string, convert to octets and dump the octest to the file. This is easier, and should still be fast for writing fasls.)
Merge change from HEAD that fixes the case of changing the external format from :iso8859-1 to something else.
o When changing external format from :iso8859-1 to another format, we need to call octets-to-string-counted to setup the octet count array correctly. o Minor cleanup of code.
Clean up how (setf stream-external-format) works. This change requires a cross-compile using the default cross-compile script. fd-stream.lisp: o Remove the SAVED-OC-STATE slot from an FD-STREAM because we don't need it anymore since we have the OCTET-COUNT slot. stream.lisp: o Don't need to copy the current OC-STATE to SAVED-OC-STATE. Remove code and update comments. fd-stream-extfmt.lisp: o Use the OCTET-COUNT slot to figure out how many octets have been consumed so far to produce the characters that have already been read. Don't need to do the re-conversion anymore, so we don't need the SAVED-OC-STATE anymore. o Add support for the case where we were using the ISO8859-1 external format and are now switching to another external format that requires the string-buffer.
Merge changes from 20b-pre2.
Fix a couple of issues with changing the stream external format in midstream. o When changing the external format, the new converted characters need to go into the string-buffer at the point after the last character was read. This maintains consistency if the format is changed again before we have to read in another buffer of octets. o Handle the case where an external format that uses the string-buffer is changed to ISO8859-1, which doesn't. In this case, the converted characters need to be copied to the in-buffer in the right place and the string-buffer needs to be set to NIL to indicate that the string-buffer is not used anymore. o Add a note that we don't handle the case of changing ISO8859-1 to another external-format that uses a string-buffer.
Need to precompile the ef slots for both unicode and non-unicode builds. (This was broken in the 2010-04 snapshot.) code/fd-stream-comp.lisp: o Precompile ef slots there for ISO8859-1. code/fd-stream-extfmts.lisp: o Remove the precompile stuff. tools/worldcom.lisp: o Compile fd-stream-comp. tools/worldload.lisp: o Load fd-stream-comp.
Implement more of the external format error handlers. code/extfmts.lisp o Call the error handler for iso8859-1 output. o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external format with the error argument. o In OCTETS-TO-CHAR - Call OCTETS-TO-CODEPOINT with the error handler. - For all of the error conditions, call the error handler if defined. o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can handle errors. Call CHAR-TO-OCTETS with the error handler. o Add error parameter to STRING-TO-OCTETS and use it. o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can handle errors. Call OCTETS-TO-CHAR with the error handler. o Add error parameter to OCTETS-TO-STRING and use it. o In STRING-ENCODE and STRING-DECODE, call the ef function with the error handler. o Change STRING-ENCODE to use keyword args instead of optional args. Add error parameter and use it. code/fd-stream-extfmt.lisp: o Tell OCTETS-TO-STRING about the error handler stored in the fd-stream. code/fd-stream.lisp: o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and ENCODING-ERROR keyword arguments for specifying how to handle decoding and encoding errors in external formats. code/stream.lisp: o Make sure the error handler is called in FAST-READ-CHAR-STRING-REFILL. pcl/simple-streams/external-formats/utf-8.lisp: o Initial cut at calling the error handler for the various possible invalid octet streams for a utf-8 encoding.
Change uses of _"foo" to (intl:gettext "foo"). This is because slime may get confused with source locations if the reader macros are installed.
Merge intl-branch 2010-03-18 to HEAD. To build, you need to use boot-2010-02-1 as the bootstrap file. You should probably also use the new -P option for build.sh to generate and update the po files while building.
Restart internalization work. This new branch starts with code from the intl-branch on date 2010-02-12 18:00:00+0500. This version works and LANG=en@piglatin bin/lisp works (once the piglatin translation is added).
Need to precompile ef-en and ef-de.
Mark translatable strings; update cmucl.pot and ko/cmucl.po accordingly.
Add (intl:textdomain "cmucl") to the files to set the textdomain.
Oops. Last change to fd-stream doesn't actually compile because no everything is defined yet. Hence, add dummy %SET-FD-STREAM-EXTERNAL-FORMAT and move the real one to fd-stream-extfmt.lisp. This builds. code/fd-stream.lisp: o Always call %SET-FD-STREAM-EXTERNAL-FORMAT, even if LISP::*ENABLE-STREAM-BUFFER-P* is NIL. code/stream.lisp: o Move %SET-FD-STREAM-EXTERNAL-FORMAT to fd-stream-extfmt.lisp. o Add dummy implementation of %SET-FD-STREAM-EXTERNAL-FORMAT. code/fd-stream-extfmt.lisp: o %SET-FD-STREAM-EXTERNAL-FORMAT moved here.
Remove commented-out call to set-terminal-coding-system, which doesn't exist anymore anyway. (It's called set-system-external-format.)
Merge changes from unicode-string-buffer-impl-branch which gives faster reads on external-formats. This is done by adding an additional buffer to streams so we can convert the entire in-buffer into characters all at once. To build this change, you need to do a cross-compile using boot-2009-10-1-cross.lisp. Using that build, do a normal build with these sources. For a non-unicode build use boot-2009-10-01.lisp with a 20a non-unicode build. code/extfmts.lisp: o Add another slot to the extfmts for copying the state. o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the necesssary changes for fast formats. This is incompatible with the previous version because the string is not grown if needed. code/fd-stream-extfmt.lisp: o Set *enable-stream-buffer-p* to T so we have fast streams. code/fd-stream.lisp: o Add new slots to support fast strams. o In SET-ROUTINES, initialize the new slots appropriately. o Update UNREAD-CHAR to be able to back up in the string buffer to unread. o Add implementation to copy the state of an external format. code/stream.lisp: o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats even if we've already converted the buffer with a different format. We reconvert the buffer with the old format until we reach the current character. Then the remaining octets are converted using the new format and stored in the string buffer. o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like FAST-READ-CHAR-REFILL does for the octet in-buffer. code/struct.lisp: o Add new slots to hold the string buffer, the current index, and length. These are needed for the fast formats. code/sysmacs.lisp: o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and FAST-READ-CHAR to support the string buffer. code/string.lisp: o Microoptimization of SURROGATEP to reduce the number of branchs. general-info/release-20b.txt: o Update with these changes pcl/simple-streams/external-formats/utf-16-be.lisp: pcl/simple-streams/external-formats/utf-16-le.lisp: pcl/simple-streams/external-formats/utf-16.lisp: o These formats actually have state, so update them to take a handle an initial state. These are needed if the string buffer ends with a leading surrogate and the next string buffer starts with a trailing surrogate. The conversion needs to combine the surrogates together.
Implement fast external formats. This is done for buffered streams where an in-buffer would be allocated. For character streams, a string-buffer is also allocated. Whenever the string-buffer is empty, the in-buffer is filled with new octets, and the octets are immediately converted to a string. FAST-READ-CHAR pulls the characters from string-buffer as needed. This speeds up utf8 processing by 30-40% (for read-line). fd-stream-extfmt.lisp: o Precompile the +ef-os+ slot for iso8859-1. (Not necessary?) o Set *enable-stream-buffer-p* to T to enable the buffer. fd-stream.lisp: o In SET-ROUTINES, allocate the string-buffer when appropriate. o Update FD-STREAM-MISC-ROUTINE to handle :unread with the string buffer. stream.lisp: o Update UNREAD-CHAR to handle the string buffer. May need more work. o Add FAST-READ-CHAR-STRING-REFILL to refill the in-buffer and string-buffer
Fixes from Paul Foley: o Standard streams no longer change formats when *default-external-format* changes. Use stream:set-system-external-format instead, or (setf external-format). o char-to-octets properly handles surrogate characters being written. o Makes simple-streams work again. This change needs to be cross-compiled. 2009-07 binaries work for cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp cross-compile script.
Merge Unicode work to trunk. From label unicode-utf16-extfmt-2009-06-11.
More updates from Paul. code/extfmts.lisp: o Fixed bug with shared code between formats o Built a cache into the ef-macro functions so it doesn't need to call find-external-format so often at runtime code/fd-stream-extfmt.lisp o Use the changes in code/extfmts code/fd-stream.lisp: o Removed all the commented-out code in fd-stream which is duplicated in fd-stream-extfmt.
More updates from Paul. fd-stream-extfmt.lisp actually implements the external formats which now work. Cross-compile works fine. code/fd-stream-extfmt.lisp: o New file implementing external formats tools/worldcom.lisp: o Compile extfmts.lisp before fd-stream, since fd-stream uses some macros from extfmts. o Compile fd-stream-extfmt tools/worldload.lisp: o Load fd-stream-extfmt at the end. (Can't load it as part of kernel.core. Not enough is set up yet.) code/extfmts.lisp: o Avoid loading files, etc., early in the boot sequence o Add INVERT-TABLE function needed by some formats. code/fd-stream.lisp: o Some cleanups (I think) o Fix EOF handling
file fd-stream-extfmt.lisp was initially added on branch unicode-utf16-extfmt-branch.
This form allows you to request diffs between any two revisions of this file. For each of the two "sides" of the diff, select a symbolic revision name using the selection box, or choose 'Use Text Field' and enter a numeric revision.
|Powered by ViewVC 1.1.5|