/[cmucl]/src/code/fd-stream-extfmt.lisp
ViewVC logotype

Log of /src/code/fd-stream-extfmt.lisp

Parent Directory Parent Directory | Revision Log Revision Log


Links to HEAD: (view) (annotate)
Sticky Tag:

Revision 1.14 - (view) (annotate) - [select for diffs]
Tue Oct 12 21:52:44 2010 UTC (3 years, 6 months ago) by rtoy
Branch: MAIN
CVS Tags: GIT-CONVERSION, HEAD, cross-sol-x86-2010-12-20, cross-sol-x86-base, cross-sol-x86-merged, cross-sparc-branch-base, snapshot-2010-11, snapshot-2010-12, snapshot-2011-01, snapshot-2011-02, snapshot-2011-03, snapshot-2011-04, snapshot-2011-06, snapshot-2011-07, snapshot-2011-09
Branch point for: cross-sol-x86-branch, cross-sparc-branch
Changes since 1.13: +95 -83 lines
Diff to previous 1.13
Make read-char and read-byte signal errors when given the wrong kind
of streams.  This is a change from current 20a and 20b behavior which
didn't signal errors, but matches the behavior for releases 19f and
earlier.

But allow them to work on binary-text-streams.  This is the same
behavior as before for binary-text-stream streams.

However, read-sequence no longer allows reading from streams into
arbitrary objects, unless the stream is a binary-text-stream stream.

code/fd-stream-extfmt.lisp:
o In %SET-FD-STREAM-EXTERNAL-FORMAT, only update
  fd-stream-in/fd-stream-out if we have a character or
  binary-text-stream stream.
o Don't update the fd-stream-string-buffer or lisp-stream-in-buffer if
  we have a binary-text-stream because that will mess up how
  fast-read-char and fast-read-byte dispatch to do the right thing for
  binary-text-stream streams.

code/fd-stream.lisp:
o Set the fd-stream-in and fd-stream-bin slots appropriately depending
  on whether we have a character, binary, or binary-text-stream
  stream.
o Only create the lisp-stream-in-buffer if we do NOT have a
  binary-text-stream.  (Binary streams didn't use the
  lisp-stream-buffer previously, so no change there.  Character
  streams use the lisp-stream-buffer and/or lisp-string-buffer.)
o Set the fd-stream-flags appropriately for the kind of stream this
  is.  Checking a fixnum is faster than checking the type of a stream.

code/struct.lisp:
o Add FLAGS slot to LISP-STREAM so we can tell what kind of stream
  (character, binary, binary-text-stream) we have.

code/sysmacs.lisp:
o Change FAST-READ-CHAR so that if we have a have a binary or
  binary-text-stream stream, we dispatch to the fast-read-char methods
  to do the right thing, including signaling an error for the wrong
  kind of stream.
o Change FAST-READ-BYTE so that if we do not have a binary stream, we
  dispatch to the fast-read-char method to do the right thing.

compiler/dump.lisp:
o With the above changes, we can no longer write characters to a
  binary stream, like a FASL file.  Make the fasl file a
  binary-text-stream so that we can.  (Alternatively, we could create
  the FASL header as a string, convert to octets and dump the octest
  to the file.  This is easier, and should still be fast for writing
  fasls.)

Revision 1.10.2.2 - (view) (annotate) - [select for diffs]
Fri Sep 24 00:51:03 2010 UTC (3 years, 6 months ago) by rtoy
Branch: RELEASE-20B-BRANCH
CVS Tags: RELEASE_20b
Changes since 1.10.2.1: +90 -60 lines
Diff to previous 1.10.2.1 , to branch point 1.10 , to next main 1.14
Merge change from HEAD that fixes the case of changing the external
format from :iso8859-1 to something else.

Revision 1.13 - (view) (annotate) - [select for diffs]
Fri Sep 24 00:36:03 2010 UTC (3 years, 6 months ago) by rtoy
Branch: MAIN
Changes since 1.12: +55 -50 lines
Diff to previous 1.12
o When changing external format from :iso8859-1 to another format, we
  need to call octets-to-string-counted to setup the octet count array
  correctly.
o Minor cleanup of code.

Revision 1.12 - (view) (annotate) - [select for diffs]
Wed Sep 8 03:04:54 2010 UTC (3 years, 7 months ago) by rtoy
Branch: MAIN
Changes since 1.11: +85 -60 lines
Diff to previous 1.11
Clean up how (setf stream-external-format) works.  This change
requires a cross-compile using the default cross-compile script.

fd-stream.lisp:
o Remove the SAVED-OC-STATE slot from an FD-STREAM because we don't
  need it anymore since we have the OCTET-COUNT slot.

stream.lisp:
o Don't need to copy the current OC-STATE to SAVED-OC-STATE.  Remove
  code and update comments.

fd-stream-extfmt.lisp:
o Use the OCTET-COUNT slot to figure out how many octets have been
  consumed so far to produce the characters that have already been
  read.  Don't need to do the re-conversion anymore, so we don't need
  the SAVED-OC-STATE anymore.
o Add support for the case where we were using the ISO8859-1 external
  format and are now switching to another external format that
  requires the string-buffer.

Revision 1.11 - (view) (annotate) - [select for diffs]
Mon Sep 6 19:01:56 2010 UTC (3 years, 7 months ago) by rtoy
Branch: MAIN
Changes since 1.10: +30 -11 lines
Diff to previous 1.10
Merge changes from 20b-pre2.

Revision 1.10.2.1 - (view) (annotate) - [select for diffs]
Mon Sep 6 15:41:30 2010 UTC (3 years, 7 months ago) by rtoy
Branch: RELEASE-20B-BRANCH
CVS Tags: release-20b-pre2
Changes since 1.10: +30 -11 lines
Diff to previous 1.10
Fix a couple of issues with changing the stream external format in
midstream.

o When changing the external format, the new converted characters need
  to go into the string-buffer at the point after the last character
  was read.  This maintains consistency if the format is changed
  again before we have to read in another buffer of octets.
o Handle the case where an external format that uses the string-buffer
  is changed to ISO8859-1, which doesn't.  In this case, the converted
  characters need to be copied to the in-buffer in the right place and
  the string-buffer needs to be set to NIL to indicate that the
  string-buffer is not used anymore.
o Add a note that we don't handle the case of changing ISO8859-1 to
  another external-format that uses a string-buffer.

Revision 1.10 - (view) (annotate) - [select for diffs]
Tue Jul 20 21:34:29 2010 UTC (3 years, 8 months ago) by rtoy
Branch: MAIN
CVS Tags: release-20b-pre1, snapshot-2010-08
Branch point for: RELEASE-20B-BRANCH
Changes since 1.9: +1 -8 lines
Diff to previous 1.9
Need to precompile the ef slots for both unicode and non-unicode
builds.  (This was broken in the 2010-04 snapshot.)

code/fd-stream-comp.lisp:
o Precompile ef slots there for ISO8859-1.

code/fd-stream-extfmts.lisp:
o Remove the precompile stuff.

tools/worldcom.lisp:
o Compile fd-stream-comp.

tools/worldload.lisp:
o Load fd-stream-comp.

Revision 1.9 - (view) (annotate) - [select for diffs]
Fri Jul 2 02:50:35 2010 UTC (3 years, 9 months ago) by rtoy
Branch: MAIN
CVS Tags: sparc-tramp-assem-2010-07-19, sparc-tramp-assem-base
Branch point for: sparc-tramp-assem-branch
Changes since 1.8: +5 -3 lines
Diff to previous 1.8
Implement more of the external format error handlers.

code/extfmts.lisp
o Call the error handler for iso8859-1 output.
o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external
  format with the error argument.
o In OCTETS-TO-CHAR
  - Call OCTETS-TO-CODEPOINT with the error handler.
  - For all of the error conditions, call the error handler if
    defined.
o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can
  handle errors.  Call CHAR-TO-OCTETS with the error handler.
o Add error parameter to STRING-TO-OCTETS and use it.
o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can
  handle errors.  Call OCTETS-TO-CHAR with the error handler.
o Add error parameter to OCTETS-TO-STRING and use it.
o In STRING-ENCODE and STRING-DECODE, call the ef function with the
  error handler.
o Change STRING-ENCODE to use keyword args instead of optional args.
  Add error parameter and use it.

code/fd-stream-extfmt.lisp:
o Tell OCTETS-TO-STRING about the error handler stored in the
  fd-stream.

code/fd-stream.lisp:
o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and
  ENCODING-ERROR keyword arguments for specifying how to handle
  decoding and encoding errors in external formats.

code/stream.lisp:
o Make sure the error handler is called in
  FAST-READ-CHAR-STRING-REFILL.

pcl/simple-streams/external-formats/utf-8.lisp:
o Initial cut at calling the error handler for the various possible
  invalid octet streams for a utf-8 encoding.

Revision 1.8 - (view) (annotate) - [select for diffs]
Tue Apr 20 17:57:44 2010 UTC (3 years, 11 months ago) by rtoy
Branch: MAIN
CVS Tags: snapshot-2010-05, snapshot-2010-06, snapshot-2010-07
Changes since 1.7: +4 -4 lines
Diff to previous 1.7
Change uses of _"foo" to (intl:gettext "foo").  This is because slime
may get confused with source locations if the reader macros are
installed.

Revision 1.7 - (view) (annotate) - [select for diffs]
Fri Mar 19 15:18:59 2010 UTC (4 years ago) by rtoy
Branch: MAIN
CVS Tags: post-merge-intl-branch, snapshot-2010-04
Changes since 1.6: +9 -4 lines
Diff to previous 1.6
Merge intl-branch 2010-03-18 to HEAD.  To build, you need to use
boot-2010-02-1 as the bootstrap file.  You should probably also use
the new -P option for build.sh to generate and update the po files
while building.

Revision 1.6.4.1 - (view) (annotate) - [select for diffs]
Thu Feb 25 20:34:49 2010 UTC (4 years, 1 month ago) by rtoy
Branch: intl-2-branch
Changes since 1.6: +9 -4 lines
Diff to previous 1.6 , to next main 1.14
Restart internalization work.  This new branch starts with code from
the intl-branch on date 2010-02-12 18:00:00+0500.  This version works
and

LANG=en@piglatin bin/lisp

works (once the piglatin translation is added).

Revision 1.6.2.3 - (view) (annotate) - [select for diffs]
Fri Feb 12 02:32:06 2010 UTC (4 years, 2 months ago) by rtoy
Branch: intl-branch
CVS Tags: intl-branch-2010-03-18-1300, intl-branch-working-2010-02-11-1000, intl-branch-working-2010-02-19-1000
Changes since 1.6.2.2: +4 -1 lines
Diff to previous 1.6.2.2 , to branch point 1.6 , to next main 1.14
Need to precompile ef-en and ef-de.

Revision 1.6.2.2 - (view) (annotate) - [select for diffs]
Tue Feb 9 20:07:04 2010 UTC (4 years, 2 months ago) by rtoy
Branch: intl-branch
Changes since 1.6.2.1: +4 -4 lines
Diff to previous 1.6.2.1 , to branch point 1.6
Mark translatable strings; update cmucl.pot and ko/cmucl.po
accordingly.

Revision 1.6.2.1 - (view) (annotate) - [select for diffs]
Mon Feb 8 17:15:47 2010 UTC (4 years, 2 months ago) by rtoy
Branch: intl-branch
Changes since 1.6: +3 -1 lines
Diff to previous 1.6
Add (intl:textdomain "cmucl") to the files to set the textdomain.

Revision 1.6 - (view) (annotate) - [select for diffs]
Sat Jan 23 18:02:04 2010 UTC (4 years, 2 months ago) by rtoy
Branch: MAIN
CVS Tags: intl-2-branch-base, intl-branch-base, pre-merge-intl-branch, snapshot-2010-02, snapshot-2010-03
Branch point for: intl-2-branch, intl-branch
Changes since 1.5: +53 -1 lines
Diff to previous 1.5
Oops.  Last change to fd-stream doesn't actually compile because no
everything is defined yet.  Hence, add dummy
%SET-FD-STREAM-EXTERNAL-FORMAT and move the real one to
fd-stream-extfmt.lisp.  This builds.

code/fd-stream.lisp:
o Always call %SET-FD-STREAM-EXTERNAL-FORMAT, even if
  LISP::*ENABLE-STREAM-BUFFER-P* is NIL.

code/stream.lisp:
o Move %SET-FD-STREAM-EXTERNAL-FORMAT to fd-stream-extfmt.lisp.
o Add dummy implementation of %SET-FD-STREAM-EXTERNAL-FORMAT.

code/fd-stream-extfmt.lisp:
o %SET-FD-STREAM-EXTERNAL-FORMAT moved here.

Revision 1.5 - (view) (annotate) - [select for diffs]
Tue Dec 15 17:22:41 2009 UTC (4 years, 4 months ago) by rtoy
Branch: MAIN
CVS Tags: snapshot-2010-01
Changes since 1.4: +1 -5 lines
Diff to previous 1.4
Remove commented-out call to set-terminal-coding-system, which doesn't
exist anymore anyway.  (It's called set-system-external-format.)

Revision 1.4 - (view) (annotate) - [select for diffs]
Sun Oct 18 14:21:24 2009 UTC (4 years, 5 months ago) by rtoy
Branch: MAIN
CVS Tags: amd64-dd-start, snapshot-2009-11, snapshot-2009-12
Branch point for: amd64-dd-branch
Changes since 1.3: +4 -1 lines
Diff to previous 1.3
Merge changes from unicode-string-buffer-impl-branch which gives
faster reads on external-formats.  This is done by adding an
additional buffer to streams so we can convert the entire in-buffer
into characters all at once.

To build this change, you need to do a cross-compile using
boot-2009-10-1-cross.lisp.  Using that build, do a normal build with
these sources.

For a non-unicode build use boot-2009-10-01.lisp with a 20a
non-unicode build.

code/extfmts.lisp:
o Add another slot to the extfmts for copying the state.
o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the
  necesssary changes for fast formats.  This is incompatible with the
  previous version because the string is not grown if needed.

code/fd-stream-extfmt.lisp:
o Set *enable-stream-buffer-p* to T so we have fast streams.

code/fd-stream.lisp:
o Add new slots to support fast strams.
o In SET-ROUTINES, initialize the new slots appropriately.
o Update UNREAD-CHAR to be able to back up in the string buffer to
  unread.
o Add implementation to copy the state of an external format.

code/stream.lisp:
o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats
  even if we've already converted the buffer with a different format.
  We reconvert the buffer with the old format until we reach the
  current character.  Then the remaining octets are converted using
  the new format and stored in the string buffer.
o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like
  FAST-READ-CHAR-REFILL does for the octet in-buffer.

code/struct.lisp:
o Add new slots to hold the string buffer, the current index, and
  length.  These are needed for the fast formats.

code/sysmacs.lisp:
o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and
  FAST-READ-CHAR to support the string buffer.

code/string.lisp:
o Microoptimization of SURROGATEP to reduce the number of branchs.

general-info/release-20b.txt:
o Update with these changes

pcl/simple-streams/external-formats/utf-16-be.lisp:
pcl/simple-streams/external-formats/utf-16-le.lisp:
pcl/simple-streams/external-formats/utf-16.lisp:
o These formats actually have state, so update them to take a handle
  an initial state.  These are needed if the string buffer ends with a
  leading surrogate and the next string buffer starts with a trailing
  surrogate.  The conversion needs to combine the surrogates together.

Revision 1.3.6.1 - (view) (annotate) - [select for diffs]
Sat Sep 19 15:16:05 2009 UTC (4 years, 6 months ago) by rtoy
Branch: unicode-string-buffer-impl-branch
Changes since 1.3: +4 -1 lines
Diff to previous 1.3 , to next main 1.14
Implement fast external formats.  This is done for buffered streams
where an in-buffer would be allocated.  For character streams, a
string-buffer is also allocated.  Whenever the string-buffer is empty,
the in-buffer is filled with new octets, and the octets are
immediately converted to a string.  FAST-READ-CHAR pulls the
characters from string-buffer as needed.  This speeds up utf8
processing by 30-40% (for read-line).

fd-stream-extfmt.lisp:
o Precompile the +ef-os+ slot for iso8859-1. (Not necessary?)
o Set *enable-stream-buffer-p* to T to enable the buffer.

fd-stream.lisp:
o In SET-ROUTINES, allocate the string-buffer when appropriate.
o Update FD-STREAM-MISC-ROUTINE to handle :unread with the string
  buffer.

stream.lisp:
o Update UNREAD-CHAR to handle the string buffer.  May need more
  work.
o Add FAST-READ-CHAR-STRING-REFILL to refill the in-buffer and
  string-buffer

Revision 1.3 - (view) (annotate) - [select for diffs]
Mon Aug 10 16:47:41 2009 UTC (4 years, 8 months ago) by rtoy
Branch: MAIN
CVS Tags: RELEASE_20a, release-20a-base, release-20a-pre1, snapshot-2009-08, unicode-string-buffer-base, unicode-string-buffer-impl-base
Branch point for: RELEASE-20A-BRANCH, unicode-string-buffer-branch, unicode-string-buffer-impl-branch
Changes since 1.2: +26 -48 lines
Diff to previous 1.2
Fixes from Paul Foley:

o Standard streams no longer change formats when
  *default-external-format* changes.  Use
  stream:set-system-external-format instead, or (setf
  external-format).
o char-to-octets properly handles surrogate characters being written.
o Makes simple-streams work again.

This change needs to be cross-compiled.  2009-07 binaries work for
cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp
cross-compile script.

Revision 1.2 - (view) (annotate) - [select for diffs]
Thu Jun 11 16:03:57 2009 UTC (4 years, 10 months ago) by rtoy
Branch: MAIN
CVS Tags: merged-unicode-utf16-extfmt-2009-06-11, portable-clx-base, portable-clx-import-2009-06-16, snapshot-2009-07
Branch point for: portable-clx-branch
Changes since 1.1: +94 -0 lines
Diff to previous 1.1
Merge Unicode work to trunk.  From label
unicode-utf16-extfmt-2009-06-11.

Revision 1.1.2.2 - (view) (annotate) - [select for diffs]
Mon Jul 14 14:01:56 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
CVS Tags: unicode-snapshot-2009-05, unicode-snapshot-2009-06, unicode-utf16-extfmt-2009-03-27, unicode-utf16-extfmt-2009-06-11, unicode-utf16-extfmts-pre-sync-2008-11, unicode-utf16-extfmts-sync-2008-12
Changes since 1.1.2.1: +6 -11 lines
Diff to previous 1.1.2.1 , to branch point 1.1 , to next main 1.14
More updates from Paul.

code/extfmts.lisp:
o Fixed bug with shared code between formats
o Built a cache into the ef-macro functions so it doesn't need to call
  find-external-format so often at runtime

code/fd-stream-extfmt.lisp
o Use the changes in code/extfmts

code/fd-stream.lisp:
o Removed all the commented-out code in fd-stream which is duplicated
  in fd-stream-extfmt.

Revision 1.1.2.1 - (view) (annotate) - [select for diffs]
Sat Jul 5 12:37:42 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.1: +99 -0 lines
Diff to previous 1.1
More updates from Paul.  fd-stream-extfmt.lisp actually implements the
external formats which now work.

Cross-compile works fine.

code/fd-stream-extfmt.lisp:
o New file implementing external formats

tools/worldcom.lisp:
o Compile extfmts.lisp before fd-stream, since fd-stream uses some
  macros from extfmts.
o Compile fd-stream-extfmt

tools/worldload.lisp:
o Load fd-stream-extfmt at the end.  (Can't load it as part of
  kernel.core.  Not enough is set up yet.)

code/extfmts.lisp:
o Avoid loading files, etc., early in the boot sequence
o Add INVERT-TABLE function needed by some formats.

code/fd-stream.lisp:
o Some cleanups (I think)
o Fix EOF handling

Revision 1.1
Sat Jul 5 12:37:42 2008 UTC (5 years, 9 months ago) by rtoy
Branch: MAIN
Branch point for: unicode-utf16-extfmt-branch
FILE REMOVED
file fd-stream-extfmt.lisp was initially added on branch unicode-utf16-extfmt-branch.

This form allows you to request diffs between any two revisions of this file. For each of the two "sides" of the diff, select a symbolic revision name using the selection box, or choose 'Use Text Field' and enter a numeric revision.

  Diffs between and
  Type of Diff should be a

Sort log by:

  ViewVC Help
Powered by ViewVC 1.1.5