/[cmucl]/src/code/extfmts.lisp
ViewVC logotype

Log of /src/code/extfmts.lisp

Parent Directory Parent Directory | Revision Log Revision Log


Links to HEAD: (view) (annotate)
Links to unicode-snapshot-2009-05: (view) (annotate)
Sticky Tag:

Revision 1.2.4.3.2.18 - (view) (annotate) - [select for diffs]
Thu Apr 30 18:52:43 2009 UTC (4 years, 11 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
CVS Tags: unicode-snapshot-2009-05
Changes since 1.2.4.3.2.17: +54 -20 lines
Diff to previous 1.2.4.3.2.17 , to branch point 1.2.4.3
Update from Paul to make the external formats that use invert-tables
uses tries instead of a hash table.

code/extfmts.lisp:
o Change (unsigned-byte 31) to (unsigned-byte 21).  (Should probably
  add a codepoint deftype for this.)
o Use a trie instead of a hash-table for the invert-table stuff
o Fix a typo in a comment.

pcl/simple-streams/external-formats/iso8859-2.lisp:
pcl/simple-streams/external-formats/macroman.lisp:
o Use a trie

Revision 1.2.4.3.2.17 - (view) (annotate) - [select for diffs]
Fri Apr 24 19:18:12 2009 UTC (4 years, 11 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.16: +10 -9 lines
Diff to previous 1.2.4.3.2.16 , to branch point 1.2.4.3
Use gensyms.

Revision 1.2.4.3.2.16 - (view) (annotate) - [select for diffs]
Fri Apr 24 11:28:45 2009 UTC (4 years, 11 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.15: +8 -8 lines
Diff to previous 1.2.4.3.2.15 , to branch point 1.2.4.3
o Fix typo
o Put local variables onto &aux list.

Revision 1.2.4.3.2.15 - (view) (annotate) - [select for diffs]
Thu Apr 23 18:04:36 2009 UTC (4 years, 11 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.14: +31 -35 lines
Diff to previous 1.2.4.3.2.14 , to branch point 1.2.4.3
A nicer implementation of OCTETS-TO-CHAR from Paul.

Revision 1.2.4.3.2.14 - (view) (annotate) - [select for diffs]
Thu Apr 23 15:10:08 2009 UTC (4 years, 11 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.13: +20 -28 lines
Diff to previous 1.2.4.3.2.13 , to branch point 1.2.4.3
string.lisp:
o Add Paul's SURROGATES-TO-CODEPOINT and remove
  CODEPOINT-FROM-SURROGATES.
o Change SURROGATES to return characters, not numbers.
o Update callers of SURROGATES to match.

extfmts.lisp:
o Update callers of SURROGATES to match.
o Use CODEPOINT to extract the correct codepoint from a string in
  EF-STRING-TO-OCTETS and EF-OCTETS-TO-STRING.

Revision 1.2.4.3.2.13 - (view) (annotate) - [select for diffs]
Wed Apr 22 17:09:43 2009 UTC (4 years, 11 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.12: +34 -7 lines
Diff to previous 1.2.4.3.2.12 , to branch point 1.2.4.3
Update OCTETS-TO-CHAR so that it can return codepoints outside the
BMP.  In this case, the first char returned is the high surrogate
value.  A subsequent call returns the low surrogate value.

This is done by making the state be a cons whose car is for
OCTETS-TO-CHAR for its own state and whose cdr is the state for the
external format.

(Idea based on a suggestion by Paul.)

Revision 1.2.4.3.2.12 - (view) (annotate) - [select for diffs]
Thu Apr 16 20:14:14 2009 UTC (5 years ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.11: +2 -10 lines
Diff to previous 1.2.4.3.2.11 , to branch point 1.2.4.3
Fix up comment.  External formats now do really work on code points,
not code units.  The conversion from code points to code units (and
vice versa) is done at a higher level.

Revision 1.2.4.3.2.11 - (view) (annotate) - [select for diffs]
Tue Apr 14 21:10:02 2009 UTC (5 years ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.10: +33 -12 lines
Diff to previous 1.2.4.3.2.10 , to branch point 1.2.4.3
pcl/simple-stream/external-formats/utf-8.lisp:
o Revert to the previous version where the UTF-8 external format
  produces full 21-bit codepoints.

code/extfmts.lisp:
o Modify EF-STRING-TO-OCTETS to process code points and convert them
  to code units to be stored in our strings.
o Modify EF-OCTETS-TO-STRING to convert code units from the string to
  codepoints for processing by the external format.

These need more work, especially with respect to Lisp
characters/codeunits, but utf-8 appears to be working fine with
surrogate pairs.

Revision 1.2.4.3.2.10 - (view) (annotate) - [select for diffs]
Tue Apr 14 02:15:24 2009 UTC (5 years ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.9: +34 -6 lines
Diff to previous 1.2.4.3.2.9 , to branch point 1.2.4.3
Add some comments for the macro DEFINE-EXTERNAL-FORMAT.

Revision 1.2.4.3.2.9 - (view) (annotate) - [select for diffs]
Sat Mar 28 13:31:42 2009 UTC (5 years ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.8: +5 -4 lines
Diff to previous 1.2.4.3.2.8 , to branch point 1.2.4.3
Compile the external format.  Use compile-from-stream so we don't
leave fasls lying around.

Revision 1.2.4.3.2.8 - (view) (annotate) - [select for diffs]
Mon Jul 14 14:01:56 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
CVS Tags: unicode-utf16-extfmt-2009-03-27, unicode-utf16-extfmts-pre-sync-2008-11, unicode-utf16-extfmts-sync-2008-12
Changes since 1.2.4.3.2.7: +95 -89 lines
Diff to previous 1.2.4.3.2.7 , to branch point 1.2.4.3
More updates from Paul.

code/extfmts.lisp:
o Fixed bug with shared code between formats
o Built a cache into the ef-macro functions so it doesn't need to call
  find-external-format so often at runtime

code/fd-stream-extfmt.lisp
o Use the changes in code/extfmts

code/fd-stream.lisp:
o Removed all the commented-out code in fd-stream which is duplicated
  in fd-stream-extfmt.

Revision 1.2.4.3.2.7 - (view) (annotate) - [select for diffs]
Wed Jul 9 15:52:12 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.6: +8 -4 lines
Diff to previous 1.2.4.3.2.6 , to branch point 1.2.4.3
code/extfmts.lisp:
o Bind *DEFAULT-EXTERNAL-FORMAT* to :iso8859-1 when compiling the
  new external format code.  Then messages from the compiler at least
  have a chance of getting printed.
o Removed *compile-verbose*, *compile-progress*, and *gc-verbose*,
  since the compiler messages are working now.  (Should we leave them
  in?)

pcl/simple-streams/external-formats/utf-8.lisp
o Revert back to previous version, without LOCALLY.

Revision 1.2.4.3.2.6 - (view) (annotate) - [select for diffs]
Tue Jul 8 16:09:06 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.5: +6 -2 lines
Diff to previous 1.2.4.3.2.5 , to branch point 1.2.4.3
Turn off *compile-verbose*, *compile-progress*, and *gc-verbose* to
minimize output messages when compiling the external format.  There's
a problem if COMPILE wants to produce output and the external format
isn't completely setup yet.  (Seems only to be a problem when you
change *default-external-format*.)

This is a workaround.  There ought to be a better solution.  This
change doesn't solve every issue since compiler notes are still output
sometimes.

Revision 1.2.4.3.2.5 - (view) (annotate) - [select for diffs]
Sat Jul 5 12:37:42 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.4: +58 -16 lines
Diff to previous 1.2.4.3.2.4 , to branch point 1.2.4.3
More updates from Paul.  fd-stream-extfmt.lisp actually implements the
external formats which now work.

Cross-compile works fine.

code/fd-stream-extfmt.lisp:
o New file implementing external formats

tools/worldcom.lisp:
o Compile extfmts.lisp before fd-stream, since fd-stream uses some
  macros from extfmts.
o Compile fd-stream-extfmt

tools/worldload.lisp:
o Load fd-stream-extfmt at the end.  (Can't load it as part of
  kernel.core.  Not enough is set up yet.)

code/extfmts.lisp:
o Avoid loading files, etc., early in the boot sequence
o Add INVERT-TABLE function needed by some formats.

code/fd-stream.lisp:
o Some cleanups (I think)
o Fix EOF handling

Revision 1.2.4.3.2.4 - (view) (annotate) - [select for diffs]
Wed Jul 2 14:53:44 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.3: +3 -3 lines
Diff to previous 1.2.4.3.2.3 , to branch point 1.2.4.3
code/lispinit.lisp:
o Revert previous change, preserving order of initialization.

Changes from Paul to allow building of the new code from non-unicode
version:

code/extfmts.lisp
code/fd-stream.lisp

Revision 1.2.4.3.2.3 - (view) (annotate) - [select for diffs]
Wed Jul 2 02:35:58 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.2: +2 -2 lines
Diff to previous 1.2.4.3.2.2 , to branch point 1.2.4.3
Oops.  Fix typo (missing paren).

Revision 1.2.4.3.2.2 - (view) (annotate) - [select for diffs]
Wed Jul 2 01:27:09 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3.2.1: +3 -2 lines
Diff to previous 1.2.4.3.2.1 , to branch point 1.2.4.3
Oops.  Don't know how to read #\U+FFFD yet.  Use (code-char #xfffd)
instead.

Revision 1.2.4.3.2.1 - (view) (annotate) - [select for diffs]
Wed Jul 2 01:22:07 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-extfmt-branch
Changes since 1.2.4.3: +178 -71 lines
Diff to previous 1.2.4.3
More external format support from Paul Foley.

To get external format support I think you need to add :extfmts to
*features*.  But you can't bootstrap with that feature yet.

Initial support for pathname translations to so that namestrings can
be converted to an appropriate format before being given to the OS.

Many, many new external formats added.

These changes are all on their own branch for now, until the bootstrap
issue is resolved.  And also so we don't lose these changes from Paul.

Revision 1.2.4.3 - (view) (annotate) - [select for diffs]
Mon Jun 23 15:03:31 2008 UTC (5 years, 9 months ago) by rtoy
Branch: unicode-utf16-branch
CVS Tags: unicode-utf16-char-support-2009-03-25, unicode-utf16-char-support-2009-03-26, unicode-utf16-sync-2008-07, unicode-utf16-sync-2008-09, unicode-utf16-sync-2008-11, unicode-utf16-sync-2008-12, unicode-utf16-sync-label-2009-03-16
Branch point for: unicode-utf16-extfmt-branch
Changes since 1.2.4.2: +66 -39 lines
Diff to previous 1.2.4.2 , to branch point 1.2 , to next main 1.42
Sync to HEAD branch.

Revision 1.2.4.2 - (view) (annotate) - [select for diffs]
Thu Jun 19 21:08:17 2008 UTC (5 years, 10 months ago) by rtoy
Branch: unicode-utf16-branch
Changes since 1.2.4.1: +3 -7 lines
Diff to previous 1.2.4.1 , to branch point 1.2
Merge changes from HEAD for the ext-formats search-list change.

Revision 1.2.4.1 - (view) (annotate) - [select for diffs]
Thu Jun 19 03:30:44 2008 UTC (5 years, 10 months ago) by rtoy
Branch: unicode-utf16-branch
Changes since 1.2: +252 -148 lines
Diff to previous 1.2
Merge changes from HEAD to the unicode-utf16 branch.

Revision 1.2 - (view) (annotate) - [select for diffs]
Wed Oct 31 14:37:38 2007 UTC (6 years, 5 months ago) by rtoy
Branch: MAIN
CVS Tags: release-19d, release-19e, release-19e-base, release-19e-pre1, release-19e-pre2, snapshot-2007-11, snapshot-2007-12, snapshot-2008-01, snapshot-2008-02, snapshot-2008-03, snapshot-2008-04, snapshot-2008-05, snapshot-2008-06, unicode-utf16-base, unicode-utf16-string-support
Branch point for: release-19e-branch, unicode-utf16-branch
Changes since 1.1: +23 -19 lines
Diff to previous 1.1
Update from Paul Foley.

o Disable package errors when loading up external formats.
o A minor patch allowing string-to-octets and vice versa to write into
  a preallocated array (though they might still allocate a bigger one
  if necessary),
o Fix up any confusion between simple-base-string and simple-string so
  that nothing breaks when/if they're not the same.

Revision 1.1 - (view) (annotate) - [select for diffs]
Thu Oct 25 15:17:07 2007 UTC (6 years, 5 months ago) by rtoy
Branch: MAIN
Import Paul Foley's external-formats support.

New files:
o code/extfmts.lisp
o pcl/simple-streams/external-formats/iso8859-1.lisp
o pcl/simple-streams/external-formats/void.lisp

code/exports.lisp:
o Export the new symbols STRING-TO-OCTETS, OCTETS-TO-STRING,
  *DEFAULT-EXTERNAL-FORMAT*, ENCODE-STRING, and DECODE-STRING from the
  STREAM package
o Make the symbols in the EXT package too.

pcl/simple-streams/internal.lisp:
o Move the implementation of STRING-TO-OCTETS and friends to a new
  file (extfmts.lisp).

pcl/simple-streams/external-formats/utf-8.lisp:
o New implementation.

tools/make-main-dist.sh:
o Create new target directory to hold external formats
o Copy all of the external formats to the new directory.

tools/pclcom.lisp:
o Compile new code

tools/worldcom.lisp:
o Compile code/extfmts.lisp

tools/worldload.lisp:
o Load code/extfmts.lisp

This form allows you to request diffs between any two revisions of this file. For each of the two "sides" of the diff, select a symbolic revision name using the selection box, or choose 'Use Text Field' and enter a numeric revision.

  Diffs between and
  Type of Diff should be a

Sort log by:

  ViewVC Help
Powered by ViewVC 1.1.5