Skip to content
  1. Sep 18, 2012
    • Scott McKay's avatar
      Speed up unoptimized serialization · 9a1d9899
      Scott McKay authored 
      Testing : precheckin --full --strict-errors
Reviewer: Fare

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

The unoptimized serialization code was slower than
it needed to be and was a Niagara falls of consing.

The problem is that (map () (curry ...)) is poorly
optimized, which is a shame because it's such a nice
coding style. :-P

The fix is to replace 'map' with iteration (sigh) in
a few critical places.

- Add 'doseq', which chooses between 'dolist' or 'dovector'.
- Make 'serialize-object', 'deserialize-object' and 'object-size'
  use 'doseq' instead of using 'map'.
- Ditto for 'print-text-format'.
- Fix the 'serialize-packed' and 'packed-size' optimizers to
  use 'dolist' or 'dovector' based on whether the field is
  a vector field. This makes the optimized code faster, too.
- Make the optimized serializer generators pass in 'vectorp'
  so that the wire-format optimizers can do a better job.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@562921 f8382938-511b-0410-9cdd-bb47b084005c
      9a1d9899
  3. Jun 18, 2012
  5. Jun 13, 2012
  7. Jun 11, 2012
  9. Jun 01, 2012
  11. May 25, 2012
  13. May 23, 2012
  15. May 22, 2012
    • Scott McKay's avatar
      Add more stuff to the test suite, and fix what it uncovered. · b6317f29
      Scott McKay authored 
       - Behavior of ':default xxx'/[default=xxx] was not stable,
   so add an "empty default" marker and respect it during
   parsing and printing.
 - 'find-option' now returns a third "found-p" value, useful
   for the better handling of defaults.
 - Don't print slot readers/writers if they have the same name
   as what would be (silently) generated by the macrology.
 - Don't print 'import descriptor.proto' lines if there are
   no 'lisp_xxx' options in the generated code.
 - Add 'clear-field' to the API; add the functions it needs.
 - Be more careful about how :conc-name interacts with the
   generation of readers/writers.
 - When parsing from a .proto file, force the conc-name to
   be "" unless requested explicitly not to. This has the
   effect of generating readers/writers for all the fields
   of imported Protobufs messages, which I think is probably
   the desired behavior.
 - Add 'make-byte-vector' for convenience.

Passes 'precheckin'. Even with the new unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545177 f8382938-511b-0410-9cdd-bb47b084005c
      b6317f29
  17. May 16, 2012
    • Scott McKay's avatar
      Tie up a few loose ends with 'unittests.proto'; there are still a few · a13fca91
      Scott McKay authored 
      edge case problems, but it's good enough to continue the Stubby work.
The critical .proto files (descriptor.proto, etc) all appear to work.

More precise mapping of (mainly integer) types from .lisp to .proto files
so that we can a truly exact model once we compile the .lisp file. It may
not make any difference, but better safe than sorry.
 - Introduce Lisp type 'int32', 'int64', 'uint32', etc
 - Make Lisp->Protobufs recognize the new Lisp types
 - Make Protobufs schema printers recognize the new types

Fix up the handling of imports and namespaces.
 - Always use truenames to record schemas
 - Fix typo in 'find-enum' that caused it never to search other schemas
 - Record the schema in *all-protobufs* at exactly the right time,
   and simplify 'make-load-form' methods

A few tweaks to (de)serialization.
 - Make the bodies for empty messages not generate compiler warnings
 - Don't make ':packed t' be the default, the user has to say so
 - The user documentation doesn't say so, but booleans can be packed.

Passes 'precheckin'.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544222 f8382938-511b-0410-9cdd-bb47b084005c
      a13fca91
  19. May 15, 2012
  21. May 14, 2012
    • Scott McKay's avatar
      A few more model fixes to fully support google/protobuf/unittest.proto · 4157162a
      Scott McKay authored 
      (I love this file, it's got one of everything.)

 - The printer should not print extended fields in any message
   that further extends an extended message.
 - If parsing or importing a .proto file needs a Lisp package that
   does not exist, go ahead and create it.
 - Importing now tracks all the schemas imported by the "parent",
   this so that we can search namespaces thoroughly.
 - It looks like 'define-extend' can accept groups. For a deprecated
   feature, it sure is a pain in the neck.
 - Make 'find-message' and 'find-enum' search all namespaces.
 - Print escaped strings, just for completeness.
 - Annotate a couple of common options with their types.

Passes 'precheckin'. Passes my by-hand tests and is epsilon away from
handling the worst .proto file around.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@543912 f8382938-511b-0410-9cdd-bb47b084005c
      4157162a
  23. May 13, 2012
    • Scott McKay's avatar
      Get google/protobuf/unittest.proto working, insofar as it parses · 3d3775ce
      Scott McKay authored 
      completely and produces what appears to be a correct model. Next
up, ensure (de)serialization works with this complex beast, which
sets the stage for running the (very complete) Google tests.

 - Be much more careful to preserve type information in option
   values and field defaults, both in the macros and the parser.
   Make the slots in their model objects be untyped.
 - This makes it easier for the Proto and Lisp printers to produce
   exactly the right output, so fix them, too. Be extra careful
   with booleans.
 - Fix the printer to recognize more of the standard options so
   that their values get printed with the correct type.
 - Learn to parse more complicated numeric and integer constants.
 - Learn to parse escaped strings.
 - Learn to parse comma-separated options.
 - Learn to parse extensions with no 'to' specifier.
 - Fix the places that used (deprecated) :single instead of :float.
 - Proto enum values always map to Lisp keywords, so don't futz
   with packages for them.
 - Add some more doc strings.
 - Add some more examples-that-will-become-tests.

Passes 'precheckin'. Passes my by-hand tests.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@543727 f8382938-511b-0410-9cdd-bb47b084005c
      3d3775ce
  25. May 11, 2012
  27. May 09, 2012
  29. May 08, 2012
  31. May 04, 2012
  33. Apr 26, 2012
  35. Apr 25, 2012
  37. Apr 11, 2012
  39. Apr 05, 2012
  41. Apr 03, 2012
  43. Apr 02, 2012
  45. Apr 01, 2012
  47. Mar 29, 2012
  49. Mar 15, 2012
  51. Mar 14, 2012
  53. Mar 12, 2012
  55. Mar 07, 2012
  57. Mar 06, 2012