Skip to content
  1. Sep 18, 2012
    • Scott McKay's avatar
      Speed up unoptimized serialization · 9a1d9899
      Scott McKay authored 
      Testing : precheckin --full --strict-errors
Reviewer: Fare

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

The unoptimized serialization code was slower than
it needed to be and was a Niagara falls of consing.

The problem is that (map () (curry ...)) is poorly
optimized, which is a shame because it's such a nice
coding style. :-P

The fix is to replace 'map' with iteration (sigh) in
a few critical places.

- Add 'doseq', which chooses between 'dolist' or 'dovector'.
- Make 'serialize-object', 'deserialize-object' and 'object-size'
  use 'doseq' instead of using 'map'.
- Ditto for 'print-text-format'.
- Fix the 'serialize-packed' and 'packed-size' optimizers to
  use 'dolist' or 'dovector' based on whether the field is
  a vector field. This makes the optimized code faster, too.
- Make the optimized serializer generators pass in 'vectorp'
  so that the wire-format optimizers can do a better job.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@562921 f8382938-511b-0410-9cdd-bb47b084005c
      9a1d9899
  3. Sep 12, 2012
    • Scott McKay's avatar
      Fix some user-reported bugs · 5dc5d8d1
      Scott McKay authored 
      Testing : precheckin --full --strict-errors
Reviewer: Sergey V, Shaun M

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

I had made a "policy decision" that the package created
by importing a .proto file would (:use :common-lisp). In
practice, this turned out to be a mistake. Sergey suggested
a fix that I think is correct, so in it goes.

Shaun noticed that the optimized 'serialize-object' and
'object-size' methods didn't quite implement what they were
supposed to for optional boolean fields whose value was never
supplied. Fix the optimized methods to implement the same
(correct) semantics as the unoptimized ones.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@561714 f8382938-511b-0410-9cdd-bb47b084005c
      5dc5d8d1
  5. Aug 31, 2012
    • Scott McKay's avatar
      Add type aliases to CL-Protobufs · a732dcb3
      Scott McKay authored 
      Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

Add type aliases to CL-Protobufs. It's a Lisp-only
feature that allows you define a new Protobufs type in
terms of a Lisp type, a serializer and a deserializer.
 - Add a new model class, 'protobuf-type-alias'.
 - Extend schemas and messages so that they can hold
   a set of type aliases; add 'find-type-alias'.
 - Add 'define-type-alias'.
 - Factor out 'lisp-type-to-protobuf-type' from
   'clos-type-to-protobuf-type', I need it for the
   'define-type-alias' macro.
 - Fix the schema printer to print 'define-type-alias'
   for .lisp schemas, and to add a comment describing
   the type alias in .proto schemas.
 - Extend the wire format (de)serializer to know
   about type aliases; it just (de)serializes the
   Lisp type as the Protobufs primitive type using
   the provided (de)serializer functions. There's
   zero cost to this if you don't use aliases.
 - Update the optimized serialization generation.
 - Similarly extend the text format (de)serializer
   to know about type aliases.
 - Change an example to use a type alias for 'date'.
 - Update the documentation.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559861 f8382938-511b-0410-9cdd-bb47b084005c
      a732dcb3
  7. Aug 29, 2012
  9. Jun 20, 2012
    • Scott McKay's avatar
      Random things discovered while working on Stubby support... · 5ae2da97
      Scott McKay authored 
       - Make the 'find-xxx' support searching "relative to" another namespace.
 - There was a bug in the non-optimized deserializer when deserializing
   a repeated slot into a vector; create a stretchy vector on demand
   if it's needed.
 - 'define-extends' should wrap 'eval-when' around the generated 'defsetf'
   forms so that they are visible at compile time.
 - Fix a formatting bug in the export list in the Lisp printer.
 - Straighten of the ASDF declaration for the tests.
 - Add the Protobufs test suite to QRes, to keep things honest.

Passes 'precheckin' with the new Protobufs unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@550053 f8382938-511b-0410-9cdd-bb47b084005c
      5ae2da97
  11. Jun 13, 2012
  13. Jun 11, 2012
  15. Jun 01, 2012
  17. May 25, 2012
    • Scott McKay's avatar
      Fix a few more things discovered by the tests: · 67fc7299
      Scott McKay authored 
       - 'reinitialize-slot' didn't quite work.
 - Add a geodata example that uses vectors for repeated fields,
   which uncovered a bug in the optimized deserializers.
 - Importing the geodata CLOS classes revealed a bug in default
   handling when the default is provided only in 'defclass'.
 - Fix the knock-on bug in deserialization and the optimized
   (de)serialization caused the above.
 - Add tighter types in 'decode-uint32' and 'decode-uint64'.

Passes 'precheckin'. Even with the new unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545865 f8382938-511b-0410-9cdd-bb47b084005c
      67fc7299
  19. May 23, 2012
  21. May 22, 2012
    • Scott McKay's avatar
      Add more stuff to the test suite, and fix what it uncovered. · b6317f29
      Scott McKay authored 
       - Behavior of ':default xxx'/[default=xxx] was not stable,
   so add an "empty default" marker and respect it during
   parsing and printing.
 - 'find-option' now returns a third "found-p" value, useful
   for the better handling of defaults.
 - Don't print slot readers/writers if they have the same name
   as what would be (silently) generated by the macrology.
 - Don't print 'import descriptor.proto' lines if there are
   no 'lisp_xxx' options in the generated code.
 - Add 'clear-field' to the API; add the functions it needs.
 - Be more careful about how :conc-name interacts with the
   generation of readers/writers.
 - When parsing from a .proto file, force the conc-name to
   be "" unless requested explicitly not to. This has the
   effect of generating readers/writers for all the fields
   of imported Protobufs messages, which I think is probably
   the desired behavior.
 - Add 'make-byte-vector' for convenience.

Passes 'precheckin'. Even with the new unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545177 f8382938-511b-0410-9cdd-bb47b084005c
      b6317f29
  23. May 16, 2012
    • Scott McKay's avatar
      At Sergey's request, make some of the names better: · a31c828e
      Scott McKay authored 
       - 'define-proto' -> 'define-schema'
 - 'protobuf' (class name) -> 'protobuf-schema'
 - 'find-protobuf' -> 'find-schema'
 - 'parse-protobuf-from-file' -> 'parse-schema-from-file'
 - 'parse-protobuf-from-stream' -> 'parse-schema-from-file'
 - 'write-protobuf' -> 'write-schema'
 - 'write-protobuf-as' -> 'write-schema-as'
 - 'ensure-all-protobufs' -> 'ensure-all-protobufs'
 - 'ensure-protobuf' -> 'ensure-protobuf'
 - 'protobuf-upgradable' -> 'schema-upgradable'
 - 'protobufs-equal' -> 'schemas-equal'
 - 'generate-protobuf-schema-for-classes' -> 'generate-schema-for-classes'
 - 'write-protobuf-schema-for-classes' -> 'write-schema-for-classes'

Update the Quake Protobufs schema generator to use the new names.

The Protobufs documentation doesn't say so, but enums can be packed.
 - Update the wire-level protocol to (de)serialize packed enums.
 - Update the serialization code to use the new wire functions.

Passes 'precheckin'.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544299 f8382938-511b-0410-9cdd-bb47b084005c
      a31c828e
    • Scott McKay's avatar
      Tie up a few loose ends with 'unittests.proto'; there are still a few · a13fca91
      Scott McKay authored 
      edge case problems, but it's good enough to continue the Stubby work.
The critical .proto files (descriptor.proto, etc) all appear to work.

More precise mapping of (mainly integer) types from .lisp to .proto files
so that we can a truly exact model once we compile the .lisp file. It may
not make any difference, but better safe than sorry.
 - Introduce Lisp type 'int32', 'int64', 'uint32', etc
 - Make Lisp->Protobufs recognize the new Lisp types
 - Make Protobufs schema printers recognize the new types

Fix up the handling of imports and namespaces.
 - Always use truenames to record schemas
 - Fix typo in 'find-enum' that caused it never to search other schemas
 - Record the schema in *all-protobufs* at exactly the right time,
   and simplify 'make-load-form' methods

A few tweaks to (de)serialization.
 - Make the bodies for empty messages not generate compiler warnings
 - Don't make ':packed t' be the default, the user has to say so
 - The user documentation doesn't say so, but booleans can be packed.

Passes 'precheckin'.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544222 f8382938-511b-0410-9cdd-bb47b084005c
      a13fca91
  25. May 11, 2012
  27. May 10, 2012
  29. May 08, 2012
  31. May 07, 2012
  33. May 04, 2012
  35. Apr 24, 2012
  37. Apr 23, 2012
  39. Apr 19, 2012
  41. Apr 11, 2012
  43. Apr 10, 2012
  45. Apr 05, 2012
  47. Apr 04, 2012
  49. Apr 03, 2012
  51. Apr 02, 2012
  53. Apr 01, 2012
  55. Mar 31, 2012
  57. Mar 29, 2012
  59. Mar 27, 2012
  61. Mar 26, 2012
  63. Mar 23, 2012
  65. Mar 22, 2012
  67. Mar 21, 2012
  69. Mar 15, 2012