Skip to content
  1. Jan 02, 2013
  2. Dec 17, 2012
  3. Nov 07, 2012
    • Francois-Rene Rideau's avatar
      Improve schema generation performance on large schemas · ae041468
      Francois-Rene Rideau authored
      Avoid SUBTYPEP during macroexpansion, it can be very expensive on SBCL.
      Actually recognize LIST-OF:LIST-OF constructs.
      In DEFINE-SCHEMA, don't nest the DEFMETHODs inside the LET,
      this also can cause SBCL's control flow analysis to blow up.
      
      Work done with Steven Spitz for QPX.
      
      Tested: (asdf:test-system :cl-protobufs)
      ae041468
  4. Sep 14, 2012
    • Sergey Vasilyev's avatar
      Tsunami framework + use tsunami in customer profiles · 48aa0eee
      Sergey Vasilyev authored
      
       Testing : precheckin --full (both against Oracle and against megastore for customers)
       Reviewer: eschwartz
      
       JTB impact: n
       Ops impact: n
       Security impact: n
      
       Change to config                        : n
       Change to XML schema                    : Y (XMLREQ bug 120820)
       Change to DB schema                     : Y (DBREQ bug 119801)
       Change to transport (timeouts, headers) : n
       Any change (or new use) of OAQs         : n
       Change to inter-component transactions  : sort of (use megastore in development-only environments)
       Depends on any other checkin / bug      : n
      
       Tests that will verify : tsunami and megastore unit tests + existing customer profile tests
      
      Tsunami is a new layer that sits between business logic code and the
      persistence logic. The main manifestation of this layer is
      define-domain-class macro that defines an interface to a domain
      class. You can then define how to perist in various implementations,
      e.g. using define-quake-db-class-bindings for quake and
      define-proto-binding & define-megastore-proto-binding for megastore.
      
      Tsunami framework is still a work in progress, and there is a bunch of
      outstanding tasks/issues.
      
      As the first application, I refactored customer profile code to use
      the tsunami framework. This enables storing of customer profiles in
      megastore (or other data stores in the future), while everything else
      is stored in Oracle.
      
      Here is the list of other miscellaneous/harder-to-understand changes:
      
      - I changed @id attribute of ContactInfo objects to use the actual
        numeric ID, rather than passing it through encode-record-locator (in
        qres-dev schema only). This required changing XML schema type from
        typeBasicLocatorString to typeBasicID.
      
      - Drop quake/relationships.lisp. Use a more straightforward way to
        associated ADO agencies to customer profiles.
      
      - Customer usernames are now stored as lowercase strings (this means
        that I don't need a separate field in megastore protobuf to
        implement case-insensitive index).
      
      - agency-id permission attribute is now the agency-key (serialized to
        string), not the agency's record ID. This way you don't need to load
        agency to do permission check when all you have is the agency-key
      
      - moved agency-key from core/agencies.lisp to
        lisp/core/agency-types.lisp, so that agency-key type can be used in
        permissions
      
      - add when* test handler that takes a random Lisp expression (as
        opposed to the existing when that takes a variable name and looks at
        whether it is empty)
      
      - remove cc-deleted-error -- there is no such thing as deleting
        cc-info
      
      - refactored cc-information, so that it is used as a value object (ie
        a mostly-immutable object with no identity) in the application. When
        saving to Oracle database, it is saved in the central
        tbl_cc_information table with appropriately populated ownership
        columns (pointing to the customer, journey,
        accountable_doc_container, etc)
      
      - drop unneeded tbl_cc_transaction columns that were caching cc_info
        values
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@562275 f8382938-511b-0410-9cdd-bb47b084005c
      48aa0eee
  5. Sep 10, 2012
    • Scott McKay's avatar
      Add a more readable syntax for enum and field indices · 58c795d8
      Scott McKay authored
      Testing : precheckin --full --strict-errors
      Reviewer: Fare (please)
      
      JTB impact: No
      Ops impact: No
      
      Change to config                        : No
      Change to XML schema                    : No
      Change to DB schema                     : No
      Change to transport (timeouts, headers) : No
      Any change (or new use) of OAQs         : No
      Change to inter-component transactions  : No
      Depends on any other checkin / bug      : No
      
      Tests that will verify:
      
      I extended the CL-Protobufs tests
      
      Description:
      
      Add a more readable syntax for enum and field indices.
      
      For define-enum, it was (name value).
        Allow (name :index value).
      
      For define-message, it was ((name index) ...).
        Allow (name :index index ...).
        Complain if both forms appear in the same field.
      
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@561136 f8382938-511b-0410-9cdd-bb47b084005c
      58c795d8
  6. Sep 04, 2012
    • Scott McKay's avatar
      Add stub support for streaming RPC APIs. · 5293ada2
      Scott McKay authored
      Testing : precheckin --full --strict-errors
      Reviewer: Fare (please)
      
      JTB impact: No
      Ops impact: No
      
      Change to config                        : No
      Change to XML schema                    : No
      Change to DB schema                     : No
      Change to transport (timeouts, headers) : No
      Any change (or new use) of OAQs         : No
      Change to inter-component transactions  : No
      Depends on any other checkin / bug      : No
      
      Tests that will verify:
      
      I extended the CL-Protobufs tests
      
      Description:
      
      Add support for a 'streams' type in Protobufs RPC methods.
       - Add a few slots to 'protobufs-method'
       - Add '&key streams' to the method arglists in 'define-service'
       - Extend the parser to handle "streams" in addition to the
         'streams_type' option
       - Extend the printer
       - Change a test to use a 'streams' type
       - While we're in the neighborhood, allow an optional '=>'
         between the input and output methods in 'define-service';
         it makes it a bit more readable
       - Update the documen...
      5293ada2
  7. Aug 31, 2012
    • Scott McKay's avatar
      Add type aliases to CL-Protobufs · a732dcb3
      Scott McKay authored
      Testing : precheckin --full --strict-errors
      Reviewer: Fare (please)
      
      JTB impact: No
      Ops impact: No
      
      Change to config                        : No
      Change to XML schema                    : No
      Change to DB schema                     : No
      Change to transport (timeouts, headers) : No
      Any change (or new use) of OAQs         : No
      Change to inter-component transactions  : No
      Depends on any other checkin / bug      : No
      
      Tests that will verify:
      
      The CL-Protobufs tests
      
      Description:
      
      Add type aliases to CL-Protobufs. It's a Lisp-only
      feature that allows you define a new Protobufs type in
      terms of a Lisp type, a serializer and a deserializer.
       - Add a new model class, 'protobuf-type-alias'.
       - Extend schemas and messages so that they can hold
         a set of type aliases; add 'find-type-alias'.
       - Add 'define-type-alias'.
       - Factor out 'lisp-type-to-protobuf-type' from
         'clos-type-to-protobuf-type', I need it for the
         'define-type-alias' macro.
       - Fix the schema printer to print 'define-type-alias'
         for .lisp schemas, and to add a comment describing
         the type alias in .proto schemas.
       - Extend the wire format (de)serializer to know
         about type aliases; it just (de)serializes the
         Lisp type as the Protobufs primitive type using
         the provided (de)serializer functions. There's
         zero cost to this if you don't use aliases.
       - Update the optimized serialization generation.
       - Similarly extend the text format (de)serializer
         to know about type aliases.
       - Change an example to use a type alias for 'date'.
       - Update the documentation.
      
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559861 f8382938-511b-0410-9cdd-bb47b084005c
      a732dcb3
  8. Aug 28, 2012
  9. Aug 27, 2012
  10. Aug 24, 2012
  11. Jul 03, 2012
  12. Jun 28, 2012
  13. Jun 27, 2012
  14. Jun 26, 2012
  15. Jun 20, 2012
    • Scott McKay's avatar
      Random things discovered while working on Stubby support... · 5ae2da97
      Scott McKay authored
       - Make the 'find-xxx' support searching "relative to" another namespace.
       - There was a bug in the non-optimized deserializer when deserializing
         a repeated slot into a vector; create a stretchy vector on demand
         if it's needed.
       - 'define-extends' should wrap 'eval-when' around the generated 'defsetf'
         forms so that they are visible at compile time.
       - Fix a formatting bug in the export list in the Lisp printer.
       - Straighten of the ASDF declaration for the tests.
       - Add the Protobufs test suite to QRes, to keep things honest.
      
      Passes 'precheckin' with the new Protobufs unit tests in place.
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@550053 f8382938-511b-0410-9cdd-bb47b084005c
      5ae2da97
  16. Jun 18, 2012
  17. Jun 14, 2012
  18. Jun 13, 2012
  19. Jun 11, 2012
  20. Jun 05, 2012
  21. May 25, 2012
  22. May 23, 2012
  23. May 22, 2012
    • Scott McKay's avatar
      Add more stuff to the test suite, and fix what it uncovered. · b6317f29
      Scott McKay authored
       - Behavior of ':default xxx'/[default=xxx] was not stable,
         so add an "empty default" marker and respect it during
         parsing and printing.
       - 'find-option' now returns a third "found-p" value, useful
         for the better handling of defaults.
       - Don't print slot readers/writers if they have the same name
         as what would be (silently) generated by the macrology.
       - Don't print 'import descriptor.proto' lines if there are
         no 'lisp_xxx' options in the generated code.
       - Add 'clear-field' to the API; add the functions it needs.
       - Be more careful about how :conc-name interacts with the
         generation of readers/writers.
       - When parsing from a .proto file, force the conc-name to
         be "" unless requested explicitly not to. This has the
         effect of generating readers/writers for all the fields
         of imported Protobufs messages, which I think is probably
         the desired behavior.
       - Add 'make-byte-vector' for convenience.
      
      Passes 'precheckin'. Even with the new unit tests in place.
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545177 f8382938-511b-0410-9cdd-bb47b084005c
      b6317f29
  24. May 18, 2012
    • Scott McKay's avatar
      Now that Protobufs has a test suite, it found a few things to fix. · 0d1611ab
      Scott McKay authored
       - Don't generate warnings for anonymous enums, they're harmless
         and ubiquitous.
       - 'member' types where are all the members are keywords, is a symbol,
         which got converted to a Protobufs 'string'. Wrong.
       - If a field has an enum type, the (Lisp) default value should be
         treated as a keyword.
       - When parsing a .proto file or using the 'define-xxx' macros, any
         options that are handled specially should be trimmed from the
         options list so that they don't get printed twice.
       - Add 'remove-options' for the above.
       - Make the options printer be more type-aware.
       - Clean up examples.lisp
       - 'schemas-equal' can ignore the schema name/class if they're null.
      
      Passes 'precheckin'. Even with the new unit tests in place.
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544737 f8382938-511b-0410-9cdd-bb47b084005c
      0d1611ab
  25. May 16, 2012
    • Scott McKay's avatar
      At Sergey's request, make some of the names better: · a31c828e
      Scott McKay authored
       - 'define-proto' -> 'define-schema'
       - 'protobuf' (class name) -> 'protobuf-schema'
       - 'find-protobuf' -> 'find-schema'
       - 'parse-protobuf-from-file' -> 'parse-schema-from-file'
       - 'parse-protobuf-from-stream' -> 'parse-schema-from-file'
       - 'write-protobuf' -> 'write-schema'
       - 'write-protobuf-as' -> 'write-schema-as'
       - 'ensure-all-protobufs' -> 'ensure-all-protobufs'
       - 'ensure-protobuf' -> 'ensure-protobuf'
       - 'protobuf-upgradable' -> 'schema-upgradable'
       - 'protobufs-equal' -> 'schemas-equal'
       - 'generate-protobuf-schema-for-classes' -> 'generate-schema-for-classes'
       - 'write-protobuf-schema-for-classes' -> 'write-schema-for-classes'
      
      Update the Quake Protobufs schema generator to use the new names.
      
      The Protobufs documentation doesn't say so, but enums can be packed.
       - Update the wire-level protocol to (de)serialize packed enums.
       - Update the serialization code to use the new wire functions.
      
      Passes 'precheckin'.
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544299 f8382938-511b-0410-9cdd-bb47b084005c
      a31c828e
    • Scott McKay's avatar
      Tie up a few loose ends with 'unittests.proto'; there are still a few · a13fca91
      Scott McKay authored
      edge case problems, but it's good enough to continue the Stubby work.
      The critical .proto files (descriptor.proto, etc) all appear to work.
      
      More precise mapping of (mainly integer) types from .lisp to .proto files
      so that we can a truly exact model once we compile the .lisp file. It may
      not make any difference, but better safe than sorry.
       - Introduce Lisp type 'int32', 'int64', 'uint32', etc
       - Make Lisp->Protobufs recognize the new Lisp types
       - Make Protobufs schema printers recognize the new types
      
      Fix up the handling of imports and namespaces.
       - Always use truenames to record schemas
       - Fix typo in 'find-enum' that caused it never to search other schemas
       - Record the schema in *all-protobufs* at exactly the right time,
         and simplify 'make-load-form' methods
      
      A few tweaks to (de)serialization.
       - Make the bodies for empty messages not generate compiler warnings
       - Don't make ':packed t' be the default, the user has to say so
       - The user documentation doesn't say so, but booleans can be packed.
      
      Passes 'precheckin'.
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544222 f8382938-511b-0410-9cdd-bb47b084005c
      a13fca91
  26. May 14, 2012
    • Scott McKay's avatar
      A few more model fixes to fully support google/protobuf/unittest.proto · 4157162a
      Scott McKay authored
      (I love this file, it's got one of everything.)
      
       - The printer should not print extended fields in any message
         that further extends an extended message.
       - If parsing or importing a .proto file needs a Lisp package that
         does not exist, go ahead and create it.
       - Importing now tracks all the schemas imported by the "parent",
         this so that we can search namespaces thoroughly.
       - It looks like 'define-extend' can accept groups. For a deprecated
         feature, it sure is a pain in the neck.
       - Make 'find-message' and 'find-enum' search all namespaces.
       - Print escaped strings, just for completeness.
       - Annotate a couple of common options with their types.
      
      Passes 'precheckin'. Passes my by-hand tests and is epsilon away from
      handling the worst .proto file around.
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@543912 f8382938-511b-0410-9cdd-bb47b084005c
      4157162a
  27. May 13, 2012
    • Scott McKay's avatar
      Get google/protobuf/unittest.proto working, insofar as it parses · 3d3775ce
      Scott McKay authored
      completely and produces what appears to be a correct model. Next
      up, ensure (de)serialization works with this complex beast, which
      sets the stage for running the (very complete) Google tests.
      
       - Be much more careful to preserve type information in option
         values and field defaults, both in the macros and the parser.
         Make the slots in their model objects be untyped.
       - This makes it easier for the Proto and Lisp printers to produce
         exactly the right output, so fix them, too. Be extra careful
         with booleans.
       - Fix the printer to recognize more of the standard options so
         that their values get printed with the correct type.
       - Learn to parse more complicated numeric and integer constants.
       - Learn to parse escaped strings.
       - Learn to parse comma-separated options.
       - Learn to parse extensions with no 'to' specifier.
       - Fix the places that used (deprecated) :single instead of :float.
       - Proto enum values always map to Lisp keywords, so don't futz
         with packages for them.
       - Add some more doc strings.
       - Add some more examples-that-will-become-tests.
      
      Passes 'precheckin'. Passes my by-hand tests.
      
      
      git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@543727 f8382938-511b-0410-9cdd-bb47b084005c
      3d3775ce
  28. May 10, 2012
  29. May 09, 2012
  30. May 08, 2012
  31. May 07, 2012
  32. May 04, 2012
  33. Apr 30, 2012