- Use 'defparameter' instead of 'defvar' where appropriate.
 - Fix 'defvar' doc strings to distinguish between globals and "thread locals".
 - Avoid using 'nconc'. introduce a new 'appendf' macro instead.
 - Add a comment lamenting the fact that exporting something like 'proto-options'
   also exports the writer '(setf proto-options)'. Fixed in Dylan.

The client-side stub is now CALL-FOO
The server-side stub is now FOO-IMPL

They are interned in package XXX-RPC, where XXX is the name of the
package being used by the rest of the schema.

[proto -> lisp]
1) Do case-sensitive comparisons between the name we have and the name
   we would derive from #'CLASS-NAME->PROTO. Case matters.

2) Do not downcase :name options when we're outputting them to a .lisp
   file. Again, case matters.

3) Preserve names for PROTOBUF-SERVICEs, as necessary.

4) Preserve names, input-names, and output-names for PROTOBUF-METHODs,
   as necessary.

Here we should have been consuming both the thing we would be printing
and the indentation we would use on the next line, but in the case
where the printed thing was nil, we would only consume it, and not the
subsequent indentation. It worked out here, but the mistake could only
be made once, and only at the end.

* Previously, if a field in a .proto file referenced a message in
  another proto file using a different lisp package, the cl-protobufs
  library would silently fail to serialize the field.  A similar
  problem would occur if a message defined later in the file used the
  lisp_name option to override the name generated by cl-protobufs.
  This change fixes these issues and others.
* Add conditions that are signaled when encountering an undefined
  type.
* Delay assigning lisp classes/types to fields and methods until all
  possible forward references have been parsed.
   * This allows the class slot to be unbound, so check for that case
     in print-object methods.
   * Add a test for forward references to messages that override the
     lisp name.
   * Add a test for references to messages and enums defined in
     another proto file with a different lisp package.
   * Change color-wheel-stability test, because it used "string" as
     the input type for an rpc, which seems to be disallowed (although
     I haven't found this documented anywhere).
* Signal errors during parsing for undefined types.
   * Add a test for these errors.  Add assert-error macro to qtest.
* Signal a condition if we are unable to find the definition for a
  field's type during serialization, deserialization, determining an
  object's serialized size, printing text format, parsing text format,
  or generating code for one of the above.
* Remove logic in find-qualified-name that indirects through lisp
  packages.  Proto packages and lisp packages do not necessarily map
  1-to-1.
* Always use the schema's lisp package for any symbols generated when
  parsing proto files.
* When generating lisp code using write-schema-as, set the package to
  the package used in the generated file, so that ~s will print the
  package prefix in the correct circumstances.
* Remove broken proto1 "streams" parsing ("returns" comes before
  "streams" in every example I've found); replace with proto2 syntax.
* In process-imports, the call to find-schema using a pathname was not
  giving the expected result.  Sidestep this issue by using the same
  logic to find the schema as is used earlier in the function.

 Testing : precheckin --full (both against Oracle and against megastore for customers)
 Reviewer: eschwartz

 JTB impact: n
 Ops impact: n
 Security impact: n

 Change to config                        : n
 Change to XML schema                    : Y (XMLREQ bug 120820)
 Change to DB schema                     : Y (DBREQ bug 119801)
 Change to transport (timeouts, headers) : n
 Any change (or new use) of OAQs         : n
 Change to inter-component transactions  : sort of (use megastore in development-only environments)
 Depends on any other checkin / bug      : n

 Tests that will verify : tsunami and megastore unit tests + existing customer profile tests

Tsunami is a new layer that sits between business logic code and the
persistence logic. The main manifestation of this layer is
define-domain-class macro that defines an interface to a domain
class. You can then define how to perist in various implementations,
e.g. using define-quake-db-class-bindings for quake and
define-proto-binding & define-megastore-proto-binding for megastore.

Tsunami framework is still a work in progress, and there is a bunch of
outstanding tasks/issues.

As the first application, I refactored customer profile code to use
the tsunami framework. This enables storing of customer profiles in
megastore (or other data stores in the future), while everything else
is stored in Oracle.

Here is the list of other miscellaneous/harder-to-understand changes:

- I changed @id attribute of ContactInfo objects to use the actual
  numeric ID, rather than passing it through encode-record-locator (in
  qres-dev schema only). This required changing XML schema type from
  typeBasicLocatorString to typeBasicID.

- Drop quake/relationships.lisp. Use a more straightforward way to
  associated ADO agencies to customer profiles.

- Customer usernames are now stored as lowercase strings (this means
  that I don't need a separate field in megastore protobuf to
  implement case-insensitive index).

- agency-id permission attribute is now the agency-key (serialized to
  string), not the agency's record ID. This way you don't need to load
  agency to do permission check when all you have is the agency-key

- moved agency-key from core/agencies.lisp to
  lisp/core/agency-types.lisp, so that agency-key type can be used in
  permissions

- add when* test handler that takes a random Lisp expression (as
  opposed to the existing when that takes a variable name and looks at
  whether it is empty)

- remove cc-deleted-error -- there is no such thing as deleting
  cc-info

- refactored cc-information, so that it is used as a value object (ie
  a mostly-immutable object with no identity) in the application. When
  saving to Oracle database, it is saved in the central
  tbl_cc_information table with appropriately populated ownership
  columns (pointing to the customer, journey,
  accountable_doc_container, etc)

- drop unneeded tbl_cc_transaction columns that were caching cc_info
  values


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@562275 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Sergey V, Shaun M

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

I had made a "policy decision" that the package created
by importing a .proto file would (:use :common-lisp). In
practice, this turned out to be a mistake. Sergey suggested
a fix that I think is correct, so in it goes.

Shaun noticed that the optimized 'serialize-object' and
'object-size' methods didn't quite implement what they were
supposed to for optional boolean fields whose value was never
supplied. Fix the optimized methods to implement the same
(correct) semantics as the unoptimized ones.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@561714 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

I extended the CL-Protobufs examples

Description:

Add a few missing features that aren't in any documentation.

String literals can look like "foo"<whitespace>"bar".
 - Fix 'parse-string' to handle this.

Option values can be complex structures, not just atoms.
 - If 'parse-proto-option' sees a '{' character after the '=',
   it should use 'parse-text-format' to read the option value.
 - Fix the 'protobuf-option' printer to call 'print-text-format'
   for complex option values.
 - Minor refactoring to the text format parser and printer in
   order to support the above.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@560639 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

I extended the CL-Protobufs tests

Description:

Add support for a 'streams' type in Protobufs RPC methods.
 - Add a few slots to 'protobufs-method'
 - Add '&key streams' to the method arglists in 'define-service'
 - Extend the parser to handle "streams" in addition to the
   'streams_type' option
 - Extend the printer
 - Change a test to use a 'streams' type
 - While we're in the neighborhood, allow an optional '=>'
   between the input and output methods in 'define-service';
   it makes it a bit more readable
 - Update the documentation



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@560147 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

Add type aliases to CL-Protobufs. It's a Lisp-only
feature that allows you define a new Protobufs type in
terms of a Lisp type, a serializer and a deserializer.
 - Add a new model class, 'protobuf-type-alias'.
 - Extend schemas and messages so that they can hold
   a set of type aliases; add 'find-type-alias'.
 - Add 'define-type-alias'.
 - Factor out 'lisp-type-to-protobuf-type' from
   'clos-type-to-protobuf-type', I need it for the
   'define-type-alias' macro.
 - Fix the schema printer to print 'define-type-alias'
   for .lisp schemas, and to add a comment describing
   the type alias in .proto schemas.
 - Extend the wire format (de)serializer to know
   about type aliases; it just (de)serializes the
   Lisp type as the Protobufs primitive type using
   the provided (de)serializer functions. There's
   zero cost to this if you don't use aliases.
 - Update the optimized serialization generation.
 - Similarly extend the text format (de)serializer
   to know about type aliases.
 - Change an example to use a type alias for 'date'.
 - Update the documentation.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559861 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559296 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@558959 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@558792 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@552185 f8382938-511b-0410-9cdd-bb47b084005c

Passes 'precheckin' with the new Protobufs unit tests in place.
CL-Stubby tests pass in Google3 environment.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@551401 f8382938-511b-0410-9cdd-bb47b084005c

 - Make the 'find-xxx' support searching "relative to" another namespace.
 - There was a bug in the non-optimized deserializer when deserializing
   a repeated slot into a vector; create a stretchy vector on demand
   if it's needed.
 - 'define-extends' should wrap 'eval-when' around the generated 'defsetf'
   forms so that they are visible at compile time.
 - Fix a formatting bug in the export list in the Lisp printer.
 - Straighten of the ASDF declaration for the tests.
 - Add the Protobufs test suite to QRes, to keep things honest.

Passes 'precheckin' with the new Protobufs unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@550053 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@548806 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@548441 f8382938-511b-0410-9cdd-bb47b084005c

some things that needed to be addressed:
 - We need to do a more thorough search of namespaces so that
   qualified references to imported messages, etc, work properly.
   It's still not 100%, but it covers most of what I can find.
   - Name mappers now do their best to preserve qualified names.
   - 'find-xxx' methods search the qualified names.
 - Fix the importation process to cooperate better with ASDF, i.e.,
   don't bother re-importing things that have already compiled
   as part of an ASDF compile/load operation.
 - When generating .lisp from .proto, if no package has been
   declared, do it. And fill it with nice exports, too. This
   won't be exactly right, but it'll make it easier to create
   a proper package declaration, if you so desire.
 - Do a better job of generating client- and server-side stubs.

Passes 'precheckin' with the new Protobufs unit tests in place.
And more to the point, the Fortuneseeker server now works as a
Stubby client using my new Google3 http2 and rpc2 libraries.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@548386 f8382938-511b-0410-9cdd-bb47b084005c

 - Use 'defgeneric' with ':method' for things that are meant to
   have just a single method.
 - Lose some unreachable code in the parser.
 - Weak hash tables in SBCL need to use ':weakness :value'.
 - The wire-level stuff can use some more type declarations.
 - Fix a package problem in one of the examples.

Passes 'precheckin' with the new Protobufs unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@547515 f8382938-511b-0410-9cdd-bb47b084005c

 - Better implementation of 'uncamel-case' gives better names.
 - Add an index slot to 'protobuf-method' so that we can more
   easily do automatic service registration.
 - Give a better type to the index slot in 'protobuf-message'
   and 'protobuf-enum'.
 - Parser and macros now assign an index to each service method.
 - Define 'find-service' and 'find-method' for use by Stubby.

Passes 'precheckin' with the new Protobufs unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545964 f8382938-511b-0410-9cdd-bb47b084005c

 - 'reinitialize-slot' didn't quite work.
 - Add a geodata example that uses vectors for repeated fields,
   which uncovered a bug in the optimized deserializers.
 - Importing the geodata CLOS classes revealed a bug in default
   handling when the default is provided only in 'defclass'.
 - Fix the knock-on bug in deserialization and the optimized
   (de)serialization caused the above.
 - Add tighter types in 'decode-uint32' and 'decode-uint64'.

Passes 'precheckin'. Even with the new unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545865 f8382938-511b-0410-9cdd-bb47b084005c

 some additions to the macrology and to the (de)serialization
 code. No runtime performance cost to being able to have things
 both ways.
Fix a bug that caused unprocessed options to get dropped.
Improve some of the unit tests.

Passes 'precheckin'. Even with the new unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545367 f8382938-511b-0410-9cdd-bb47b084005c

 - Behavior of ':default xxx'/[default=xxx] was not stable,
   so add an "empty default" marker and respect it during
   parsing and printing.
 - 'find-option' now returns a third "found-p" value, useful
   for the better handling of defaults.
 - Don't print slot readers/writers if they have the same name
   as what would be (silently) generated by the macrology.
 - Don't print 'import descriptor.proto' lines if there are
   no 'lisp_xxx' options in the generated code.
 - Add 'clear-field' to the API; add the functions it needs.
 - Be more careful about how :conc-name interacts with the
   generation of readers/writers.
 - When parsing from a .proto file, force the conc-name to
   be "" unless requested explicitly not to. This has the
   effect of generating readers/writers for all the fields
   of imported Protobufs messages, which I think is probably
   the desired behavior.
 - Add 'make-byte-vector' for convenience.

Passes 'precheckin'. Even with the new unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@545177 f8382938-511b-0410-9cdd-bb47b084005c

 - Don't generate warnings for anonymous enums, they're harmless
   and ubiquitous.
 - 'member' types where are all the members are keywords, is a symbol,
   which got converted to a Protobufs 'string'. Wrong.
 - If a field has an enum type, the (Lisp) default value should be
   treated as a keyword.
 - When parsing a .proto file or using the 'define-xxx' macros, any
   options that are handled specially should be trimmed from the
   options list so that they don't get printed twice.
 - Add 'remove-options' for the above.
 - Make the options printer be more type-aware.
 - Clean up examples.lisp
 - 'schemas-equal' can ignore the schema name/class if they're null.

Passes 'precheckin'. Even with the new unit tests in place.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544737 f8382938-511b-0410-9cdd-bb47b084005c

 - 'define-proto' -> 'define-schema'
 - 'protobuf' (class name) -> 'protobuf-schema'
 - 'find-protobuf' -> 'find-schema'
 - 'parse-protobuf-from-file' -> 'parse-schema-from-file'
 - 'parse-protobuf-from-stream' -> 'parse-schema-from-file'
 - 'write-protobuf' -> 'write-schema'
 - 'write-protobuf-as' -> 'write-schema-as'
 - 'ensure-all-protobufs' -> 'ensure-all-protobufs'
 - 'ensure-protobuf' -> 'ensure-protobuf'
 - 'protobuf-upgradable' -> 'schema-upgradable'
 - 'protobufs-equal' -> 'schemas-equal'
 - 'generate-protobuf-schema-for-classes' -> 'generate-schema-for-classes'
 - 'write-protobuf-schema-for-classes' -> 'write-schema-for-classes'

Update the Quake Protobufs schema generator to use the new names.

The Protobufs documentation doesn't say so, but enums can be packed.
 - Update the wire-level protocol to (de)serialize packed enums.
 - Update the serialization code to use the new wire functions.

Passes 'precheckin'.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544299 f8382938-511b-0410-9cdd-bb47b084005c

edge case problems, but it's good enough to continue the Stubby work.
The critical .proto files (descriptor.proto, etc) all appear to work.

More precise mapping of (mainly integer) types from .lisp to .proto files
so that we can a truly exact model once we compile the .lisp file. It may
not make any difference, but better safe than sorry.
 - Introduce Lisp type 'int32', 'int64', 'uint32', etc
 - Make Lisp->Protobufs recognize the new Lisp types
 - Make Protobufs schema printers recognize the new types

Fix up the handling of imports and namespaces.
 - Always use truenames to record schemas
 - Fix typo in 'find-enum' that caused it never to search other schemas
 - Record the schema in *all-protobufs* at exactly the right time,
   and simplify 'make-load-form' methods

A few tweaks to (de)serialization.
 - Make the bodies for empty messages not generate compiler warnings
 - Don't make ':packed t' be the default, the user has to say so
 - The user documentation doesn't say so, but booleans can be packed.

Passes 'precheckin'.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@544222 f8382938-511b-0410-9cdd-bb47b084005c

(I love this file, it's got one of everything.)

 - The printer should not print extended fields in any message
   that further extends an extended message.
 - If parsing or importing a .proto file needs a Lisp package that
   does not exist, go ahead and create it.
 - Importing now tracks all the schemas imported by the "parent",
   this so that we can search namespaces thoroughly.
 - It looks like 'define-extend' can accept groups. For a deprecated
   feature, it sure is a pain in the neck.
 - Make 'find-message' and 'find-enum' search all namespaces.
 - Print escaped strings, just for completeness.
 - Annotate a couple of common options with their types.

Passes 'precheckin'. Passes my by-hand tests and is epsilon away from
handling the worst .proto file around.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@543912 f8382938-511b-0410-9cdd-bb47b084005c

completely and produces what appears to be a correct model. Next
up, ensure (de)serialization works with this complex beast, which
sets the stage for running the (very complete) Google tests.

 - Be much more careful to preserve type information in option
   values and field defaults, both in the macros and the parser.
   Make the slots in their model objects be untyped.
 - This makes it easier for the Proto and Lisp printers to produce
   exactly the right output, so fix them, too. Be extra careful
   with booleans.
 - Fix the printer to recognize more of the standard options so
   that their values get printed with the correct type.
 - Learn to parse more complicated numeric and integer constants.
 - Learn to parse escaped strings.
 - Learn to parse comma-separated options.
 - Learn to parse extensions with no 'to' specifier.
 - Fix the places that used (deprecated) :single instead of :float.
 - Proto enum values always map to Lisp keywords, so don't futz
   with packages for them.
 - Add some more doc strings.
 - Add some more examples-that-will-become-tests.

Passes 'precheckin'. Passes my by-hand tests.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@543727 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@543135 f8382938-511b-0410-9cdd-bb47b084005c

In particular, uninitialized optional boolean fields
whose Lisp type is (or (member t nil) null) now work.

Implement 'object-initialized-p' and 'slot-initialized-p'
and their Python lookalikes, 'is-initialized' and 'has-field'.

Update the documentation.

Passes 'precheckin --full+', which makes sense since none
of this is called in normal use yet.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@542953 f8382938-511b-0410-9cdd-bb47b084005c

which has been deprecated for years, is still in wide
use, e.g., in Chubby's bnsresolver.proto

So, implement support for groups:
 - Add a model class for it
 - Add .proto and .lisp printers
 - Add a 'define-group' macro
 - Make the .proto parser know how to parse them

Passes 'precheckin --full+', which makes sense since none
of this is used in normal use yet.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@542500 f8382938-511b-0410-9cdd-bb47b084005c

to fully serialize a LegacyPnr aggregate.

Fix 'skip-element' to ignore (deprecated) groups.

Passes 'precheckin --full+', which makes sense since none
of this is used in normal use yet.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/quux/protobufs@542375 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/branches/qres/swm/borgify-1/qres/lisp/quux/protobufs@540933 f8382938-511b-0410-9cdd-bb47b084005c