Conflicts:
	serialize.lisp

Reviewer: tunes

which shall fix the issue that causes of these actions never being operation-done-p.
However this does NOT handle recursive import dependencies,
which requires further work.

Tested: QRes doesn't always recompile the proto and all dependees anymore.
Reviewer: asedeno

* Previously, if a field in a .proto file referenced a message in
  another proto file using a different lisp package, the cl-protobufs
  library would silently fail to serialize the field.  A similar
  problem would occur if a message defined later in the file used the
  lisp_name option to override the name generated by cl-protobufs.
  This change fixes these issues and others.
* Add conditions that are signaled when encountering an undefined
  type.
* Delay assigning lisp classes/types to fields and methods until all
  possible forward references have been parsed.
   * This allows the class slot to be unbound, so check for that case
     in print-object methods.
   * Add a test for forward references to messages that override the
     lisp name.
   * Add a test for references to messages and enums defined in
     another proto file with a different lisp package.
   * Change color-wheel-stability test, because it used "string" as
     the input type for an rpc, which seems to be disallowed (although
     I haven't found this documented anywhere).
* Signal errors during parsing for undefined types.
   * Add a test for these errors.  Add assert-error macro to qtest.
* Signal a condition if we are unable to find the definition for a
  field's type during serialization, deserialization, determining an
  object's serialized size, printing text format, parsing text format,
  or generating code for one of the above.
* Remove logic in find-qualified-name that indirects through lisp
  packages.  Proto packages and lisp packages do not necessarily map
  1-to-1.
* Always use the schema's lisp package for any symbols generated when
  parsing proto files.
* When generating lisp code using write-schema-as, set the package to
  the package used in the generated file, so that ~s will print the
  package prefix in the correct circumstances.
* Remove broken proto1 "streams" parsing ("returns" comes before
  "streams" in every example I've found); replace with proto2 syntax.
* In process-imports, the call to find-schema using a pathname was not
  giving the expected result.  Sidestep this issue by using the same
  logic to find the schema as is used earlier in the function.

Avoid SUBTYPEP during macroexpansion, it can be very expensive on SBCL.
Actually recognize LIST-OF:LIST-OF constructs.
In DEFINE-SCHEMA, don't nest the DEFMETHODs inside the LET,
this also can cause SBCL's control flow analysis to blow up.

Work done with Steven Spitz for QPX.

Tested: (asdf:test-system :cl-protobufs)

Testing: [trunk r567444] precheckin --parallel 11  PASSED
Reviewers: dcrawford, asedeno

Rationale: there is a problem in quicklisp due to recent asdf package changes,
that requires urgent fix to cl-protobufs,
so I'm updating these for the sake of issuing
a cl-protobufs fix to quicklisp ASAP.
I require these updates for some further changes I'm making to QUUX, anyway,
so it's just a matter of timing.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@567483 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@563247 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@563209 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@563144 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@563140 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

The unoptimized serialization code was slower than
it needed to be and was a Niagara falls of consing.

The problem is that (map () (curry ...)) is poorly
optimized, which is a shame because it's such a nice
coding style. :-P

The fix is to replace 'map' with iteration (sigh) in
a few critical places.

- Add 'doseq', which chooses between 'dolist' or 'dovector'.
- Make 'serialize-object', 'deserialize-object' and 'object-size'
  use 'doseq' instead of using 'map'.
- Ditto for 'print-text-format'.
- Fix the 'serialize-packed' and 'packed-size' optimizers to
  use 'dolist' or 'dovector' based on whether the field is
  a vector field. This makes the optimized code faster, too.
- Make the optimized serializer generators pass in 'vectorp'
  so that the wire-format optimizers can do a better job.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@562921 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@562757 f8382938-511b-0410-9cdd-bb47b084005c

 Testing : precheckin --full (both against Oracle and against megastore for customers)
 Reviewer: eschwartz

 JTB impact: n
 Ops impact: n
 Security impact: n

 Change to config                        : n
 Change to XML schema                    : Y (XMLREQ bug 120820)
 Change to DB schema                     : Y (DBREQ bug 119801)
 Change to transport (timeouts, headers) : n
 Any change (or new use) of OAQs         : n
 Change to inter-component transactions  : sort of (use megastore in development-only environments)
 Depends on any other checkin / bug      : n

 Tests that will verify : tsunami and megastore unit tests + existing customer profile tests

Tsunami is a new layer that sits between business logic code and the
persistence logic. The main manifestation of this layer is
define-domain-class macro that defines an interface to a domain
class. You can then define how to perist in various implementations,
e.g. using define-quake-db-class-bindings for quake and
define-proto-binding & define-megastore-proto-binding for megastore.

Tsunami framework is still a work in progress, and there is a bunch of
outstanding tasks/issues.

As the first application, I refactored customer profile code to use
the tsunami framework. This enables storing of customer profiles in
megastore (or other data stores in the future), while everything else
is stored in Oracle.

Here is the list of other miscellaneous/harder-to-understand changes:

- I changed @id attribute of ContactInfo objects to use the actual
  numeric ID, rather than passing it through encode-record-locator (in
  qres-dev schema only). This required changing XML schema type from
  typeBasicLocatorString to typeBasicID.

- Drop quake/relationships.lisp. Use a more straightforward way to
  associated ADO agencies to customer profiles.

- Customer usernames are now stored as lowercase strings (this means
  that I don't need a separate field in megastore protobuf to
  implement case-insensitive index).

- agency-id permission attribute is now the agency-key (serialized to
  string), not the agency's record ID. This way you don't need to load
  agency to do permission check when all you have is the agency-key

- moved agency-key from core/agencies.lisp to
  lisp/core/agency-types.lisp, so that agency-key type can be used in
  permissions

- add when* test handler that takes a random Lisp expression (as
  opposed to the existing when that takes a variable name and looks at
  whether it is empty)

- remove cc-deleted-error -- there is no such thing as deleting
  cc-info

- refactored cc-information, so that it is used as a value object (ie
  a mostly-immutable object with no identity) in the application. When
  saving to Oracle database, it is saved in the central
  tbl_cc_information table with appropriately populated ownership
  columns (pointing to the customer, journey,
  accountable_doc_container, etc)

- drop unneeded tbl_cc_transaction columns that were caching cc_info
  values


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@562275 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@561802 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Sergey V, Shaun M

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

I had made a "policy decision" that the package created
by importing a .proto file would (:use :common-lisp). In
practice, this turned out to be a mistake. Sergey suggested
a fix that I think is correct, so in it goes.

Shaun noticed that the optimized 'serialize-object' and
'object-size' methods didn't quite implement what they were
supposed to for optional boolean fields whose value was never
supplied. Fix the optimized methods to implement the same
(correct) semantics as the unoptimized ones.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@561714 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

I extended the CL-Protobufs tests

Description:

Add a more readable syntax for enum and field indices.

For define-enum, it was (name value).
  Allow (name :index value).

For define-message, it was ((name index) ...).
  Allow (name :index index ...).
  Complain if both forms appear in the same field.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@561136 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

I extended the CL-Protobufs examples

Description:

Add a few missing features that aren't in any documentation.

String literals can look like "foo"<whitespace>"bar".
 - Fix 'parse-string' to handle this.

Option values can be complex structures, not just atoms.
 - If 'parse-proto-option' sees a '{' character after the '=',
   it should use 'parse-text-format' to read the option value.
 - Fix the 'protobuf-option' printer to call 'print-text-format'
   for complex option values.
 - Minor refactoring to the text format parser and printer in
   order to support the above.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@560639 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

I extended the CL-Protobufs tests

Description:

Add support for a 'streams' type in Protobufs RPC methods.
 - Add a few slots to 'protobufs-method'
 - Add '&key streams' to the method arglists in 'define-service'
 - Extend the parser to handle "streams" in addition to the
   'streams_type' option
 - Extend the printer
 - Change a test to use a 'streams' type
 - While we're in the neighborhood, allow an optional '=>'
   between the input and output methods in 'define-service';
   it makes it a bit more readable
 - Update the documentation



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@560147 f8382938-511b-0410-9cdd-bb47b084005c

Testing : precheckin --full --strict-errors
Reviewer: Fare (please)

JTB impact: No
Ops impact: No

Change to config                        : No
Change to XML schema                    : No
Change to DB schema                     : No
Change to transport (timeouts, headers) : No
Any change (or new use) of OAQs         : No
Change to inter-component transactions  : No
Depends on any other checkin / bug      : No

Tests that will verify:

The CL-Protobufs tests

Description:

Add type aliases to CL-Protobufs. It's a Lisp-only
feature that allows you define a new Protobufs type in
terms of a Lisp type, a serializer and a deserializer.
 - Add a new model class, 'protobuf-type-alias'.
 - Extend schemas and messages so that they can hold
   a set of type aliases; add 'find-type-alias'.
 - Add 'define-type-alias'.
 - Factor out 'lisp-type-to-protobuf-type' from
   'clos-type-to-protobuf-type', I need it for the
   'define-type-alias' macro.
 - Fix the schema printer to print 'define-type-alias'
   for .lisp schemas, and to add a comment describing
   the type alias in .proto schemas.
 - Extend the wire format (de)serializer to know
   about type aliases; it just (de)serializes the
   Lisp type as the Protobufs primitive type using
   the provided (de)serializer functions. There's
   zero cost to this if you don't use aliases.
 - Update the optimized serialization generation.
 - Similarly extend the text format (de)serializer
   to know about type aliases.
 - Change an example to use a type alias for 'date'.
 - Update the documentation.



git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559861 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559563 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559296 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@559035 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@558959 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@558792 f8382938-511b-0410-9cdd-bb47b084005c

git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@552185 f8382938-511b-0410-9cdd-bb47b084005c

makes my CL-Stubby client code simpler. So, fill in the 'parent'
slot for more than just messages. No cost at runtime, so WTH.

Passes 'precheckin' with the new Protobufs unit tests in place.
CL-Stubby tests pass in Google3 environment.


git-svn-id: http://svn.internal.itasoftware.com/svn/ita/trunk/qres/lisp/libs/cl-protobufs@551616 f8382938-511b-0410-9cdd-bb47b084005c