TO DO - Address the remaining "---" issues - Get the rest of the unit tests working - Ensure we can parse/print descriptor.proto, unittests.proto, etc - unittests.proto completely fails when 'optimize = SPEED' - Ensure we can (de)serialize "golden" data against unittests.proto - Get the rest of Bob Brown's tests working [ - Convert examples into a test suite ] [ - Implement 'protobufs-equal' predicate ] - Get "namespaces" (i.e., nested classes) fully working [ - Messages and enums should have a 'fully-qualified-names' slot ] [ that records both symbols and strings, and in the correct package ] [ - Printers need to print qualified names, as needed ] [ - 'find-message' and 'find-enum' need to search namespaces ] [ if the quick compare against the name/class fails ] - 'message Foo { message Bar { ... } ... }' should (maybe!) produce a Lisp class defaultly named 'foo.bar' - Add a keyword arg, :class-name, that overrides this convention; in .proto files this should be called lisp_class - Also, add :slot-name for slots and :type-name for enums (called lisp_slot and lisp_type in .proto files) - Or maybe not, since 'foo.bar' isn't very Lispy; for the most part, packages provide enough room to avoid name clashes - Get 'merge-from-message' fully working - See the place in 'deserialize-object' that needs to merge, too - Maybe this isn't so useful in Lisp, where I'd prefer to encourage a more function style - Lose 'proto-class', replace it with 'proto-type'; its type is (or null symbol) 'proto-type' should always be the name from the Lisp side [ All 'define-xxx' macros should have, as their first argument, ] [ a type (not a "name") ] [ In .proto files, there should be an 'option lisp_name="pkg:name"' ] [ which can be used to override the 'proto->class-name' default ] [ for messages and enums ] - Refactor 'protobuf-field' so the names are clearer: - 'proto-value' should be the Lisp name of the slot (this is already the case) - Add new 'proto-field' slot, which will hold the Protobufs name of the field (this used to be in 'proto-name') - Lose the existing 'proto-type' slot, then rename the 'proto-class' slot to 'proto-type'; this will hold the Lisp name of the slot type - 'proto-name' will hold the Protobufs name of the field type (this used to hold the Protobufs name of the field) [ - &key name should give the Protobufs name for the field ] [ - 'option lisp_name="pkg:name"' should give the Lisp name for the slot ] [- Need search paths in the ASDF .proto module ] [- Make 'import' really work ] [ - *all-protobufs* should record keyed by pathname, too ] [ - 'import' should not bother loading if the import is present, ] [ and should record the imported schema when it's done ] [ - 'find-message' and 'find-enum' should search imports, too ] [- Implement 'vector-of' in addition to 'list-of' ] [- Do we need 'has-extension' and 'clear-extension'? ] [ If so, just generate methods for them in 'define-extend' ] [- 'package'/'lisp_package' should create the package, if necessary. ] [- Don't generate 'import "net/proto2/proto/descriptor.proto"' and ] [ 'extend proto2.MessageOptions { ..}' lines unless we need to include ] [ one of the lisp_* options in the generated model ] [- Handle 'import' ] [ It should probably just compile/load the file when the import ] [ is seen ] [ - The Lisp macros should be able to import both .lisp and .proto files ] [ - The .proto parser should only be able to import .proto files ] [- Handle 'extension' and 'extends' ] [ - 'define-extension' just declares the extension within a message ] [ and adds it to 'proto-extensions' for the message ] [ - Easiest is to add an "extension-p" flag to 'protobuf-message' ] [ - Define new 'define-extends' macro ] [ which *copies* all slots from the message being extended, ] [ then just adds the extension fields as (flagged) ordinary fields ] [ with reader/writer function (see below) ] [ - Factor out field-processing code from 'define-message' ] [ - If we do it this way, no other special magic is needed ] [ Serialization, e.g., should just work ] [ Scoping should also just work ] [ - We will need 'proto-writer' as an analog to 'proto-reader' ] [ - Get deserialization working; ] [ Need to call the 'proto-writer' function if its around ] [ instead of just using (setf (slot-value ...) ...) ] [ - Don't forget to check that the "extends" indexes match ] [ what was declared by 'define-extension' ] [ - And don't allow overlapping extension indexes ] [ - Generate new getter and setter methods for each extended slot: ] [ (let ((an-int 10)) ] [ (declare (type (or integer null) an-int)) ] [ (defmethod an-int ((object extendable-foo)) ] [ an-int) ] [ (defmethod (setf an-int) (val (object extendable-foo)) ] [ (declare (type (or integer null) val) ] [ (setq an-int val)))) ] [ - Handle "extends" in the parser, too ] [- Support "deprecated" groups. Sigh. ] [ - Need 'proto:define-group', which is like 'define-message' ] [ - Need to parse 'group' things ] [ - "option group Locator = 1 { ... }" ] [ means create a submessage (and class!) called "Locator" ] [ whose field is named "locator" (note lower case) ] [ then just treat it like a message ] [ - Need a printer for groups, too ] [ - Make 'proto-extension-p' be 'proto-message-type', ] [ which is one of :message, :extends or :group ] [ - (De)serialization of groups uses a start and end tag, ] [ and the end tag counts as part of the group length ] [ - Serialization uses the start-group wire type, ] [ then serializes the message, ] [ then writes the end-group tag ] [ - Deserialization works the usual way, ] [ except that it stops at a matching end-group tag ] [ (i.e., has the same field number as the start tag) ] [ - Add new 'end-tag' argument to 'serialize-object' and ] [ the 'serialize' internal function, it's the expected ] [ end tag value to indicate end-of-group; if the default ] [ value for this is 0, then the (i= tag 0) test can be ] [ changed to (i= tag end-tag) (sweet!) ] [- We need both 'package' and 'lisp-package' for protobufs ] [ 'package' gets set from the 'package' line in .proto files ] [ 'lisp-package' gets set from a new 'option lisp_package="pkg"' ] [ 'define-protobuf' needs both :package and :proto/:lisp-package ] [- Speed up 'list-of' by shadowing the Quux version, ] [ and don't do any type-checking (!) ] [ Having done this, we can probably remove the hack of using map:map, ] [ which should make deserialization much faster ] [- 'proto-name' should always be the name from the Protobufs side; ] [ its type is (or null string) ] [ All 'define-xxx' macros should have '&key name' which gives the Protobufs ] [ name, which can be used to override the 'class-name->proto' default ] [- 'class-override' should be called 'alias-for'; ] [ its type is (or null symbol) ] [ 'define-message' and 'define-enum' both get '&key alias-for' ] [- 'find-message' and 'find-enum' should only look at (the new) 'proto-type', ] [ and never (the new) 'proto-alias-for' ] [- 'deserialize-object' uses the alias when it calls 'make-instance', ] [ or if there is no alias, 'proto-type' ] [- 'serialize-object' and 'object-size' should take both 'object' and 'type' ] [ args, and should not call 'class-of' (and do call it "type") ] [ The 'cl' (and 'class') locals in '(de)serialize-object' and 'object-size' ] [ should instead be called 'type' ] [ In .proto files, there should be 'option lisp_alias="pkg:name"' ] [ that gets used to fill in 'proto-alias-for' ] [- 'serialize-object' and 'object-size' never use the alias; ] [ instead all the "dispatching" should be done of off 'proto-type', ] [ Ditto, 'print-text-format' ] [- Deserialization should stop if it sees a #x00 in the position where ] [ it's expecting a tag, since no tag can be #x00 (fields start at 1) ] [- The "name" of a protobuf should be the camel-cased file name ] [ The "type" of a protobuf should just be the file name as a keyword symbol ] [- RPCs should have 'input-name', 'input-type', 'output-name' and 'output-type' ] [ The "name" ones are for the Protobufs side ] [ The "types" ones are for the Lisp side; we look up on "type" only ] [ In the Lisp, the RPC types can be either 'type' or '(type :name )' ] [ In .proto files, there should be an 'option lisp_name="pkg:name"' ] [ which can be used to override the 'proto->class-name' default ] [ for the name of the RPC function ] [- Finishing eliminating 'protobuf' argument from the API ] [ - Lose it in 'serialize-object-to-stream' and 'deserialize-object-from-stream'] [ - Rename it to 'trace' in '(de)serialize-object' and 'object-size', ] [ make it be the last supplied argument ] [ - Fix up the '(de)serialize-object' and 'object-size' generators, too ] [ - Apply the same fix to 'print-text-format' ] [ - Then make the compatibility API actuallty work ] [- Implement 'parse-text-format' ] [- Renamings ] [ 'define-proto' -> 'define-schema' ] [ 'protobuf' (class name) -> 'protobuf-schema' ] [ 'find-protobuf' -> 'find-schema' ] [ *all-protobufs* -> *all-schemas* ] [ 'parse-protobuf-from-file' -> 'parse-schema-from-file' ] [ 'parse-protobuf-from-stream' -> 'parse-schema-from-file' ] [ 'write-protobuf' -> 'write-schema' ] [ 'write-protobuf-as' -> 'write-schema-as' ] [ 'ensure-all-protobufs' -> 'ensure-all-protobufs' ] [ 'ensure-protobuf' -> 'ensure-protobuf' ] [ 'protobuf-upgradable' -> 'schema-upgradable' ] [ 'protobufs-equal' -> 'schemas-equal' ] [ 'generate-protobuf-schema-for-classes' -> 'generate-schema-for-classes' ] [ 'write-protobuf-schema-for-classes' -> 'write-schema-for-classes' ] [- Write documentation ]