Feature ID  : 740150
Response Due: xx/xx/xxxx
Title       : New format for multi-valued result


Description
-----------

This feature will provide support for:

1) Marshalling/Unmarshalling multi-valued data in a language-centric manner.  At
   a minimum, we will provide support for Arrays/Lists and Maps

2) Updating existing STAF services to use this new capability when returning
   a result string that contains more than one value.

3) The STAF command line utility will be updated to support a consistent
   presentation scheme for this new structured data.


Problem(s) Solved
-----------------

Currently, there is no consistent scheme used when a STAF service needs to
return more than one logical piece of data.  Each service has been left to its
own device on how it wants to separate one piece of data from another.  In
addition, it can even be difficult to come up with a scheme given that the
content of each "field" of data may not be generated by the service itself, but
may, instead, have come from other source (such as the user or another service).

Another problem is that it is very difficult to add new data to an existing
return value.  Most of the existing return values weren't designed with
expansion in mind, and even where they were, most users only coded their
decoding routines to the current encoding, without care for future values.

This feature will provide a common marshalling/unmarshalling scheme that will
support the native data structures that most programming languages provide.  At
a minimum, it will provide support for Arrays/Lists and Maps.

All existing services will be updated to use this new mechanism.

This new mechanism is designed to ensure forward compatibility from STAF V3.0
onward, by allowing new information to be added to an existing return value
without affecting existing clients.


External Changes
----------------

1) Marshalled data format

This feature introduces the concept of a STAF Data Type (SDT).  This feature
supports (un)marshalling any valid SDT.  The specific SDTs supported by this
feature will be described in detail below.

The general format of a marshalled STAF Data Type is

  @SDT/<type-info>:<data-length>:<data>

Note that <type-info> must not contain a ':' so that the unmarshalling routines
can determine the beginning of the marshalled content.  This is mainly a note
for those implementing the marshalling routines and for those who may add new
data types in the future.  The unmarshalling routines will still need to be able
to deal with data that is not properly formatted.

The string "@SDT/" should be sufficiently unique that it won't randomly appear
in strings (particularly at the beginning of strings), so there is little chance
of us confusing a piece of marshalled data.  By enforcing the use of
<data-length>, we can ensure that we can always access any other (possibly
marshalled) data beyond a given marshalled SDT.

When unmarshalling an unrecognized data type, the unmarshalling routine should
simply return the entire mashalled object string as a string type.

In pseduo-code a reference to any arbitrary SDT will be represented as <SDT-Any>.

1a) Scalar Data Types

The general form of a marshalled scalar SDT is

  @SDT/$<scalar-type>:<scalar-length>:<scalar-value-as-string>

In pseduo-code, a reference to a generic scalar value will be <SDT-Scalar>.

Currently, we will support two scalar types, the "null" type and a string type.

There is only one possible value for the "null" type.  Its marshalled
representation is

  @SDT/$0:0:

In pseduo-code, a reference to the "null" SDT will be represented as <SDT-NULL>.

The format for a marshalled string type is

  @SDT/$S:<string-length>:<string-length>

The unmarshalling routines are required to support any scalar value.  If the
scalar is of a type that is not understood, the scalar is treated as of type
string.

In pseduo-code, a reference to a string value will be represented as
<SDT-String>.

1b) Array/List Data Type

This feature will support the notion of an order sequence of items.  The items
are not required to be homogeneous.  The general form of a marshalled array is

  @SDT/[<number-of-items>:<array-length>:<SDT-Any-1>...<SDT-Any-n>

In pseduo-code, a reference to an array value will be represented as
<SDT-Array> or <SDT-List>.  A more verbose form may be used as well.  This form
will be represented as

  [
    <SDT-Any-1>
    ...
    <SDT-Any-n>
  ]

1c) Map Data Type

This feature will support the notion of a container of key/value pairs, called
a map.  The key portion of each pair  must be a string.  The value of each pair
may be of any supported SDT.  The values in the map are not required to be
homogenous.  The general form of a marshalled map is

  @SDT/{:<map-length>:<key-1-length>:<key-1><SDT-Any>
                     ...
                     :<key-n-length>:<key-1><SDT-Any>

Please note, there are no actual carriage returns/newlines in the above.  It is
split on multiple lines to help aid readability.

In pseudo-code, a reference to a map value will be represented as <SDT-Map>.  A
more verbose form may be used as well.  This form will be represented as

  {
    'key-1': <SDT-Any-1>
    ...
    'key-n': <SDT-Any-n>
  }

1d) Map Class (Definition/Instance) Data Type

When a result consists of a large set of maps with homogenous keys, a lot of
data can be wasted on the redundant key information.  In addition, when
marshalling an arbitrary map (from some target language), it is possible that
only a portion of the key/value pairs are intended to be marshalled.  Also, for
display purposes, there may be a certain order in which keys should be displayed
(along with possibly other display meta-data).  In order to support this, this
feature provides for the notion of "map class".  A "map class definition"
consists of the meta-data mentioned above and has a corresponding name.

A map class definition is simply a map of the following form

  {
    'keys' :                   # ordered list of keys to marshall
      [                       
        {
          'key': <SDT-String>  # key name
        }
      ]
  }

A map class definition is marshalled as a regular <SDT-Map> as part of a
Marshalling Context (see below).  Although unlikely to normally appear in
pseudo-code, a map class will be referenced in pseudo-code as
<SDT-MapClassDefinition>.

There is also a corrsponding "map class instance" marshalled format.  This
represents a map that has been marshalled in the context of a specific map class.
The general form of a marshalled map class instance is

  @SDT/%:<map-class-instance-length>::<map-class-name-length>:<map-class-name>
    <SDT-Any-value-1>
    ...
    <SDT-Any-value-n>

Please note, there are no actual carriage returns/newlines in the above.  It is
split on multiple lines to help aid readability.

Typically, a map class instance will be referred to in the same way as a normal
map.  In cases where the distinction is significant, a map class instance will
be referred to in pseudo-code as <SDT-MapClassInstance>.

In general, a map in the native language specifies that it should be marshalled
using a map class by specifying a key named "staf-map-class-name" whose value
is the name of the map class definition.  When marshalled, the marshalling
routines will attempt to lookup this map class definition in the marshalling
context.  If found, it will be used.  If not, the map is marshalled like an
ordinary map.

When unmarshalled, a map class instance will set the same key,
"staf-map-class-name", so that client code has access to the display meta-data.

1e) Marshalling Context

A marshalling context is a container for map class definitions and an object
that is marshalled in that context.

The map class definitions are marshalled as part of a larger context meta-data
map.  The format of this map is as follows

  {
    'map-class-map' :
      {
        '<map-class-name-n>' : <SDT-MapClassDefinition> 
      }
  }

The extra layer of indirection provided by encapsulating the map class
definitions in an outer map is that it allows us to add additional contextual
data in the future that may be beneficial in later versions of STAF.

The general form of a marshalled marshalling context is

  @SDT/*:<marshalling-context-length>:<marshalling-context-metadata-map><SDT-Any>

In pseudo-code, a reference to a marshalling context value will be represented
as <SDT-MarshallingContext>.

1f) Example marshalled data

XXX: Not sure if we need this section.  If we do, it should contain some examples
     of how actual marshalled data would look.

2) <C/C++ interfaces>

3) <Java interfaces>

package com.ibm.staf;

public class MarshallingContext
{
    // Constructors
    public MarshallingContext()
    public MarshallingContext(Object obj)
    public MarshallingContext(Object obj, Map mapClassMap)

    // General bean methods    
    public void setMapClassDefinition(String mapClassName,
                                      Map mapClassDefinition)
    public Map getMapClassDefinition(String mapClassName)
    public void setRootObject(Object rootObject)
    public Object getRootObject()

    // Returns the root object if the context has no map class definitions.
    // Otherwise, returns the context itself.
    public Object getPrimaryObject()

    // Returns a read-only copy of the map class map
    public Map getMapClassMap()

    // Marshalls the actual marshalling context.
    // This is the primary marshalling interface users should use.
    public String marshall()

    // This is the primary unmarshalling interface users should use
    public static MarshallingContext unmarshall(String marshalledObject)

    // These are utility methods that should, in general only be used by
    // marshalling code.
    //
    // Note: If the XXX below isn't implemented, then these methods be able to
    //       be made private.

    public static String marshall(Object objectToMarshall,
                                  MarshallingContext context)
    public static MarshallingContext unmarshall(String marshalledObject,
                                                MarshallingContext context)
}

XXX: I'd like to support an interface which a class could implement which would
     cause the marshalling routines to call a method on the interface to
     allow the implementing class to do some custom marshalling.  I have not yet
     prototyped this though.

XXX: It is likely that we will need/want to add some specific classes to aid in
     the creation of map class definitions.

4) <Command line interfaces/behavior>

XXX: Is pretty printing going to be provided for other interfaces?

5) Modified service request results

The list of known service requests that will be changing is as listed.
The existing format as well as psuedo-code for the new format is provided.
Please note that most of these examples DO NOT take into account the new data
that will be added as part of STAF V3.0, such as network interface,
logical network id, physical network id, authenticator, userid, etc.

1) LOG QUERY

1a) LOG QUERY (Entries)

  Old format:

    <Timestamp-1>|<Machine-1>|<Handle-1>|<Handle name-1>|<Level-1>|<Message-1>
    ...
    <Timestamp-n>|<Machine-n>|<Handle-n>|<Handle name-n>|<Level-n>|<Message-n>

    Note: The '|' can be replaced by any separator by the user
    Note: There is a distinct lack of separator from one "record" to another,
          except for a carriage-return/newline.

  New format:

    [
      {
        'timestamp'   : <Timestamp>
        'machine'     : <Machine>
        'handle'      : <Handle>
        'handle-name' : <Handle name>
        'level'       : <Level>
        'message'     : <Message>
      }
    ]

  XXX: Are there enough results similar to this to warrant a row/column specific
  format?

XXX: I added this one to get us started.  Sharon has the complete list of
     commands that will be changing.


Internal Changes
----------------



Design Considerations
---------------------

Some of the forward compatibiliy items (such as the ability to add new data
types and the ability to add new data to the marshalling context) don't come for
free.  Down level clients still won't be able to understand the new data types
or interpret the meaning of the additional marshalling context data.  So, you
would likely still want to understand what version of STAF is running on the
client so that you could marshall things so that it would understand them.
Nevertheless, the additional functionality is helpful in that if a downlevel
client obtained up-level information, it would still be able to parse the
marshalled data without error.


Backward Compatibility Issues
-----------------------------

This feature forces a large number of backward compatibilites, as it forces
every service result which contains multiple logical pieces of information to
be updated to use the new marshalled data format.


What works (or is implemented) right now
----------------------------------------

Only a rudimentary stand-alone Java prototype.



