(C) 2020 Geotab Inc

All files and artifacts in this repository are licensed under the provisions of the license provided by the LICENSE file in this repository.

Reference compression design

The design is context aware, i. e. it knows that the payload is JSON, and the set of key-value pairs that can be expected. As the payload sizes are typically less than 100 characters, this is probably necessary to achieve high compression ratios. Tests with standard compression algorithms like GZIP, LZW, and DEFLATE showed comparably low compression ratios.

The overall compression consists of the following compression features:

1. Paths are replaced by a two byte index from the list of paths found in vsspathlist.json.
2. Fixed keywords, such as "action", "path", "get", etc., are replaced by a one byte value.
3. Timestamps are replaced by a four byte value.
4. The value part for the keys "value", "requestId", "subscriptionId" are replaced by a binary value, if possible.
5. JSON reserved characters are removed in the case their excistence in a certain position is a MUST according to the JSON format rules.
6. Payload data that does not fall under any of the features above is left uncompressed.

Below follows further details about the compression features.

1. The list vsspathist.json is generated by the VISSv2 server at every startup. The input to the server comes from the "vss_vissv2.binary" file, which is in its turn generated by the VSS Tools, that take the VSS tree as represented by the vspec files from the VSS repo as input. This means that if the VISSv2 server starts up with an updated VSS tree, then the clients should be aware of this, and get a copy of the updated vsspathlist.json file from the server. This could be handled by a request to the server asking for this file, but the VISSv2 spec does not currenly support that.
   The pathlist in the file generated by the VISSv2 server is ordered, so searches in the list can be optimised.
   The list index is represented by two bytes, which means that a maximum of 65536 paths can be handled. The list only contains paths for leaf nodes, so this is the maximum number of leaf nodes that the VSS tree can have for this compression feature to work.
   

2. The value replacing the keywords are the index from the kewordlist plus 128. The reason to add 128 is that an uncompressed UTF8 JSON string only contains byte values less than 128, see an ASCII table. The keywordlist original is found in the "common.go" source file. It is not expected to change. A client compression impl needs a copy of it.
   

3. The uncompressed timestamps follow the ISO8601 standard and has the format YYYY-MM-DDTHH:MM:SS.ssssssZ
   This is compressed as follows:

• The least significant year digit is represented by four bits.
• The month is represented by four bits.
• The day is represented by five bits.
• The hour is represented by five bits.
• The minute is represented by six bits.
• The second is represented by six bits.

Subsecond usage is not supported currently.

The sum of bits above is 30 bits, which fits into 4 bytes as shown below.
Byte 1 | Byte 2 | Byte 3 | Byte 4
00yyyymm | mmdddddh | hhhhmmmm | mmssssss

The timestamp encoding has the limitation that when the year goes from 2029 to 2030, any messages that are received by a client after the new year has begun, that contain a timestamp from the previous year, will be incorrectly decoded as they will be assigned the year 2039 instead of 2029. A solution to this could be to use Unix time instead, but then the 2038 problem will be coming. Another would be for clients to check for this 10-year problem around the time when it becomes likely. Vehicles “should not” respond with too stale data, so a few months after 2030 begins should be max what is needed.

4. The value types that are supported are integer, float32, and boolean. The first byte is and index from the keywordlist, which also contains this (+128). The type indicated also sets the number of bytes used for the representation, as shown below. The type "nuintx" means it is a negative uint value, as all integer values are represented as uints.
   Type | #bytes
   nuint8 | 1
   uint8 | 1
   nuint16 | 2
   uint16v | 2
   nuint24 | 3
   uint24 | 3
   nuint32 | 4
   uint32 | 4
   bool | 1
   float32 | 4
   

The absolute value of integers are compared to the max value of the different types, and the smallest possible size is selected. Values that are not of any of these types are kept uncompressed.

5. The following JSON reserved character usages can be removed as at decompression the JSON rules will infer their reinstatement.

• The leading and trailing curly brackets.
• The colon between key and value.
• The comma between JSON data entities.

6. The fact that data that does not relate to any of the mentioned compression features is left uncompressed leads to that the usage of this compression does not apply any restriction to the syntax scope of VISSv2. It also enables the client to send uncompressed requests, where the response will be compressed.

The current compression solution does not support service discovery requests.