Add support for array-of-struct

[PeterJohnson]

Right now we have support for array-of-native and raw data types. The latter can easily be used to create an array of structured data but there's currently no standard way of doing this. A standardized format for array-of-struct would enable viewers and users to have a common way to communicate more complex data in an atomic fashion. It could be as simple as prepending a string of data types (ala Python's struct module) to the raw data elements, as you can derive the element size from the string of data types and the array size from the raw data length divided by the element size.

[PeterJohnson]

Since we probably want to allow structures with variable length fields (e.g. string inside a struct) we should provide the array size (ULEB128 encoded) so it can be allocated without needing to prescan the array.

[333fred]

I'm wondering if a better way of doing this would be to introduce a transaction message, and exploit the already-structured and marked nature of network tables to introduce atomicity.

What I'm thinking is something more like test and set on a number of keys, as opposed to a single structured data type. If we expose transaction numbers to the user, that can serve as a mark, and allow them to guarantee that a certain set of data is at the same version. If the transaction numbers don't match during retrieval, they can attempt to retrieve it again. We could potentially even automate the process.

This would also need to introduce a transaction message type, which causes the server to attempt to change all keys at once. If a key is updated individually during that transaction, the transaction fails, and the client can try again. This could allow us to have a more lockless design and support arbitrarily complex objects, simply by the nature of NetworkTables structured format, without having to add support for a new complex data type.

[ThadHouse]

Didn't NetworkTables 1.0 have transactions, but they got removed because it was too complicated for the teams? I definitely think the way we do atomic transactions should be invisible to the users, and using a raw data type specially encoded seems to me like it would be much easier then modifying the entire internals of NetworkTables to add transactions.

If we do end up using a new custom data type, could we use something like JSON to encode it? If we did that, it would be fairly simple to encode and decode that from both the Java side and the C++ side, and we can get libraries that already support it. My only worry about writing a custom encoding is that C++ couldn't directly read a Java object (at least I don't think), and vise versa, so we would end up having to write the encoding and decoding both in Java and C++.

[PeterJohnson]

I'm not a big fan of JSON, because it's significantly larger than binary-encoded values and string parsing is incredibly inefficient. Serialization code isn't that complicated for basic data types (just a for loop with a switch...), so I'm not too concerned about implementing it in both Java and C++ if we keep it simple (e.g. no nested structs). We wouldn't even need to touch the internals of ntcore, just add functionality to the user NetworkTable layer.

I can see the argument for a more generalized synchronization/transactions, but NT is currently built around atomicity only at the individual entry level, so it's going to be a lot of work and additional complexity to make sure it's 100% robust. Keep in mind that any change to the low level protocol will require LabView to be updated as well. If we want to create a "raw" variant type for structured data to disambiguate it I don't think that's a big deal (RPCs are done this way) but I'm not sure we even need that.

[PeterJohnson]

CBOR and MessagePack are both now available for C++ for storing raw structured data. Once a suitable Java library is decided upon, this can be closed.