The first generation of SolarTech message boards ran on embedded i386 controllers and had 1MB of RAM and as permanent storage 1 MB of battery-backed RAM. It had a 3-line character-based LED display, and a waterproof keyboard. The OS was an embedded version of DOS, and the controller software was written in C. They were only capable of talking to first generation modules in a 3x8 configuration (48x27 pixels). The initial communication protocol, which was used by ControlCenter, only worked over serial modems (or direct RS232 connection) and was as specific to how the original controller worked as the original controller was specific to the one module configuration it supported. No aspect of this communication protocol survives to the present. This ran from the founding of the company until may of 2007.
The second generation controller was was a softcore NIOS processor running at 50MHz on an Altera FPGA, with 128MB RAM and 512MB of flash storage on a CompactFlash card. It had a 640x480 screen run at 8 bits per pixel, and used a touchscreen for input. Owing to the lack of MMU, it ran uClinux. The software for this system was written completely from scratch, and was in Java largely in order to gain the benefits of null pointer checking and array bounds checking, but also for portability of code with the client. The controller was supposed to be able to control a variety of sign panels, and integration with remote control was becoming a tantalizing possibility with the advent of digital cellular communications (and, it was thought, WiFi), so the system was designed as 5 separate components (a display driver, a librarian, a schedular, a (sensor) data source manager, and a catch-all information daemon) which each had its own protocol. Originally, these components talked to each other exclusively through these protocols, over which remote control was also performed. Unfortunately, between the slow processor and the pure java interpreter which was the only one that could run on uClinux, this did not offer sufficient performance, and so the components began to speak to each other directly, but the 5 separate protocols were retained. Unfortunately, since a thread was required for each protocol, that means that 1 conceptual connection to the unit consumed 5 threads. Since threads were expensive in a uClinux environment (in uClinux stacks cannot grow dynamically, and must be pre-allocated), this limited us to approximately 6 simultaneous connections (which, to ensure reliability, were pre-allocated in a pool).
Control Center 3000, which replaced Control Center, was created simultaneously with the second generation NIOS controller, using a large amount of shared code (especially the packet parsing code) to speed development. It was primarily aimed at packet-switched modems which supported TCP connections. It would make a connection to the unit when requested, which it would leave up until the connection was manually closed by the user or the program was closed. After a while, an inactivity timeout was added to Control Center 3000, and in general we would spend in excess of 95% of the time not connected to a unit.
After about a year or so, our belief that we could do without serial modems proved not tolerable to the market, and so a multiplexer protocol was created. It ran in a small python program which was running on the controller, which would accept a multiplexed connection and make the 5 TCP/IP connections to localhost, then proxy the data back and forth. Since it was trivial to do, the multiplexer also accepted connections over TCP/IP (on its own port). The idea was that this might make configuring firewalls easier.
In December of 2010, the third generation of controller was introduced. It ran uClinux on a 125MHz NIOS softcore processor with 256MB RAM and 512MB CF card storage. It had an 800x600 16bpp screen with a touchscreen. The java code was ported without significant change, still running on jamvm, but faster due to the faster processor. As remote communication was getting more popular, shortcomings in the 5-protocol design were becoming apparent, and so a new, single-port secure protocol was designed. It required only 1 port, and consequently 1 thread, per connection, and was (in its normal case) encrypted using AES-128. For simplicity, it was put on port 1, and made use of the TCP/MUX protocol to do protocol selection. This allowed an unencrypted variant to be selected based on the same port. This protocol essentially multiplexed the 5 original protocols, and allowed room for expansion to add other protocols without consuming more resources, which eventually happened in the form of the control protocol, as well as the server-side user-library in Command Center. Once the secure protocol was introduced, we dropped the original 5-protocol thread pools down to allowing 1 (conceptual) connection, and allowed at least 8 simultaneous connections over the secure protocol.
In 2014 the fourth generation of controller was released, this time using an SoC with an 800MHz ARM Cortex-A8 processor, 512MB RAM, and 4GB storage on an SD card. The java software was modified to run on either the second generation of NIOS controller or the new ARM controller, though since the controller runs proper Linux, the JRE was the JITing VM in openjdk. The java software being the same, however, the communications protocols have remained unchange.
The secure protocol has served us very well, but in the Command Center era of constant connections and ultra-low bandwidth, it has several shortcomings.
Experience has shown that the open internet is not very friendly to idle TCP connections. It is virtually impossible to connect to a unit without going through at least one firewall, and possibly more than one, any one of which can do NAT or something else which requires it to keep track of TCP connections, and consequently might be configured to kill off idle TCP connections in order to prevent resource leaks. We've seen that timeout as low as 5 minutes, which means a keep-alive of approximately 4 minutes, which costs a lot of bandwidth. In general, keep-alives cost bandwidth and are problematic because we have to spend time figuring out keep-alives which are shorter than our default. And of course they silently drop the idle TCP connections, rather than sending a FIN packet out to us, so we have no way of knowing that the connection has dropped except by sending data and TCP timing out.
A second problem arises from the nature of TCP timeouts. We need to notice when a unit is no longer connected to the cell network in some reasonable amount of time, so we can't have extremely long timeouts, but on the flip side, this means that intermittent connections become nearly unusuable to us. (This problem could be solved by TCP being configured with huge timeouts, and adding external timers for high priority data only, but given the problem of intermediates silently killing off idle TCP connections, this doesn't seem like a winning strategy.)
TCP's stream-based nature means that processing multiple streams using a single thread requires a complex state machine (especially complex because the secure protocols and its underlying protocols do not include lengths or flag bytes, and some packets are not fixed length, so a parser needs to understand the packets). In practice, in Java, this means using a thread for each connection to a unit. While a single machine should be able to handle this (in the failover case), we don't know what the practical limits really are.
While TCP is not wasteful of bandwidth, it does make tradeoffs which are not optimal for our use case. Due to backwards compatibility issues, traffic tends to originate on the server, which then gets replied to, then acknolwedged by the server, resulting in 3 round trips. While nothing about TCP in itself requires this pattern (traffic could originate on the unit, in which case only 2 trips would be required), work would be needed to change over to this on the current connection approach, which could as easily go into a new and better connection type.
TCP does, however, inherently make no distinction between high and low priority data, so the full bandwidth for acknolwedgements and retries must be applied to low-priority data if it is to be applied to high priority data. (It is possible to maintain two TCP connections, one for high and one for low priority data, but this just brings us back to the problem of keep-alive times costing bandwidth.)
The goals of the connectionless protocol are to maximize reliability over intermitent connections while minimizing bandwidth consumed and trying to ensure rapid packet delivery. Roughly, in that order. It's also supposed to provide encryption, message authentication, and replay protection.
In order to accomplish the primary goal of reliability, we will create a basic packet type which includes sequence numbers so that we can ensure delivery despite an unreliable lower-level connection. Since data comes in two types: transient and permanently interesting, there will be two channels with their own sequences numbers: a high priorty channel and a low priority channel. The high priority channel will be for data whose importance is long-lived, and thus will be persisted to disk and transmission to the servers will be ensured even days later if that is the first opportunity. The low priority channel will be for transient data whose importance is short-lived and which will be forgotten about if the controller reboots, or if delivery isn't possible within a few minutes. Interactive packets, i.e. those coming from the server and the packets generated in response to that, will be sent on the low-priority channel.
The low level delivery will be either UDP/IP or ad-hoc (i.e. +/- one-connection-per-packet) TCP. The ad-hoc TCP/IP connections will be used for large packets and as a fallback for if the UDP delivery method doesn't work.
In order to accomplish the low-bandwidth side of things, we're going to go with a scheme where the controller can be configured to know about the servers it should talk to, so it can originate packets on its own schedule. So that we can support both development and production server pools, each collection of servers will defined on the unit together with a port number to use for communication (both the port to listen on as well as the port to send to, which need not be the same).
In order to keep things simple in this protocol, higher level configuration (which should be rare) will be performed using the secure protocol in its standard TCP/IP configuration (i.e. on TCP port 1), over the Control Protocol.
Encryption is optional, and is configured for the server channel at setup time. It is required on that channel if it is configured. Encryption mandates the use of the 4-byte packet window (to avoid collisions and replay attacks), though the reverse is not true. Encryption is performed with AES-128 while message authentication is performed using MD5-HMAC (the HMAC algorithm makes the collision and extension attacks against the MD5 algorithm irrelevant, and MD5 is well supported even on our legacy hardware). There are two separate keys used for encryption and authentication. The AES-128 key is (obviously) 128 bits in size, while the HMAC key is 512 bits. Both keys are installed onto the unit by the server setting them up for connectionless communication, and are specific to the server channel (the same keys are used for both the low and high priority channels on a particular server channel). There is also a 128 bit initialization vector which is installed onto the unit by the server which is paired with the encryption key.
Encryption is performed on the entire packet (excluding the signature), using AES-128 as the basis for CBC with PKCS5 padding, corresponding to the Java cipher specification of "AES/CBC/PKCS5Padding". The initialization vector is constant, and specified in the encryption setup the server performs. (Since encryption requires the 4-byte sequence number, the initial block will always be different, so we will never have a perfectly repeated packet.)
The HMAC is performed on the entire message, after encryption, and then (as a compromise with bandwidth considerations) the first eight bytes of the HMAC are transmitted.
Byte Count | Data Type | Value | Description | ||||||||||||||||
1 | bitfield | Flags. The flags have the following meanings:
|
|||||||||||||||||
0, 1 or 4 | integer | Sequence Number. (see flags for presence and size) | |||||||||||||||||
0, 1, or 4 | ACK number. (see flags for presence and size) | ||||||||||||||||||
0 or 4 | unsigned integer | Timestamp, unix epoch seconds - 1,400,000,000. (see flags for presence) | |||||||||||||||||
variable | data | The data payload. | |||||||||||||||||
0 or 8 | Signature | (Present if the channel is encrypted.) The first six bytes of the MD5-HMAC of the encrypted message. See Encryption for details. |
If the control packet flag is set, the payload should be interpreted as a control packet, not as a packet to be delivered. Control packets will ordinarily have a sequence number so that delivery can be confirmed.
The first byte of a control packet is a packet ID. Currently the control packets are:
ID | Packet |
0 | Resync Packet |
Byte Count | Data Type | Value | Description | ||||||||||||||||||
1 | byte | 0 | Packet ID | ||||||||||||||||||
1 | flag byte | Flags. Currently:
|
|||||||||||||||||||
4 | unsigned int | new packet number (big endian) | |||||||||||||||||||
Notes: At present resets are only implemented on the low priority channel. Dropping earlier packets refers to packets which were received out-of-order and individually ACKed; instead of deliverying them, they should simply be discarded. For the Drop Earlier Packets flag, this refers to packets lower than or equal to the new packet number. Packets higher than that number should be retained in the received queue. For the drop-all-packets flag, all in-flight packets should be dropped. The drop-all-packets functionality does permit the new packet number to be set to one earlier than the latest delivered packet, but in order to avoid replay attacks, this should not be done on the open internet. If security settings (encryption, etc.) are used, only resets to the future will be honored. |
Each packet will be embedded in a framing protocol which defines the packet's size. UDP does this automatically, and we will use a simple framing protocol for ad-hoc TCP delivery. The size of the payload can thus be calculate from the header and the overall size of the packet. The payload may be omitted entirely in the case of packets whose function is purely to ACK receipt.
Each direction (unit→server and server→unit) has its own set of sequence numbers. The server pool is responsible for synchronizing the packet numbers it sends out.
At least one of the sequence number field and the ACK field must be present, though both may be present in the same packet.
There are two delivery strategies, one for the high priority channel and one for the low priority channel. Both, however, will start off with a random server from the server pool to attempt the first delivery to, and then will alternate thereafter with their attempts for the next 8 hours (i.e. once they get an ACK from a server, they will stick with it until they don't get an ACK, in which case they will cycle through the servers in the server pool until they find another one that ACKs). After 8 hours with the same server, the unit will then randomly pick a server from the server pool to start trying to use (continuing with the same server is a possible outcome). The purpose is to ensure even distribution of traffic among the servers.
The high priority channel is used for spontaneously generated information about the server that we want to know about as soon as possible, and that we might be interested in even if historical. (E.g. current message, GPS position.) For this reason, the packets on the high priority channel are time-stamped, with long-term replay requests being possible (the replay request should be issued with an Information Daemon command).
Though no particular retry schedule is required of the TRAFIX unit, the standard retry schedule is, in seconds: 15, 15, 15, 30, 30, 60, 60, 60, 60, 300, 300, 300, 300, 900, 900, 900, then every hour.
On failure of a packet to be delivered, the unit should attempt re-delivery of only one packet until that succeeds, at which time it should then deliver all packets which had been held up.
The low priority channel is used for spontaneously generated information which is of transient value, such as the current time, temperature, battery voltage, photocell reading, etc. It has no time stamp, it cannot be replayed, and delivery attempts will be abandoned after a few tries. The low priority channel is also used for replies to specific queries, even if the returned data would normally be spontaneously sent over the priority channel.
Though no particular retry schedule is required of the TRAFIX unit, the standard retry schedule is, in seconds: 15, 30, 60, 60, 60, 60, give up.
The network transport is selected on a per-deliver-attempt basis for each packet. Just like multiple UDP attempts may be in flight simultaneously, a UDP attempt and a TCP-based attempt may be in-flight at the same time. For bandwidth reasons, the UDP transport will typically be the first choice, but this is not a strict requirement.
The maximum packet size to be sent via UDP is 1024 bytes. Packets larger than this must use the TCP/IP network transport.
The TCP/IP packet delivery does not persist TCP connections, though it may be used to deliver multiple packets in a single connection at the discretion of both ends. There is a very simple framing protocol which is used. It consists of a length header followed by the packet:
Byte Count | Description |
4 | count. Packet lengths for the connectionless packets being transmitted during this TCP connection. (big endian) |
count | The packet |
More than one packet may be sent on this TCP connection, and packets may be sent in both directions, though a UDP ACK may be given to a packet sent my TCP.