Version 4

Events & Data Sources

Background

The system for incorporating dynamic data into messages on the current controller has worked well, but (due to the hardware it had to work under) it has several limitations:

Proposal

Overview

To solve the above limitations, I propose to create the following networked standard for data sources:

Each controller will run a Source Daemon which acts as the central repository for data sources. All data sources which wish to offer a source to a display on the controller will register with the Source Daemon. Each display driver on the controller, upon loading a sequence, will contact the Source Daemon and request connections from any sources required to display its sequence. Any software for message composition will be able to query the Source Daemon for a list of available sources and their characteristics. Also, the scheduler for each display will contact the Source Daemon and arrange notification of the special events that it wants to know about.

All protocols in this proposal are assumed to be layered over TCP/IP (or some equivalent protocol, such as unix domain sockets) to provide sequenced, reliable communication with multiple ports per host.

Security

Since this proposal defines a set of network protocols, it is important to make sure that everything is secure. Therefore, every protocol is initiated with a challenge-response authentication. Each controller will be assigned an administrator-changeable "network password" that will be used as the initial authentication for every network connection made to the box. (This only applies to the standards which we have defined; obviously the web daemon and the NTCIP daemon will authenticate to their clients in a manner appropriate to them.)

All programs on the local controller will have access to the "network password", so it won't be necessary to configure them specially (and moreover, they will always have guaranteed access to the controller). Only external network sources will need to be told the network password.

Note: the network password is separate from any individual password; it is presumed that any software client which might run on a user's computer will cache this password so that it doesn't need to ask them again.

Source Daemon

There will be one source daemon per controller which will coordinate all data sources for that controller. It will listen for connections from Data Sources on port 30101 and use Data Source to Source Daemon Protocol (DSSDP) to communicate (documented below). It will listen for connections from display drivers on port 30201 and use Display Driver to Source Daemon Protocol (DDSDP) to communicate (documented below). It will listen for connections from schedulers on port 30301 and use Scheduler to Source Daemon Protocol (SSDP) to communicate (again, documented below). It will listen for connections from clients who want a list of the available data sources on port 30401 and use Source Query Protocol (SQP) to communicate (documented below).

Data Sources and Display Drivers

When a Display Driver is loading a sequence which contains dynamic messages, it will contact the Source Daemon and request connections from the Data Sources that it needs to display the sequence. If any of the Data Sources are not available, for the affected messages the Display Driver should display the alternative message specified in the message, or blank if no alternative message is specified. If the Data Source is not available, or if at any time the Display Driver loses contact with the Data Source, it should periodically attempt to request a new connection from the Source Daemon.

Data sources will be able to specify which of two modes of operation they will use. In the first mode, when a Display Driver is about to display a dynamic message, it will ask the Data Source for the data to use in the display. The Data Source will only have a few milliseconds to respond. This mode is suitable for local data sources which do not take long to generate their data, or sources over a very fast network (such as ethernet).

In the other mode, the Data Source will send a data packet whenever it wants to. The Display Driver will maintain a 1 datum FIFO for data packets and whenever it requires a datum to display a dynamic message, it will use the contents of the FIFO. This mode is more suitable for data sources which do not frequently generate data, or which are over a slower connection.

Events, The Source Daemon & The Scheduler

Each data source has an associated signed 32 bit integer which is used to indicate events. The meaning of this integer — if it has a meaning at all — is up to the individual source. Each Data Source is required to describe whatever meaning that it has in its description text. Events will also be time-stamped and carry an indication of low long they are valid for. (The duration of validity will be dependent on how the data is read, etc.)

A Scheduler may indiciate to the source daemon, and the Source Daemon in turn to the Data Source, what data sources it's interested in. Only Data Sources with interested parties should bother to send their events. When a Data Source generates an event it then sends an Event Packet to the Source Daemon, which then forwards it to the appropriate Scheduler(s). The Scheduler then does whatever it wants with this data. (Presumably, it will cause a new sequence to be displayed.)

An integer value was chosen because most data types should be mappable to an integer range. The way to map speed from a radar gun and temperature should be obvious. Other numeric values have obvious mappings, such as the count from a car counter, the weight from a truck scale, or the intensity from a light meter. However, other values should be mappable to an integer by special meanings. For example, if two messageboards are being used in place of flag men, a data source could send 1 for the "let trucks pass" event and 0 for the "tell your trucks to stop" event. (that example is a stretch, since this would be better handled by a single source generating the data sources and a fairly high refresh rate, but hopefully you get the idea.)

The scheduler will store the most recent event packet from every Data Source it's got an event-driven schedules for. The scheduler will use this to display an event for which there is valid data in the correct threshold (unless a time-based sequence has a higher priority). The scheduler is responsible for always examining all of the valid event data to determine which if any events are active.

Data Sources and the Clients who love them

Various clients which present a user interface will want to know what data sources are available in order to present them to the user in a friendly manner. The will connect to the Source Daemon and maintain the connection for as long as they run; the source daemon will notify them of new data sources which become available or disconnected ones which are no longer available as that happens.

Data Sources in Messages

There are fundamentally two different types of messages which can use data sources: text messages and graphical messages (which might include text in text areas). Dynamic text messages will be specified like normal messages with special tags embeded in them. Images will be specified in an XML-like format.

Dynamic Text Messages

Note: the following describes data sources mostly in their implementation; it is assumed that they will typically be entered and edited via a GUI which hides the complexity involved behind a user-friendly front-end.

Data sources will be available to use in text messages by using a special escape sequence and naming the data source. For example, if a radar gun registers a data source called Radar (which provides a string of length 2), a message source which uses it might look like:

YOUR SPEED IS <Radar> MPH

Which might be displayed as:

YOUR SPEED
IS 43 MPH

If a source requires an argument, it may be passed by adding a colon to the source ID and appending the comma-separated arguments, either a string which does not contain a > character, or by enclosing the argument in quotation marks and using standard C-style backslash escaping. While this is available, it is not the recommended idiom, and sources should supply multiple source IDs in preference to using an argument.

An example argument:

YOUR SPEED IS <Radar:units=kph> KPH EH?

Which might be displayed as:

YOUR SPEED
IS 69 KPH
EH?

Alternate Messages

When a Data Source registers a source ID, it must specify the default behavior if it becomes unavailable. If the default is an alternate message, the Data Source must specify a text message to display. The defaults may be overwridden by the message using the source.

To override the default failure behavior of a message, one supplies a text message in square brackets after the source ID, but before any arguments. For example:

YOUR SPEED IS <Radar["DRIVE SAFELY"]> MPH

or, if an argument is also required:

YOUR SPEED IS <Radar["DRIVE SAFELY"]:units=kph> KPH EH?

This should typically not be necessary, as a source should specify an appropriate failure behavior, but it is available to the user if they want something else.

If a user simply wants to skip the message entirely if the source fails, and this is not already the default behavior, they should just supply a blank message. E.g.

YOUR SPEED IS <Radar[]> MPH

This will replace the current message with a brief blank if the data source is unavailable. If the message is the only one in the sequence, that will result in a blank sign. (In most cases, that would be the desired behavior in case of failure.) Note: the same will happen if the data source specifies a blank message as its default alternative message.

Dynamic Graphics Messages

Dynamic graphical messages will use an XML-like syntax (xml-like because it will not use a DTD or allow recursion, so that parsing does not require a recursive descent parser (or stronger)).

A dynamic graphic must begin with:

<graphic sourceID="sourceID">

Next, it may contain an override of the alternate message to display, in the form:

<alternate>alternate text</alternate>

Alternates, even for graphical sources, are restricted to static text messages for simplicity. Alternates should never actually be used — they represent a failure condition — so they don't need to be fancy; they only need to be functional.

Next, it may contain a parameter in the form:

<param name="key">value</param>

After this, one may specify the text to insert into one or more text areas defined by the source, in the form:

<textarea>text for textarea 1</textarea>
<textarea>text for textarea 2</textarea>
<textarea>etc.</textarea>

Textareas are ordered by their upper-left coordinate, from top-to-bottom, right-to-left. Any text specified for a textarea which is not defined by the source is simply ignored. (I.e. if a source defined 2 textareas and a message specifies 3 textarea strings, the third is silently ignored.) (Note: silently ignoring extraneous strings is not recommended for the input gui, only for the Display Driver.)

Finally, a dynamic graphic must be terminated by:

</graphic>

Protocols

The Data Source family of protocols are versioned, variable-sized packet protocols. They all begin with a challenge-response authentication phase (prior to any handshaking or version exchange) where the listener issues a unique 20 byte challenge. The correct response is the 20 byte SHA1 hash of the concatenation of the challenge and the secret password. (i.e. SHA1(challenge + password).) The initiator of the connection only gets one chance; if they do not return the correct response, the connection is terminated. Also, the connection is terminated if the complete response is not recieved within 10 seconds. (That should never be an issue since virtually all connections which don't come from the local machine will come via a high-speed network (ethernet, 802.11b, etc.), but this is here to deter DOS attacks.)

Note: unless otherwise specified, all protocols use network ordered (i.e. big-endian) orderings of multi-byte integeral types. Further, the data type String is defined as an unsigned short indicating length, followed by that many bytes of UTF-8 data.

Common Packets

The following packets may be used in more than one protocol:

Text Source Information Packet
Byte Count Data Type Value Description
1 Unsigned Byte 0 Packet Type
2-65,537 String The source ID
2-65,537 String A human-readable description of what data the source provides, as well as what its event number means (if it generates any meaningful event numbers).
1 unsigned byte The minimum length of the supplied text.
1 unsigned byte The maximum length of the supplied text.
Graphics Source Information Packet
Byte Count Data Type Value Description
1 Unsigned Byte 1 Packet Type
2-65,537 String The source ID
2-65,537 String A human-readable description of what data the source provides, as well as what its event number means (if it generates any meaningful event numbers).
2 unsigned short The minimum width of the supplied graphic.
2 unsigned short The minimum height of the supplied graphic.
2 unsigned short The maximum width of the supplied graphic.
2 unsigned short The maximum height of the supplied graphic.
Event Filter Packet
Byte Count Data Type Value Description
1 Unsigned Byte 2 Packet Type
2-65,537 String The source ID
Event Filter Cancellation Packet
Byte Count Data Type Value Description
1 Unsigned Byte 5 Packet Type
2-65,537 String The source ID
Event Packet
Byte Count Data Type Value Description
1 Unsigned Byte 3 Packet Type
2-65,537 String The source ID
4 signed integer The numerical value of the event. What this means is source-dependent.
8 signed long the timestamp in epoch milliseconds
4 unsigned integer the duration (in milliseconds) which this packet is valid for.
Data Source Connection Request Packet
Byte Count Data Type Value Description
1 Unsigned Byte 4 Packet Type
2-65,537 String The URI for the Data Source to connect to.
1 Unsigned Byte Data Source block Count
variable The data sources desired. Each block is of the form:
2-65,537 String The source ID
2-65,537 String The argument string.
The source daemon is responsible for forwarding this packet to as many data source providers as necessary.

Note: the data source will be responsible for processing the argument; the display driver should send it unaltered.
Source Unavailable Packet
Byte Count Data Type Value Description
1 Unsigned Byte 12 Packet Type
2-65,537 String The source ID for the source which is no longer available.

Data Source to Source Daemon Protocol (DSSDP)

The protocol is a versioned, variable-length packeted protocol which begins with a handshake. The handshake packets do not contain a packet type because their required order make it redundant.

The handshake packets are, in order:

Source Daemon -> Data Source Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The maximum version of DSSDP supported.
Data Source -> Source Daemon Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The version of DSSDP to be used. This must be <= the maximum supported version sent by the Source Daemon. If the Source Daemon cannot support a version great enough, the Data Source should simply drop the connection and report an error.

At this point, the Data Source may send the following packets at any time:

And the Source Daemon may send any of the following packets at any time:

If there are ever zero Event Filters registered for a source, it should not send any events. This will be the case on startup until the Source Daemon sends an Event Filter Packet, and any time all Event Filters have been canceled with Event Filter Cancellation Packets.

Display Driver Source Daemon Protocol (DDSDP)

The protocol is a versioned, variable-length packeted protocol which begins with a handshake. The handshake packets do not contain a packet type because their required order make it redundant.

The handshake packets are, in order:

Source Daemon -> Display Driver Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The maximum version of DDSDP supported.
Display Driver -> Source Daemon Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The version of DDSDP to be used. This must be <= the maximum supported version sent by the Source Daemon. If the Source Daemon cannot support a version great enough, the Data Source should simply drop the connection and report an error.

At this point, the Display Driver may send the following packets:

Note: There is no notification from the Source Daemon to the Display Driver as to what sources are available because the Display Driver can't do anything with this information anyway. All it cares about is whether it has a connection from a data source which is offering the sourceID that it wants. It will try to get these connections, but so long as it doesn't have them, it's entirely irrelevant to the Display Driver why it doesn't have them.

It is, of course, possible for the Source Daemon to generate warning messages about requests for non-existant sources to help detect stolen radar guns, but that behavior is outside of the scope of this specification.

Scheduler to Source Daemon Protocol (SSDP)

The protocol is a versioned, variable-length packeted protocol which begins with a handshake. The handshake packets do not contain a packet type because their required order make it redundant.

The handshake packets are, in order:

Source Daemon -> Scheduler Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The maximum version of SSDP supported.
Scheduler -> Source Daemon Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The version of SSDP to be used. This must be <= the maximum supported version sent by the Source Daemon. If the Source Daemon cannot support a version great enough, the Scheduler should simply drop the connection and report an error.

At this point, the Scheduler may send the following packets at any time:

The Source Daemon may send the following packets at any time:

Source Query Protocol (SQP)

The protocol is a versioned, variable-length packeted protocol which begins with a handshake. The handshake packets do not contain a packet type because their required order make it redundant.

The handshake packets are, in order:

Source Daemon -> Client Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The maximum version of SQP supported.
Client -> Source Daemon Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The version of SQP to be used. This must be <= the maximum supported version sent by the Source Daemon. If the Source Daemon cannot support a version great enough, the client should simply drop the connection and report an error.

The client may not send any packets. The Source Daemon may send any number of Text Source Information Packets, Graphics Source Information Packets, or Source Unavailable Pakets.

Data Source to Display Driver Protocol (DSDDP)

The protocol is a versioned, variable-length packeted protocol which begins with a handshake. The handshake packets do not contain a packet type because their required order make it redundant.

The handshake packets are, in order:

Display Driver -> Data Source Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The maximum version of DSDDP supported.
Data Source -> Display Driver Handshake Packet
Byte Count Data Type Value Description
1 unsigned byte 0 The version of DSDDP to be used. This must be <= the maximum supported version sent by the Display Driver. If the Display Driver cannot support a version great enough, the Data Source should simply drop the connection and report an error.

The Display Driver will then immediately send a Display Driver Description Packet. After receiving a Data Source Description Packet, the Display Driver may optionally send an Argument Packet to it.

Display Driver Description Packet
Byte Count Data Type Value Description
1 Unsigned Byte 7 Packet Type
2 unsigned short The width of the display (in pixels)
2 unsigned short The height of the display (in pixels)
1 unsigned byte The bits-per-pixel used by this display (note: this is not a restriction; images can be up- or down-converted as necessary; this is just to allow the transfer to be maximally efficient.)
1 unsigned byte The bits-per-color used by this display (note: this is not a restriction; images can be up- or down-converted as necessary; this is just to allow the transfer to be maximally efficient.)
Data Source Description Packet
Byte Count Data Type Value Description
1 Unsigned Byte 6 Packet Type
2-65,537 String The source ID
1 unsigned byte Data type:
0 Text Data
1 Graphical Data
1 unsigned byte Transfer mode:
0 request-as-needed mode
1 FIFO buffer/source push mode
2-65,537 String The text of the alternate message
Argument Packet
Byte Count Data Type Value Description
1 Unsigned Byte 8 Packet Type
2-65,537 String The source ID
2-65,537 String The argument string. Note: the data source will be responsible for processing the argument; the display driver should send it unaltered.
Data Request Packet
Byte Count Data Type Value Description
1 Unsigned Byte 9 Packet Type
2-65,537 String The source ID
Text Data Packet
Byte Count Data Type Value Description
1 Unsigned Byte 10 Packet Type
2-65,537 String The source ID
2-65,537 String The text data
Graphics Data Packet
Byte Count Data Type Value Description
1 Unsigned Byte 11 Packet Type
2-65,537 String The source ID
variable Image The image data

Note: If the Data Source selected request-as-needed mode, it has 10ms to respond to a Data Request Packet before it is considered unavailable and the alternate message is used. If it sends its Data Packet more than 10ms after it recieves the Data Request Packet, the data packet will be silently ignored.


Appendix A. Packet types

Packet Type # Packet Type
0 Text Source Information Packet
1 Graphics Source Information Packet
2 Event Filter Packet
3 Event Packet
4 Data Source Connection Request Packet
5 Event Filter Cancellation Packet
6 Data Source Description Packet
7 Display Driver Description Packet
8 Argument Packet
9 Data Request Packet
10 Text Data Packet
11 Graphics Data Packet
12 Source Unavailable Packet

Last modified: Thu Jun 2 09:10:03 EDT 2005