Revision 3

Events & Data Sources

NOTE: This document covers what is technically Version 2 of the data source protocol. Things are different enough that it's been split into its own document.

Overview

The Source System provides a comprehensive way for external properties to affect the controller, both in what it does and what it shows in messages. It provides both for Data Sources — sources of data to be used in messages — as well as Event Sources — sources of data that can be used to schedule messages. (Displaying the current reading from the radar gun uses data sources; showing a message when the battery voltage falls below 11 volts uses events.)

At the heart of the source system is the Source Daemon, which runs on the controller and acts as a nexus between the data providers and data consumers. Data providers can be located either on the controller or on other hosts that provide their data and events across a network connection. The Scheduler on the controller will be the typical consumer of events, while the Display Driver(s) rendering dynamic messages will be the typical consumer of data sources. Desktop control software will also consume both data sources and events from the controller, however.

Data Sources in Messages

There are fundamentally two different types of messages which can use data sources: text messages and graphical messages (which might include text in text areas). Dynamic text messages will be specified like normal messages with special tags embeded in them. Images will be specified in an XML-like format.

Dynamic Text Messages

Note: the following describes data sources mostly in their implementation; it is assumed that they will typically be entered and edited via a GUI which hides the complexity involved behind a user-friendly front-end.

Data sources will be available to use in text messages by using a special escape sequence and naming the data source. For example, if a radar gun registers a data source called Radar (which provides a string of length 2), a message source which uses it might look like:

YOUR SPEED IS <Radar> MPH

Which might be displayed as:

YOUR SPEED
IS 43 MPH

If a source requires an argument, it may be passed by adding a colon to the source ID and appending the comma-separated arguments, either a string which does not contain a > character, or by enclosing the argument in quotation marks and using standard C-style backslash escaping. While this is available, it is not the recommended idiom, and sources should supply multiple source IDs in preference to using an argument.

An example argument:

YOUR SPEED IS <Radar:units=kph> KPH EH?

Which might be displayed as:

YOUR SPEED
IS 69 KPH
EH?

Alternate Messages

When using a Data Source in a message, the author of the message has the option to specify what message should be shown if the data source is unavailable. To do this, one supplies a text message in square brackets after the source ID, but before any arguments. For example:

YOUR SPEED IS <Radar["DRIVE SAFELY"]> MPH

or, if an argument is also required:

YOUR SPEED IS <Radar["DRIVE SAFELY"]:units=kph> KPH EH?

This should typically not be necessary, as a source should specify an appropriate failure behavior, but it is available to the user if they want something else.

If a user simply wants to skip the message entirely if the source fails, and this is not already the default behavior, they should just supply a blank message. E.g.

YOUR SPEED IS <Radar[]> MPH

This will replace the current message with a brief blank if the data source is unavailable. If the message is the only one in the sequence, that will result in a blank sign. (In most cases, that would be the desired behavior in case of failure.) Note: the same will happen if the data source specifies a blank message as its default alternative message.

Dynamic Graphics Messages

Dynamic graphical messages will use an XML-like syntax (xml-like because it will not use a DTD or allow recursion, so that parsing does not require a recursive descent parser (or stronger)).

A dynamic graphic must begin with:

<graphic sourceID="sourceID">

Next, it may contain an override of the alternate message to display, in the form:

<alternate>alternate text</alternate>

Alternates, even for graphical sources, are restricted to static text messages for simplicity. Alternates should never actually be used — they represent a failure condition — so they don't need to be fancy; they only need to be functional.

Next, it may contain a parameter in the form:

<param name="key">value</param>

After this, one may specify the text to insert into one or more text areas defined by the source, in the form:

<textarea>text for textarea 1</textarea>
<textarea>text for textarea 2</textarea>
<textarea>etc.</textarea>

Textareas are ordered by their upper-left coordinate, from top-to-bottom, right-to-left. Any text specified for a textarea which is not defined by the source is simply ignored. (I.e. if a source defined 2 textareas and a message specifies 3 textarea strings, the third is silently ignored.) (Note: silently ignoring extraneous strings is not recommended for the input gui, only for the Display Driver.)

Finally, a dynamic graphic must be terminated by:

</graphic>

Events

Each event source uses a signed 32 bit integer to indicate events. The meaning of this integer — if it has a meaning at all — is up to the individual source. Each Event Source is required to describe whatever meaning that it has in its description text. Events will also be time-stamped and carry an indication of low long they are valid for.

An event Source may provide more than one event over the same connection to the Source Daemon. (This is the encouraged model.)

A client may subscribe to one or more event sources. The client should only receive event packets for the sources to which it is subscribed. When an Event Source generates an event it then sends an Event Packet to the Source Daemon, which then forwards it to the appropriate clients. The client (most often a Scheduler) then does whatever it wants with this data. (Presumably, it will cause a new sequence to be displayed.)

An integer value was chosen because most data types should be mappable to an integer range. The way to map speed from a radar gun and temperature should be obvious. Other numeric values have obvious mappings, such as the count from a car counter, the weight from a truck scale, or the intensity from a light meter. However, other values should be mappable to an integer by special meanings. For example, if two message boards are being used in place of flag men, a data source could send 1 for the "let trucks pass" event and 0 for the "tell your trucks to stop" event. (that example is a stretch, since this would be better handled by a single source generating the data sources at a fairly high refresh rate, but hopefully you get the idea.)

The scheduler will store the most recent event packet from every Data Source it's got an event-driven schedules for. The scheduler will use this to display an event for which there is valid data in the correct threshold (unless a time-based sequence has a higher priority). The scheduler is responsible for always examining all of the valid event data to determine which if any events are active.

The Source Daemon

There will be one source daemon per controller which will coordinate all data sources for that controller. It will listen for connections on port 30101.

The Source Daemon will also provide a local interface for other threads running in the same process (JVM) to bypass the network-oriented nature of this protocol (this interface will allow high-performance zero-allocation use of the source system). All clients which connect using the local interface will still be available — exactly as if they were connected via the network — to network-connected clients.

The Source Daemon will buffer all data and events coming from the data sources, so that the data source will send its data to the Source Daemon (once), and the Source Daemon will forward that data on to all of the clients which have subscribed to the data source.

Determining Available Sources

Clients with a user interface which present the user with a list of available sources will naturally need a way to find out what sources are available. This is somewhat complicated by the fact that sources can come and go while the system is running.

As a result, clients notify the Source Daemon of their interest in the type of sources they want (data, event, or both) and then must handle the notifications of sources and source removal. There's no packet listing the sources; since the interface must support dynamic adding and removal, it just uses that interface to show the sources as it gets notified of them. (When it first registers interest, the source daemon will immediately, for each of the currently available data sources, send a notification packet.)

Handling Arguments

Since each data source sends it packets to the Source Daemon, and the Source Daemon relays them to the clients, arguments are clearly problematic.

The solution is that the source daemon will forward argument packets on to the source daemon, and the source daemon will provide one data packet per unique argument; the data packet will include the argument that it's for. The Source Daemon will then forward the data packet on to the correct client. (When a client disconnects, the Source Daemon will inform the data source to stop generating data for the client's argument.)

Numeric Source IDs

For efficiency, packets which are sent frequently will use numeric IDs, rather than textual IDs, for their source and argument. This text-to-number mapping will be maintained by the Source Daemon, who will notify both data sources and source consumers what the mappings are (for the sources that they provide or express interest in).

These mappings are dynamically assigned by the Source Daemon, and are not reliable from one time to another, let alone across reboots or across different units. These numbers should be transparent to the user, and are only an implementation detail.

To be clear, every source/argument pair will receive its own dynamic Subscription ID.

Endpoint Roles

Even though the protocol is symmetrical (anyone connected may send any packet, and anyone using this protocol should be able to receive any packet, even if it just discards it), there are three distinct roles in the protocol: the Source Daemon, a Data Source/Event provider, and a data source/event consumer (typically, a message renderer or a scheduler).

The Source Daemon

The Source daemon receives nearly every packet and sends just about every packet. Someone implementing a source daemon needs to be intimately familiar with all of the packets, so it's role is not detailed here.

Note: the skip logic in subscriptions is always handled by the source daemon, and not by the source itself.

Data/Event Source Providers

A Data Source provider may receive:

A Data/Event Source provider may send:

When they first connect, Data & Event sources should send the appropriate description packet for each source ID they are supplying.

Sources should then wait for a subscription packet which provides them with a subscription ID before sending any data or event packets.

Schedulers/Event Consumers

An event consumer may receive:

An event consumer may send:

Scheduling User Interfaces

An scheduling user interface may receive:

An event consumer may send:

Message Composition Interfaces

A message composition interface may receive:

A message renderer may send:

Message Renderers

A message renderer may receive:

A message renderer may send:

The Protocol

Protocol Handshake

The protocol handshake works according to the Solartech Protocol Handshake & Authentication specification. The current protocol version is 2.

NOTE: since version 1 support is so complex, servers are not required to implement version 1, and may just disconnect if a client requests version 1.

The Packets

The Event & Data Source Protocol is a versioned, variable-sized packet-based protocol.

Note: unless otherwise specified, all protocols use network ordered (i.e. big-endian) orderings of multi-byte integeral types. Further, the data type String is defined as an unsigned short indicating length, followed by that many bytes of UTF-8 data.

Textual Data Source Description Packet (C←S, S←D)
Byte Count Data Type Value Description
1 Unsigned Byte 0 Packet Type
2-65,537 Java UTF-8 String The source ID
2-65,537 Java UTF-8 String A human-readable description of what data the source provides.
1 unsigned byte The minimum length of the supplied text.
1 unsigned byte The maximum length of the supplied text.
1 unsigned byte Arguments are supported by this source (0 == not supported, 1 == supported)
Graphical Data Source Description Packet (C←S, S←D)
Byte Count Data Type Value Description
1 Unsigned Byte 1 Packet Type
2-65,537 String The source ID
2-65,537 String A human-readable description of what data the source provides, as well as what its event number means (if it generates any meaningful event numbers).
2 unsigned short The minimum width of the supplied graphic.
2 unsigned short The minimum height of the supplied graphic.
2 unsigned short The maximum width of the supplied graphic.
2 unsigned short The maximum height of the supplied graphic.
1 unsigned byte Arguments are supported by this source (0 == not supported, 1 == supported)
Event Subscription Packet (CS S→D)
Byte Count Data Type Value Description
1 Unsigned Byte 2 Packet Type
2-65,537 String The source ID
1 unsigned byte Count of events to skip.
2 unsigned short Subscription ID (see notes)
Notes:
A skip-count of 0 means to send every packet, a skip-count of 1 means to send every other packet, a skip-count of 2 means to send every third packet, and so on.

The Source Daemon will respond to this packet with one of the subscription success packets (which gives the numeric ID that this event will be referred to by) or a source unavailable packet.

A source consumer sending this packet to the Source Daemon should leave the Subscription ID field blank. A data source receiving this packet from a Source Daemon should use the Subscription ID when sending its event packets.
Event Packet (C←S, S←D)
Byte Count Data Type Value Description
1 Unsigned Byte 3 Packet Type
2 unsigned short Subscription ID
4 signed integer The numerical value of the event. What this means is source-dependent.
8 signed long the timestamp in epoch milliseconds
4 unsigned integer the duration (in milliseconds) which this packet is valid for.
Notes:
A duration of 0 indicates that the packet has no expiration date, and should be considered to be always active.
Data Source Subscription Packet (CS, S→D)
Byte Count Data Type Value Description
1 Unsigned Byte 4 Packet Type
1 unsigned byte Skip Count
2 unsigned short The width of the display (in pixels)
2 unsigned short The height of the display (in pixels)
1 unsigned byte The bits-per-pixel used by this display (note: this is not a restriction; images can be up- or down-converted as necessary; this is just to allow the transfer to be maximally efficient.)
1 unsigned byte The bits-per-color used by this display (note: this is not a restriction; images can be up- or down-converted as necessary; this is just to allow the transfer to be maximally efficient.)
1 Unsigned Byte Data Source block Count
variable The data sources desired. Each block is of the form:
Data Source Block
2-65,537 String The source ID
2-65,537 String The argument string.
2 unsigned short Subscription ID (see notes)
Notes::
The display information is meant for graphical data sources, but since this is a generic packet, it's included for all data subscription requests. If it's known that all of the data sources being subscribed to are textual, it is safe to set those fields to 0.

A skip-count of 0 means to send every packet, a skip-count of 1 means to send every other packet, a skip-count of 2 means to send every third packet, and so on.

A source consumer sending this packet to the Source Daemon should leave the Subscription ID field blank. A data source receiving this packet from a Source Daemon should use the Subscription ID when sending its event packets.
Event Subscription Cancellation Packet (CS)
Byte Count Data Type Value Description
1 Unsigned Byte 5 Packet Type
2-65,537 String The source ID
Text Data Packet (C←S, S←D)
Byte Count Data Type Value Description
1 Unsigned Byte 10 Packet Type
2 unsigned short The Subscription ID
2-65,537 String The text data
Graphics Data Packet (C←S, S←D)
Byte Count Data Type Value Description
1 Unsigned Byte 11 Packet Type
2 unsigned short The Subscription ID
variable Image The image data
Notes:
The image height and width are used to match the data source to the client.
Source Unavailable Packet (CS)
Byte Count Data Type Value Description
1 Unsigned Byte 12 Packet Type
2-65,537 String The source ID for the source which is no longer available.
Notes:
The source unavailable packet is sent from a Source Daemon in response to a subscription request for a source ID which no data source provides. If a client gets a Source Unavailable Packet, it means that the subscription was not made. If the client still wants the subscription, it must try again (there won't be any notification that the source was added, the client simply has to poll until it succeeds).

This packet is also used to notify clients who've asked for notification of available data or event sources that a data source has left.
Data Source Notification Request Packet (CS)
Byte Count Data Type Value Description
1 Unsigned Byte 13 Packet Type
Notes:
This packet, when sent from the client to the server, causes the server to send a description packet for all connected data sources, as well as all new data sources that connect.
Event Source Notification Request Packet (CS)
Byte Count Data Type Value Description
1 Unsigned Byte 14 Packet Type
Notes:
This packet, when sent from the client to the server, causes the server to send a description packet for all connected event sources, as well as all new event sources that connect.
Event Source Description Packet (C←S, S←D)
Byte Count Data Type Value Description
1 Unsigned Byte 15 Packet Type
2-65,537 Java UTF-8 String The source ID
2-65,537 Java UTF-8 String A human-readable description of what the event is for and what the integer means.
Data Source Subscription Cancellation Packet (CS, S→D)
Byte Count Data Type Value Description
1 unsigned byte 17 Packet Type
2 unsigned short Subscription ID
Notes:
When the Source Daemon receives this packet, it cancels the subscription of the client. If the argument has non-zero length, the Source Daemon forwards this on to the Data Source.

If a client who subscribed with an argument disconnects, the Source Daemon is responsible for generating the appropriate subscription cancellation packets and sending them to the appropriate data source so that the data sources stop sending argument-specific packets.
Subscription Success Packet (CS)
Byte Count Data Type Value Description
1 Unsigned Byte 18 Packet Type
2-65,537 String The source ID for the source which was subscribed to.
2-65,537 String The arguments for the subscription.
2 unsigned short The Subscription ID for this subscription.
Notes:
Receipt of this packet only means that the source is connected to the Source Daemon and that the Source Daemon has the available resources for the subscription. It does not guarantee that the arguments were acceptable; it is possible that receive an invalid argument packet (forwaded from the source provider) immediately after a subscription success packet.

(The main function of the subscription success packet is to convey subscription IDs, not to guarantee success.)
Invalid Argument Packet (CS, S←D)
Byte Count Data Type Value Description
1 unsigned byte 19 Packet Type
2-65,527 Java UTF-8 String Source ID
2-65,527 Java UTF-8 String Argument
2-65,527 Java UTF-8 String Human-readable error
Notes:
Source Configuration Command Packet (SD)
Byte Count Data Type Value Description
1 unsigned byte 20 Packet Type
2-65,527 Java UTF-8 String Source ID
2-65,527 Java UTF-8 String Command / configuration variable
2-65,527 Java UTF-8 String argument / value
Notes: (This is a protocol version 18 packet.) There is no standardization on the meaning of the argument string; that is up to the data source and must be described by that data source in documentation.

Changes from Version 1

In the original version, the source daemon acted as a broker between data sources and display drivers, arranging for data sources to connect to display drivers. This model proved to be inefficient and overly complex.

In the new version of the protocol, the source daemon acts as a middle-man, taking the packets from the data sources and relaying them to the display drivers who are interested in them. This has several benefits:

  1. No one needs to have a listening TCP/IP port (except for the source daemon, who needed it already).
  2. No additional connections need to be made past the connections to the source daemon.
  3. Changing to only one model for data source aquisition will result in substantially simplified code paths.

In addition, everyone will use the same protocol, simplifying the number of protocol implementations. This protocol should also be marginally simpler than the previous protocol.

This document is meant to entirely supercede the previous specification. Since it is only a revision of the protocol, however, the old version is still around for systems which need to implement version 1 functionality.


Last modified: Thu Jun 2 09:10:03 EDT 2005