IRIG106wiki

Chapter 10 data files are organized as a sequential series of data packets. Each data packet can only contain one type of data (i.e. 1553, Pulse Code Modulation (PCM), etc.). Data packets frequently contain multiple individual data messages.

The Chapter 10 standard requires that the first data packet in a data file be an IRIG 106 Chapter 9 format, Telemetry Attributes Transfer Standard (TMATS) packet. The TMATS packet is used to configure the recorder and to describe the data being recorded. Also, TMATS is stored in a Computer Generated Data, Format 1 (Data Type = 0x01) data packet and is discussed in paragraph 6.5.2.

An important field in the packet header for the TMATS packet is the “IRIG 106 Chapter 10 Version” field. The value of this field determines the specific version of the Chapter 10 standard to use when interpreting the rest of the data file. This value should not be confused with the Data Type Version (but sometimes called Header Version) discussed in paragraph 6.3. The IRIG 106 Version field is only defined for the IRIG 106-07 and later versions of the IRIG 106 standard. Since unused reserved fields must be initialized to zero, this field will have a value of 0 in data files compliant with IRIG 106 prior to IRIG 106-07.

Starting with publication of IRIG 106-07, more than one TMATS packet is allowed at the beginning of the file if recorder configuration information exceeds the packet size limit.

It is required that a Time Data packet be the first “dynamic” data packet. Dynamic data packets are not defined in the Chapter 10 standard but it is generally understood that a dynamic data packet means any data packet other than Computer Generated Data Format 1 (setup record). The purpose of this requirement is to allow the association of clock time with the Relative Time Counter (RTC) before encountering the first data packet in the data file. Programmers are cautioned to verify a valid time packet has been received before attempting to interpret the RTC as described in paragraph 6.6.

A root index packet will be the last data packet in the data file if file indexing is used. The presence of data file indexing is indicated in the TMATS setup record.

The size of all data packets is an integer multiple of 4 bytes (32 bits) with the maximum size of 524,288 bytes. Computer Generated Data, Format 1 (TMATS) packets have a maximum packet size of 134,217,728 bytes. To provide this 4 byte alignment, padding bytes are added, if necessary, to the end of a data packet, just before the checksum. Regardless, when defining data structures representing Chapter 10 data packets and messages, these structures should be forced to be byte aligned by using the appropriate compiler directive.

Some data packet elements are two or more bytes in length. For example, the first data element of a data packet is a two byte sync pattern. Multiple byte data elements are stored in little endian format. That is, the least significant portion of the data is stored at the lowest byte offset.

Data packets are written to disk roughly in the time order in which they are received, but data packets and data messages can occur in the data file out of time order. This order can occur because data recorders receive data simultaneously on multiple channels, each channel buffering data for a period of time and then writing it to disk. Therefore, individual data messages will, in general, be somewhat out of time order because of grouping by channel. Consider the case of two 1553 channels recording the same bus at the same time in an identical fashion. Each channel receives, buffers, and writes data to disk. The first channel will write its buffered data to disk followed by the second channel. The data received from the second channel will be from the same time period as the data from the first channel and will have identical time stamps but will be recorded after the first channel in the data file.

Starting with the IRIG 106-05 standard, recorders are only allowed to buffer data for a maximum of 100 milliseconds and data packets must be written to disk within one second. This ensures that data packets can only be out of time order by a maximum of one second. Be warned, though, that the maximum amount of time data packets can be out of order for data files produced before IRIG 106-05 is unbounded and it is not unusual to encounter data files with data packets five or more seconds out of time order.

Example source code that demonstrates basic parsing of Chapter 10 data files can be found in Appendix A. An example program that demonstrates reading and interpreting a Chapter 10 file can be found in Appendix B.

Overall data packet organization is shown below. Data packets contain a standard header, a data payload containing one or multiple data messages, and a standard trailer. The standard header is composed of a required header, optionally followed by a secondary header. The data payload generally consists of a Channel Specific Data Word (CSDW) record followed by one or more data messages.

Data packets must contain data. They are not allowed to only contain filler. Filler can be inserted into a data packet in the packet trailer before the checksum. This filler is used to ensure data packet alignment on a four byte boundary. Filler is also sometimes used to keep the same length of packets from a particular channel. The standard does not expressly prohibit filler after the packet trailer but before the next data packet header; however, inserting filler after the last trailer is considered bad practice. Still, when reading data packets, the programmer should set read buffer sizes based on the value of the overall packet length found in the header. Do not make assumptions about packet length based on the data length or from information in the data payload.

When reading linearly through a Chapter 10 data file, maintaining synchronization with data packet boundaries is accomplished by using the packet length field in the header to read the appropriate amount of data or to reposition the read pointer to the beginning of the next header. In this case, it is sufficient to check the value of the Sync field at the beginning of the header to ensure the read pointer was positioned to the beginning of a data packet.

If there is an error in the data file, or if the read pointer is repositioned to a position other than the beginning of a data packet (for example to jump to the middle of a recorded data file), then the beginning of a valid data packet must be found. Unfortunately the Chapter 10 standard does not provide a way to definitively determine the beginning of a data packet in these instances. Instead, some heuristics must be applied:

1. Read the data file until the packet sync pattern (0xEB25) is found. Normally the first character of the packet sync pattern is found at a file offset which is an integer multiple of four. However, if the data file is corrupted then the sync pattern may not fall on the normal four byte boundary. Scan the file a byte at a time, ignoring the normal four byte alignment. When the Sync pattern is found then
2. Calculate and test the header checksum.
3. If a secondary header exists, calculate and test the secondary header checksum.
4. Calculate and test the data checksum.

If the packet sync pattern is found and all available checksums have been verified, then there is a high probability that the beginning of the next valid data packet has been found.

The packet header contains information about the data payload such as time, packet length, data type, data version, and other information. The layout of a Chapter 10 packet header is shown below.

struct SuI106Ch10Header 
    { 
    uint16_t    uSync;              // Packet Sync Pattern 
    uint16_t    uChID;              // Channel ID 
    uint32_t    ulPacketLen;        // Total packet length 
    uint32_t    ulDataLen;          // Data length 
    uint8_t     ubyDataVer;         // Data Version 
    uint8_t     ubySeqNum;          // Sequence Number 
    uint8_t     ubyPacketFlags;     // Packet Flags 
    uint8_t     ubyDataType;        // Data type 
    uint8_t     aubyRelTime[6];     // Reference time 
    uint16_t    uChecksum;          // Header Checksum 
    }; 

The Channel ID field uniquely identifies the source of the data. The value of the Channel ID field corresponds to the Track Number value of the TMATS “R” record, “R-m\TK1.” The Channel ID field is a 16-bit field. However, the Chapter 9 TMATS format restricts the value of Channel ID to a two digit number (i.e. from 0 to 99). It is anticipated that this TMATS restriction will be lifted in future IRIG 106 standard releases.

Typically, only one packet data type is associated with a particular Channel ID, but this is not a requirement of the Chapter 10 standard. An exception to this is Channel ID = 0, the Channel ID used for internal, computer generated format data packets. It is typical for Channel ID 0 to contain Computer Generated Data Format 1 Setup Records (0x01), Computer Generated Data Format 2 Recording Events Records (0x02), and Computer Generated Data Format 3 Recording Index Records (0x03).

The data payload format is interpreted based on the value of the Data Type field and the Data Version field in the packet header. This field is sometimes incorrectly called “Header Version.” Each packet data payload can only contain one type of data (e.g. 1553, PCM, etc.). A Chapter 10 standard release will only contain data format and layout information for the latest Data Version. The specific Data Version defined in a particular Chapter 10 release can be found in the “Data Type Names and Descriptions” table. Be warned that future Chapter 10 releases may update or change data format or layout, indicated by a different Data Version value in the header, but the Chapter 10 release will not have information about the previous Data Versions. That information can only be found in the previous Chapter 10 releases.

When processing a data file, it is common to only read the data packet header, determine if the data portion is to be read (based on packet type or other information gleaned from the header), and, if not to be read, skip ahead to the next header. Skipping the data portion and jumping ahead to the next header is accomplished by using the packet length in the packet header. Below is the algorithm for determining how many bytes to jump ahead in the file byte stream to reposition the read pointer to the beginning of the next header:

a. Read the current primary header

b. - Determine relative file offset to the next header

Offset = Packet Length - Primary Header Length (24) - Secondary Header Length (12) (if included)

c. Move read pointer

The optional secondary header is used to provide an absolute time (i.e. clock time) stamp for data packets. The secondary header time format can be interpreted several ways. The specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure below is used when secondary header time is to be interpreted as a Chapter 4 format value (Flag Bits 3-2 = 0). The following structure is used when secondary header time is to be interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1).

struct SuI106Ch10SecHeader_Ch4Time 
    { 
    uint16_t uUnused;       // 
    uint16_t uHighBinTime;  // High order time 
    uint16_t uLowBinTime;   // Low order time 
    uint16_t uUSecs;        // Microsecond time 
    uint16_t uReserved;     // 
    uint16_t uSecChecksum;  // Secondary Header Checksum 
    }; 

struct SuI106Ch10SecHeader_1588Time 
    { 
    uint32_t uNanoSeconds;  // Nano-seconds 
    uint32_t uSeconds;      // Seconds 
    uint16_t uReserved;     // 
    uint16_t uSecChecksum;  // Secondary Header Checksum 
    }; 

After the standard header and optional secondary header, each data packet begins with CSDW(s). The length of the CSDW varies depending on the data type. For example, Analog Data Format 1 may have multiple CSDWs. The CSDW provides information necessary to decode the data messages that follow. For example, it is common for the CSDW to contain a value for the number of messages that follow and to have flags that indicate what kind of intra-packet headers are used between messages.

Reading and decoding a data packet is accomplished by first reading the CSDW. Then individual data messages that follow in the data packet are read, taking into account the appropriate intra-packet headers and data formats. Move on to the next header and data packet when there are no more data messages to read.

Intra-packet headers, when they are present, typically contain one, or sometimes more than one, time stamp as well as other information about the data message that follows. Commonly used structures for intra-packet time data are shown in the three figures below. These three time structures will be referenced in most of the data format descriptions that follow.

struct SuIntrPacketTime_RTC 
    { 
    uint8_t aubyRelTime[6]; // 48 bit RTC 
    uint16_t uUnused;       // 
    }; 

struct SuIntrPacketTime_Ch4Time 
    { 
    uint16_t uUnused;       // 
    uint16_t uHighBinTime;  // High order time 
    uint16_t uLowBinTime;   // Low order time 
    uint16_t uUSecs;        // Microsecond time 
    }; 

struct SuIntrPacketTime_1588Time 
    { 
    uint32_t uNanoSeconds;  // Nano-seconds 
    uint32_t uSeconds;      // Seconds 
    }; 

Computer Generated Data

PCM Data

Time Data

MIL-STD-1553 Data

Analog Data

Discrete Data

Message Data

ARINC 429 Data

Video Data

Image Data

UART Data

IEEE 1394 Data

Parallel Data

Ethernet Data

TSPI Data

CAN Bus Data

Chapter 10 defines a 48-bit Relative Time Counter (RTC) as the basis for all packet and message time stamps. The RTC clock is 10 MHz, resulting in a clock resolution of 100 nanoseconds. There is no constraint on the absolute value of the RTC; at recorder power on, it could be initialized to zero or some random number. Some recorder vendors will preset the RTC to a value based on absolute clock time, but for interoperability reasons it is unwise to assume the RTC value will be related to absolute clock time in any meaningful fashion. Absolute clock time comes into a recorder and is recorded much like any other data source. In fact, there may be multiple time sources recorded. Time Data, Format 1 data packets are used to record input time signals. Since Time Data, Format 1 packets contain both the absolute input time value and the RTC clock value at the instant the absolute time was valid, these packets can be used to relate RTC values to the input absolute time source. For example, if a time packet is recorded with a RTC value of 1,000,000 and an absolute time value of 100:12:30:25.000, then the clock time of a subsequent data packet with an RTC value of 1,150,000 could be deduced to be 100:12:30:25.015 (150,000 clock tics x 100 nsec per tic = 15 msec).

When multiple time channels are available, it is incumbent on the programmer or data analyst to determine and select the best source of time for a particular data set. For example, there may be separate time channels for time derived from IRIG B, Global Positioning System (GPS), and an internal battery backed up clock. In this scenario, all of these time sources are present in the data file as separate channels, each correlating the RTC to its own notion of clock time. The software application may allow the user to select which source of time to use for a given analysis. Alternatively, the software may decide the “best” source of time, depending on which time channels are providing valid time. In general, each time source will provide a slightly (or not so slightly) different clock time. It is usually most correct to select one time channel only and to use this channel exclusively to correlate RTC time to absolute clock time for all data packet types.

The stability of the RTC isn’t specified in Chapter 10 other than to require it to be at least as good as a common crystal oscillator. A good grade crystal oscillator can provide stability on the order of 10 ppm. Some vendors provide a RTC source considerably more stable than this. It is tempting for an application to find a single time packet early in a recording and to use those time values to subsequently derive clock time from relative time. It is better to use the clock and relative time values from a time packet that occurs near the current data packet as the data file is decoded since there is some drift in the RTC during a recording session. It also may be the case that there is a jump in input clock time during a recording, such as when GPS locks for the first time, or when an IRIG time source is reprogrammed.

Index and Event Records Often times it is useful to make an in-memory version of the data file index. This allows rapid access to recorded data packets based on time or the occurrence of events. A general algorithm for reading all root and node index packets is as follows:

1. If “R-x\IDX\E” is not equal to “T” then index does not exist.

2. Move read pointer to last packet of data file. Store file offset of this packet.

3. If last packet data type does not equal 0x03 (Computer Generated Data, Format 3) then index does not exist.

4. Get the index count from the CSDW.

5. For each root index contained in the packet,

• Read the Node Index offset value
• Move the read pointer to the Node Index offset value
• Read the Node Index packet
• Get the node index count from the CSDW
• For each node index contained in the packet read and store the time stamp, channel ID, data type, and data packet offset values.

6. Read last root node index. If offset value is equal to current root node packet offset (stored in Step 2) then done.

7. Else the move read pointer to the next Root Index packet offset value

8. Read the next Root Index packet.

9. Go to Step 4.

Chapter 10 recorders can stream their data over one of their download interface network ports using User Datagram Protocol (UDP)/IP and Chapter 10 UDP transfer headers. This is normally done over an Ethernet port, but any network connection that supports UDP/IP can use this method. The .PUBLISH command is used to control data streaming. Chapter 6 defines the use of .PUBLISH and has numerous examples of its use. Data can be streamed to one or more specific unicast and multicast IP addresses, or broadcast address. Different channels can be addressed to different addresses.

It is common to publish different groups of data to different multicast groups. According to RFC 3171, addresses 224.0.0.0 to 239.255.255.255 are designated as multicast addresses. Different multicast address regions are designated for different purposes. According to RFC 2365, Chapter 10 data streaming should be directed to multicast addresses in the Local Scope address range 239.255.0.0 to 239.255.255.255.

IP multicast packets are delivered by using the Ethernet MAC address range 01:00:5e:00:00:00 - 01:00:5e:7f:ff:ff. This is 23 bits of available address space. The lower 23 bits of the 28-bit multicast IP address are mapped into the 23 bits of available Ethernet address space. This means that there is ambiguity in delivering packets. If two hosts on the same subnet each subscribe to a different multicast group whose address differs only in the first 5 bits, Ethernet packets for both multicast groups will be delivered to both hosts, requiring the network software in the hosts to discard the packets which are not required. If multiple multicast addresses are used, be careful to choose multicast addresses that will result in different Ethernet multicast addresses.

Multicast data is filtered by the Ethernet controller hardware, only passing subscribed packets to the software driver for decoding. This improves performance under high network traffic loads. Ethernet controllers only have a limited number of multicast addresses they can filter. 16 multicast addresses is a common hardware limit. If a workstation needs to subscribe to more multicast addresses than the Ethernet hardware provides for, then all multicast traffic is passed to the software driver for filtering, negating the benefit of multicast filtering in hardware. The size of a UDP packet is represented by a 16 bit value in the IPv4 IP and UDP headers, but some software implementation treat this as a signed value with a maximum value of 2^15 or 32768. Because of this, the maximum size of a Chapter 10 streaming packet should be no more than 32724 bytes. Physical networks have a Maximum Transfer Unit (MTU), which is the largest data packet they can carry. If a UDP packet has a size larger than the network MTU, it will be fragmented into smaller packets by the IP software driver before sending them over the underlying physical network. The fragmented UDP packets are then reassembled into a larger packet by the IP software driver at the receiving end. There is a performance penalty for this fragmentation and reassembly. Better performance may be achieved by choosing a UDP packet small enough to avoid fragmentation and reassembly. Regular Ethernet supports a maximum size of 1500 bytes of data payload (IP header, UDP header, and UDP data) but some newer Ethernet technologies support larger jumbo frames.

Chapter 10 data packets are sent in a UDP/IP packet by prepending a UDP transfer header to the UDP data payload. Chapter 10 data packet(s) smaller than the 32k maximum size will prepend the non-segmented UDP transfer header shown below.

struct SuUdpTransferHeaderNonseg 
    { 
    uint32_t uVersion    :  4;  // Version 
    uint32_t uType       :  4;  // Type of message 
    uint32_t uUdpSeqNum  : 24;  // UDP sequence number 
    }; 

A Chapter 10 data packet larger than the 32k maximum size will need to be segmented before transmission, and will prepend the segmented UDP transfer header shown below. IPv6 supports large data packets, negating the need for segmented data packets.

struct SuUdpTransferHeaderSeg 
    { 
    uint32_t uVersion    :  4;  // Version 
    uint32_t uType       :  4;  // Type of message 
    uint32_t uUdpSeqNum  : 24;  // UDP sequence number 
    uint32_t uChanID     : 16;  // Channel ID 
    uint32_t uChanSeqNum :  8;  // Channel sequence number 
    uint32_t uReserved   :  8;  // 
    uint32_t uSegOffset;        // Segment offset 
    }; 

Computer Generated Data, Format 3 (Recording Index) packets are meaningless in a network data stream. It is necessary that they be transmitted so that Channel ID 0 data packets will have contiguous sequence numbers for error detection. They should be ignored, though, when received.