User Tools

Site Tools


ch10_handbook:data_file_interpretation

This is an old revision of the document!


DATA FILE INTERPRETATION

Overall Data File Organization

Chapter 10 data files are organized as a sequential series of data packets. Each data packet can only contain one type of data (i.e. 1553, Pulse Code Modulation (PCM), etc.). Data packets frequently contain multiple individual data messages.

The Chapter 10 standard requires that the first data packet in a data file be an IRIG 106 Chapter 9 format, Telemetry Attributes Transfer Standard (TMATS) packet. The TMATS packet is used to configure the recorder and to describe the data being recorded. Also, TMATS is stored in a Computer Generated Data, Format 1 (Data Type = 0x01) data packet and is discussed in paragraph 6.5.2.

An important field in the packet header for the TMATS packet is the “IRIG 106 Chapter 10 Version” field. The value of this field determines the specific version of the Chapter 10 standard to use when interpreting the rest of the data file. This value should not be confused with the Data Type Version (but sometimes called Header Version) discussed in paragraph 6.3. The IRIG 106 Version field is only defined for the IRIG 106-07 and later versions of the IRIG 106 standard. Since unused reserved fields must be initialized to zero, this field will have a value of 0 in data files compliant with IRIG 106 prior to IRIG 106-07.

Starting with publication of IRIG 106-07, more than one TMATS packet is allowed at the beginning of the file if recorder configuration information exceeds the packet size limit.

It is required that a Time Data packet be the first “dynamic” data packet. Dynamic data packets are not defined in the Chapter 10 standard but it is generally understood that a dynamic data packet means any data packet other than Computer Generated Data Format 1 (setup record). The purpose of this requirement is to allow the association of clock time with the Relative Time Counter (RTC) before encountering the first data packet in the data file. Programmers are cautioned to verify a valid time packet has been received before attempting to interpret the RTC as described in paragraph 6.6.

A root index packet will be the last data packet in the data file if file indexing is used. The presence of data file indexing is indicated in the TMATS setup record.

The size of all data packets is an integer multiple of 4 bytes (32 bits) with the maximum size of 524,288 bytes. Computer Generated Data, Format 1 (TMATS) packets have a maximum packet size of 134,217,728 bytes. To provide this 4 byte alignment, padding bytes are added, if necessary, to the end of a data packet, just before the checksum. Regardless, when defining data structures representing Chapter 10 data packets and messages, these structures should be forced to be byte aligned by using the appropriate compiler directive.

Some data packet elements are two or more bytes in length. For example, the first data element of a data packet is a two byte sync pattern. Multiple byte data elements are stored in little endian format. That is, the least significant portion of the data is stored at the lowest byte offset.

Data packets are written to disk roughly in the time order in which they are received, but data packets and data messages can occur in the data file out of time order. This order can occur because data recorders receive data simultaneously on multiple channels, each channel buffering data for a period of time and then writing it to disk. Therefore, individual data messages will, in general, be somewhat out of time order because of grouping by channel. Consider the case of two 1553 channels recording the same bus at the same time in an identical fashion. Each channel receives, buffers, and writes data to disk. The first channel will write its buffered data to disk followed by the second channel. The data received from the second channel will be from the same time period as the data from the first channel and will have identical time stamps but will be recorded after the first channel in the data file.

Starting with the IRIG 106-05 standard, recorders are only allowed to buffer data for a maximum of 100 milliseconds and data packets must be written to disk within one second. This ensures that data packets can only be out of time order by a maximum of one second. Be warned, though, that the maximum amount of time data packets can be out of order for data files produced before IRIG 106-05 is unbounded and it is not unusual to encounter data files with data packets five or more seconds out of time order.

Example source code that demonstrates basic parsing of Chapter 10 data files can be found in Appendix A. An example program that demonstrates reading and interpreting a Chapter 10 file can be found in Appendix B.

Overall Data Packet Organization

Overall data packet organization is shown in Figure 6-1. Data packets contain a standard header, a data payload containing one or multiple data messages, and a standard trailer. The standard header is composed of a required header, optionally followed by a secondary header. The data payload generally consists of a Channel Specific Data Word (CSDW) record followed by one or more data messages.

PACKET SYNC PATTERN Packet Header
CHANNEL ID
PACKET LENGTH
DATA LENGTH
DATA VERSION
SEQUENCE NUMBER
PACKET FLAGS
DATA TYPE
RELATIVE TIME COUNTER
HEADER CHECKSUM
TIME Packet Secondary Header (Optional)
RESERVED
SECONDARY HEADER CHECKSUM
CHANNEL SPECIFIC DATA WORD Packet Body
INTRA-PACKET TIME STAMP 1
INTRA-PACKET DATA HEADER 1
DATA 1 =
:
INTRA-PACKET TIME STAMP n
INTRA-PACKET DATA HEADER n
DATA n
DATA CHECKSUM Packet Trailer

Data Packet Organization

Data packets must contain data. They are not allowed to only contain filler. Filler can be inserted into a data packet in the packet trailer before the checksum. This filler is used to ensure data packet alignment on a four byte boundary. Filler is also sometimes used to keep the same length of packets from a particular channel. The standard does not expressly prohibit filler after the packet trailer but before the next data packet header; however, inserting filler after the last trailer is considered bad practice. Still, when reading data packets, the programmer should set read buffer sizes based on the value of the overall packet length found in the header. Do not make assumptions about packet length based on the data length or from information in the data payload.

When reading linearly through a Chapter 10 data file, maintaining synchronization with data packet boundaries is accomplished by using the packet length field in the header to read the appropriate amount of data or to reposition the read pointer to the beginning of the next header. In this case, it is sufficient to check the value of the Sync field at the beginning of the header to ensure the read pointer was positioned to the beginning of a data packet.

If there is an error in the data file, or if the read pointer is repositioned to a position other than the beginning of a data packet (for example to jump to the middle of a recorded data file), then the beginning of a valid data packet must be found. Unfortunately the Chapter 10 standard does not provide a way to definitively determine the beginning of a data packet in these instances. Instead, some heuristics must be applied:

  1. Read the data file until the packet sync pattern (0xEB25) is found. Normally the first character of the packet sync pattern is found at a file offset which is an integer multiple of four. However, if the data file is corrupted then the sync pattern may not fall on the normal four byte boundary. Scan the file a byte at a time, ignoring the normal four byte alignment. When the Sync pattern is found then
  2. Calculate and test the header checksum.
  3. If a secondary header exists, calculate and test the secondary header checksum.
  4. Calculate and test the data checksum.

If the packet sync pattern is found and all available checksums have been verified, then there is a high probability that the beginning of the next valid data packet has been found.

Required Header

The packet header contains information about the data payload such as time, packet length, data type, data version, and other information. The layout of a Chapter 10 packet header is shown in Figure 6-2. FIXME

struct SuI106Ch10Header 
    { 
    uint16_t uSync;         // Packet Sync Pattern 
    uint16_t uChID;         // Channel ID 
    uint32_t ulPacketLen;   // Total packet length 
    uint32_t ulDataLen;     // Data length 
    uint8_t ubyDataVer;     // Data Version 
    uint8_t ubySeqNum;      // Sequence Number 
    uint8_t ubyPacketFlags; // Packet Flags 
    uint8_t ubyDataType;    // Data type 
    uint8_t aubyRelTime[6]; // Reference time 
    uint16_t uChecksum;     // Header Checksum 
    }; 

Packet header structure.

The Channel ID field uniquely identifies the source of the data. The value of the Channel ID field corresponds to the Track Number value of the TMATS “R” record, “R-m\TK1.” The Channel ID field is a 16-bit field. However, the Chapter 9 TMATS format restricts the value of Channel ID to a two digit number (i.e. from 0 to 99). It is anticipated that this TMATS restriction will be lifted in future IRIG 106 standard releases.

Typically, only one packet data type is associated with a particular Channel ID, but this is not a requirement of the Chapter 10 standard. An exception to this is Channel ID = 0, the Channel ID used for internal, computer generated format data packets. It is typical for Channel ID 0 to contain Computer Generated Data Format 1 Setup Records (0x01), Computer Generated Data Format 2 Recording Events Records (0x02), and Computer Generated Data Format 3 Recording Index Records (0x03).

The data payload format is interpreted based on the value of the Data Type field and the Data Version field in the packet header. This field is sometimes incorrectly called “Header Version.” Each packet data payload can only contain one type of data (e.g. 1553, PCM, etc.). A Chapter 10 standard release will only contain data format and layout information for the latest Data Version. The specific Data Version defined in a particular Chapter 10 release can be found in the “Data Type Names and Descriptions” table. Be warned that future Chapter 10 releases may update or change data format or layout, indicated by a different Data Version value in the header, but the Chapter 10 release will not have information about the previous Data Versions. That information can only be found in the previous Chapter 10 releases.

When processing a data file, it is common to only read the data packet header, determine if the data portion is to be read (based on packet type or other information gleaned from the header), and, if not to be read, skip ahead to the next header. Skipping the data portion and jumping ahead to the next header is accomplished by using the packet length in the packet header. Below is the algorithm for determining how many bytes to jump ahead in the file byte stream to reposition the read pointer to the beginning of the next header:

a. Read the current primary header

b. - Determine relative file offset to the next header

Offset = Packet Length - Primary Header Length (24) - Secondary Header Length (12) (if included)

c. Move read pointer

Optional Secondary Header

The optional secondary header is used to provide an absolute time (i.e. clock time) stamp for data packets. The secondary header time format can be interpreted several ways. The specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure in Figure 6-3 is used when secondary header time is to be interpreted as a Chapter 4 format value (Flag Bits 3-2 = 0). The structure in Figure 6-4 is used when secondary header time is to be interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1).

struct SuI106Ch10SecHeader_Ch4Time 
    { 
    uint16_t uUnused;       // 
    uint16_t uHighBinTime;  // High order time 
    uint16_t uLowBinTime;   // Low order time 
    uint16_t uUSecs;        // Microsecond time 
    uint16_t uReserved;     // 
    uint16_t uSecChecksum;  // Secondary Header Checksum 
    }; 

Optional secondary header structure with IRIG 106 Ch 4 time representation.

ch10_handbook/data_file_interpretation.1397408490.txt.gz · Last modified: 2014/04/13 12:01 by bob

Except where otherwise noted, content on this wiki is licensed under the following license: CC0 1.0 Universal
CC0 1.0 Universal Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki