Differences

This shows you the differences between two versions of the page.

--- ch10_handbook:data_file_interpretation [2014/04/13 09:33] – created bob
+++ ch10_handbook:data_file_interpretation [2014/07/16 15:18] (current) – Added links to TSPI and CAN Bus pages bob
@@ Line 1: / Line 1: @@
+~~ODT~~
 ===== DATA FILE INTERPRETATION =====
@@ Line 71: / Line 72: @@
 ==== Overall Data Packet Organization ====
- Overall data packet organization is shown in Figure 6-1. Data packets contain a standard
+Overall data packet organization is shown below. Data packets contain a standard
 header, a data payload containing one or multiple data messages, and a standard trailer. The
 standard header is composed of a required header, optionally followed by a secondary header.
@@ Line 77: / Line 78: @@
 by one or more data messages.
+| PACKET SYNC PATTERN | **Packet Header** |
+| CHANNEL ID | ::: |
+| PACKET LENGTH | ::: |
+| DATA LENGTH | ::: |
+| DATA VERSION | ::: |
+| SEQUENCE NUMBER | ::: |
+| PACKET FLAGS | ::: |
+| DATA TYPE | ::: |
+| RELATIVE TIME COUNTER | ::: |
+| HEADER CHECKSUM  | ::: |
+| TIME | **Packet Secondary Header (Optional)** |
+| RESERVED | ::: |
+| SECONDARY HEADER CHECKSUM | ::: |
+| CHANNEL SPECIFIC DATA WORD | **Packet Body** |
+| INTRA-PACKET TIME STAMP 1 | ::: |
+| INTRA-PACKET DATA HEADER 1 | ::: |
+| DATA 1  =| ::: |
+| : | ::: |
+| INTRA-PACKET TIME STAMP n | ::: |
+| INTRA-PACKET DATA HEADER n | ::: |
+| DATA n | ::: |
+| DATA CHECKSUM | **Packet Trailer** |
+<html><center><b>Data Packet Organization</b></center></html>
+Data packets must contain data. They are not allowed to only contain filler. Filler can be
+inserted into a data packet in the packet trailer before the checksum. This filler is used to ensure
+data packet alignment on a four byte boundary. Filler is also sometimes used to keep the same
+length of packets from a particular channel. The standard does not expressly prohibit filler after
+the packet trailer but before the next data packet header; however, inserting filler after the last
+trailer is considered bad practice. Still, when reading data packets, the programmer should set
+read buffer sizes based on the value of the overall packet length found in the header. Do not
+make assumptions about packet length based on the data length or from information in the data
+payload.
+ When reading linearly through a Chapter 10 data file, maintaining synchronization with
+data packet boundaries is accomplished by using the packet length field in the header to read the
+appropriate amount of data or to reposition the read pointer to the beginning of the next header.
+In this case, it is sufficient to check the value of the Sync field at the beginning of the header to
+ensure the read pointer was positioned to the beginning of a data packet.
+ If there is an error in the data file, or if the read pointer is repositioned to a position other
+than the beginning of a data packet (for example to jump to the middle of a recorded data file),
+then the beginning of a valid data packet must be found. Unfortunately the Chapter 10 standard
+does not provide a way to definitively determine the beginning of a data packet in these
+instances. Instead, some heuristics must be applied:
+  - Read the data file until the packet sync pattern (0xEB25) is found. Normally the first character of the packet sync pattern is found at a file offset which is an integer multiple of four. However, if the data file is corrupted then the sync pattern may not fall on the normal four byte boundary. Scan the file a byte at a time, ignoring the normal four byte alignment. When the Sync pattern is found then
+  - Calculate and test the header checksum.
+  - If a secondary header exists, calculate and test the secondary header checksum.
+  - Calculate and test the data checksum.
+ If the packet sync pattern is found and all available checksums have been verified, then
+there is a high probability that the beginning of the next valid data packet has been found.
+==== Required Header ====
+ The packet header contains information about the data payload such as time, packet
+length, data type, data version, and other information. The layout of a Chapter 10 packet header
+is shown below.
+<code>
+struct SuI106Ch10Header
+    {
+    uint16_t    uSync;              // Packet Sync Pattern
+    uint16_t    uChID;              // Channel ID
+    uint32_t    ulPacketLen;        // Total packet length
+    uint32_t    ulDataLen;          // Data length
+    uint8_t     ubyDataVer;         // Data Version
+    uint8_t     ubySeqNum;          // Sequence Number
+    uint8_t     ubyPacketFlags;     // Packet Flags
+    uint8_t     ubyDataType;        // Data type
+    uint8_t     aubyRelTime[6];     // Reference time
+    uint16_t    uChecksum;          // Header Checksum
+    };
+</code>
+<html><center><b>Packet header structure</b></center></html>
+The Channel ID field uniquely identifies the source of the data. The value of the
+Channel ID field corresponds to the Track Number value of the TMATS "R" record,
+“R-m\TK1.” The Channel ID field is a 16-bit field. However, the Chapter 9 TMATS format
+restricts the value of Channel ID to a two digit number (i.e. from 0 to 99). It is anticipated that
+this TMATS restriction will be lifted in future IRIG 106 standard releases.
+ Typically, only one packet data type is associated with a particular Channel ID, but this is
+not a requirement of the Chapter 10 standard. An exception to this is Channel ID = 0, the
+Channel ID used for internal, computer generated format data packets. It is typical for Channel
+ID 0 to contain Computer Generated Data Format 1 Setup Records (0x01), Computer Generated
+Data Format 2 Recording Events Records (0x02), and Computer Generated Data Format 3
+Recording Index Records (0x03).
+ The data payload format is interpreted based on the value of the Data Type field and the
+Data Version field in the packet header. This field is sometimes incorrectly called “Header
+Version.” Each packet data payload can only contain one type of data (e.g. 1553, PCM, etc.). A
+Chapter 10 standard release will only contain data format and layout information for the latest
+Data Version. The specific Data Version defined in a particular Chapter 10 release can be found
+in the "Data Type Names and Descriptions" table. Be warned that future Chapter 10 releases
+may update or change data format or layout, indicated by a different Data Version value in the
+header, but the Chapter 10 release will not have information about the previous Data Versions.
+That information can only be found in the previous Chapter 10 releases.
+ When processing a data file, it is common to only read the data packet header, determine
+if the data portion is to be read (based on packet type or other information gleaned from the
+header), and, if not to be read, skip ahead to the next header. Skipping the data portion and
+jumping ahead to the next header is accomplished by using the packet length in the packet
+header. Below is the algorithm for determining how many bytes to jump ahead in the file byte
+stream to reposition the read pointer to the beginning of the next header:
+a. Read the current primary header
+b.  - Determine relative file offset to the next header
+ Offset = Packet Length  - Primary Header Length (24)  - Secondary Header Length (12) (if included)
+c. Move read pointer
+==== Optional Secondary Header ====
+ The optional secondary header is used to provide an absolute time (i.e. clock time) stamp
+for data packets. The secondary header time format can be interpreted several ways. The
+specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure below
+is used when secondary header time is to be interpreted as a Chapter 4 format value
+(Flag Bits 3-2 = 0). The following structure is used when secondary header time is to be
+interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1).
+<code>
+struct SuI106Ch10SecHeader_Ch4Time
+    {
+    uint16_t uUnused;       //
+    uint16_t uHighBinTime;  // High order time
+    uint16_t uLowBinTime;   // Low order time
+    uint16_t uUSecs;        // Microsecond time
+    uint16_t uReserved;     //
+    uint16_t uSecChecksum;  // Secondary Header Checksum
+    };
+</code>
+<html><center><b>Optional secondary header structure with IRIG 106 Ch 4 time representation</b></center></html>
+<code>
+struct SuI106Ch10SecHeader_1588Time
+    {
+    uint32_t uNanoSeconds;  // Nano-seconds
+    uint32_t uSeconds;      // Seconds
+    uint16_t uReserved;     //
+    uint16_t uSecChecksum;  // Secondary Header Checksum
+    };
+</code>
+<html><center><b>Optional secondary header structure with IEEE 1588 time representation</b></center></html>
+==== Data Payload ====
+After the standard header and optional secondary header, each data packet begins with
+CSDW(s). The length of the CSDW varies depending on the data type. For example, Analog
+Data Format 1 may have multiple CSDWs. The CSDW provides information necessary to
+decode the data messages that follow. For example, it is common for the CSDW to contain a
+value for the number of messages  that follow and to have flags that indicate what kind of
+intra-packet headers are used between messages.
+Reading and decoding a data packet is accomplished by first reading the CSDW. Then
+individual data messages that follow in the data packet are read, taking into account the
+appropriate intra-packet headers and data formats. Move on to the next header and data packet
+when there are no more data messages to read.
+Intra-packet headers, when they are present, typically contain one, or sometimes more
+than one, time stamp as well as other information about the data message that follows.
+Commonly used structures for intra-packet time data are shown in the three figures below.
+These three time structures will be referenced in most of the data format descriptions
+that follow.
+<code>
+struct SuIntrPacketTime_RTC
+    {
+    uint8_t aubyRelTime[6]; // 48 bit RTC
+    uint16_t uUnused;       //
+    };
+</code>
+<html><center><b>Intra-packet Time Stamp, 48-bit RTC</b></center></html>
+<code>
+struct SuIntrPacketTime_Ch4Time
+    {
+    uint16_t uUnused;       //
+    uint16_t uHighBinTime;  // High order time
+    uint16_t uLowBinTime;   // Low order time
+    uint16_t uUSecs;        // Microsecond time
+    };
+</code>
+<html><center><b>Intra-packet Time Stamp, IRIG 106 Ch 4 binary</b></center></html>
+<code>
+struct SuIntrPacketTime_1588Time
+    {
+    uint32_t uNanoSeconds;  // Nano-seconds
+    uint32_t uSeconds;      // Seconds
+    };
+</code>
+<html><center><b>Intra-packet Time Stamp, IEEE 1588</b></center></html>
+==== Data Formats ====
+[[Computer Generated Data]]
+[[PCM Data]]
+[[Time Data]]
+[[MIL-STD-1553 Data]]
+[[Analog Data]]
+[[Discrete Data]]
+[[Message Data]]
+[[ARINC 429 Data]]
+[[Video Data]]
+[[Image Data]]
+[[UART Data]]
+[[IEEE 1394 Data]]
+[[Parallel Data]]
+[[Ethernet Data]]
+[[TSPI Data]]
+[[CAN Bus Data]]
+==== Time Interpretation ====
+ Chapter 10 defines a 48-bit Relative Time Counter (RTC) as the basis for all packet and
+message time stamps. The RTC clock is 10 MHz, resulting in a clock resolution of 100
+nanoseconds. There is no constraint on the absolute value of the RTC; at recorder power on, it
+could be initialized to zero or some random number. Some recorder vendors will preset the RTC
+to a value based on absolute clock time, but for interoperability reasons it is unwise to assume
+the RTC value will be related to absolute clock time in any meaningful fashion.
+Absolute clock time comes into a recorder and is recorded much like any other data source. In
+fact, there may be multiple time sources recorded. Time Data, Format 1 data packets are used to
+record input time signals. Since Time Data, Format 1 packets contain both the absolute input
+time value and the RTC clock value at the instant the absolute time was valid, these packets can
+be used to relate RTC values to the input absolute time source. For example, if a time packet is
+recorded with a RTC value of 1,000,000 and an absolute time value of 100:12:30:25.000, then
+the clock time of a subsequent data packet with an RTC value of 1,150,000 could be deduced to
+be 100:12:30:25.015 (150,000 clock tics x 100 nsec per tic = 15 msec).
+ When multiple time channels are available, it is incumbent on the programmer or data
+analyst to determine and select the best source of time for a particular data set. For example,
+there may be separate time channels for time derived from IRIG B, Global Positioning System
+(GPS), and an internal battery backed up clock. In this scenario, all of these time sources are
+present in the data file as separate channels, each correlating the RTC to its own notion of clock
+time. The software application may allow the user to select which source of time to use for a
+given analysis. Alternatively, the software may decide the “best” source of time, depending on
+which time channels are providing valid time. In general, each time source will provide a
+slightly (or not so slightly) different clock time. It is usually most correct to select one time
+channel only and to use this channel exclusively to correlate RTC time to absolute clock time for
+all data packet types.
+ The stability of the RTC isn’t specified in Chapter 10 other than to require it to be at least
+as good as a common crystal oscillator. A good grade crystal oscillator can provide stability on
+the order of 10 ppm. Some vendors provide a RTC source considerably more stable than this. It
+is tempting for an application to find a single time packet early in a recording and to use those
+time values to subsequently derive clock time from relative time. It is better to use the clock and
+relative time values from a time packet that occurs near the current data packet as the data file is
+decoded since there is some drift in the RTC during a recording session. It also may be the case
+that there is a jump in input clock time during a recording, such as when GPS locks for the first
+time, or when an IRIG time source is reprogrammed.
+Index and Event Records
+ Often times it is useful to make an in-memory version of the data file index. This allows
+rapid access to recorded data packets based on time or the occurrence of events. A general
+algorithm for reading all root and node index packets is as follows:
+. If "R-x\IDX\E" is not equal to "T" then index does not exist.
+. Move read pointer to last packet of data file. Store file offset of this packet.
+. If last packet data type does not equal 0x03 (Computer Generated Data, Format 3)
+then index does not exist.
+. Get the index count from the CSDW.
+. For each root index contained in the packet,
+  * Read the Node Index offset value
+  * Move the read pointer to the Node Index offset value
+  * Read the Node Index packet
+  * Get the node index count from the CSDW
+  * For each node index contained in the packet read and store the time stamp, channel ID, data type, and data packet offset values.
+. Read last root node index. If offset value is equal to current root node packet offset
+(stored in Step 2) then done.
+. Else the move read pointer to the next Root Index packet offset value
+. Read the next Root Index packet.
+. Go to Step 4.
+==== Data Streaming ====
+ Chapter 10 recorders can stream their data over one of their download interface network
+ports using User Datagram Protocol (UDP)/IP and Chapter 10 UDP transfer headers. This is
+normally done over an Ethernet port, but any network connection that supports UDP/IP can use
+this method. The .PUBLISH command is used to control data streaming. Chapter 6 defines the
+use of .PUBLISH and has numerous examples of its use. Data can be streamed to one or more
+specific unicast and multicast IP addresses, or broadcast address. Different channels can be
+addressed to different addresses.
+ It is common to publish different groups of data to different multicast groups. According
+to RFC 3171, addresses 224.0.0.0 to 239.255.255.255 are designated as multicast addresses.
+Different multicast address regions are designated for different purposes. According to RFC
+, Chapter 10 data streaming should be directed to multicast addresses in the Local Scope
+address range 239.255.0.0 to 239.255.255.255.
+IP multicast packets are delivered by using the Ethernet MAC address range
+:00:5e:00:00:00 - 01:00:5e:7f:ff:ff. This is 23 bits of available address space. The lower
+bits of the 28-bit multicast IP address are mapped into the 23 bits of available Ethernet
+address space. This means that there is ambiguity in delivering packets. If two hosts on the
+same subnet each subscribe to a different multicast group whose address differs only in the first
+bits, Ethernet packets for both multicast groups will be delivered to both hosts, requiring the
+network software in the hosts to discard the packets which are not required. If multiple multicast
+addresses are used, be careful to choose multicast addresses that will result in different Ethernet
+multicast addresses.
+Multicast data is filtered by the Ethernet controller hardware, only passing subscribed
+packets to the software driver for decoding. This improves performance under high network
+traffic loads. Ethernet controllers only have a limited number of multicast addresses they can
+filter. 16 multicast addresses is a common hardware limit. If a workstation needs to subscribe to
+more multicast addresses than the Ethernet hardware provides for, then all multicast traffic is
+passed to the software driver for filtering, negating the benefit of multicast filtering in hardware.
+The size of a UDP packet is represented by a 16 bit value in the IPv4 IP and UDP headers, but
+some software implementation treat this as a signed value with a maximum value of 2^15 or
+. Because of this, the maximum size of a Chapter 10 streaming packet should be no more
+than 32724 bytes. Physical networks have a Maximum Transfer Unit (MTU), which is the
+largest data packet they can carry. If a UDP packet has a size larger than the network MTU, it
+will be fragmented into smaller packets by the IP software driver before sending them over the
+underlying physical network. The fragmented UDP packets are then reassembled into a larger
+packet by the IP software driver at the receiving end. There is a performance penalty for this
+fragmentation and reassembly. Better performance may be achieved by choosing a UDP packet
+small enough to avoid fragmentation and reassembly. Regular Ethernet supports a maximum
+size of 1500 bytes of data payload (IP header, UDP header, and UDP data) but some newer
+Ethernet technologies support larger jumbo frames.
+Chapter 10 data packets are sent in a UDP/IP packet by prepending a UDP transfer header to the UDP data payload. Chapter 10 data packet(s) smaller than the 32k maximum size will prepend the non-segmented UDP transfer header shown below.
+<code>
+struct SuUdpTransferHeaderNonseg
+    {
+    uint32_t uVersion    :  4;  // Version
+    uint32_t uType       :  4;  // Type of message
+    uint32_t uUdpSeqNum  : 24;  // UDP sequence number
+    };
+</code>
+<html><center><b>UDP Transfer Header, non-segmented data</b></center></html>
+A Chapter 10 data packet larger than the 32k maximum size will need to be segmented before transmission, and
+will prepend the segmented UDP transfer header shown below. IPv6 supports large data packets, negating the need for segmented data packets.
+<code>
+struct SuUdpTransferHeaderSeg
+    {
+    uint32_t uVersion    :  4;  // Version
+    uint32_t uType       :  4;  // Type of message
+    uint32_t uUdpSeqNum  : 24;  // UDP sequence number
+    uint32_t uChanID     : 16;  // Channel ID
+    uint32_t uChanSeqNum :  8;  // Channel sequence number
+    uint32_t uReserved   :  8;  //
+    uint32_t uSegOffset;        // Segment offset
+    };
+</code>
+<html><center><b>UDP Transfer Header, segmented data</b></center></html>
+Computer Generated Data, Format 3 (Recording Index) packets are meaningless in a
+network data stream. It is necessary that they be transmitted so that Channel ID 0 data packets
+will have contiguous sequence numbers for error detection. They should be ignored, though,
+when received.