LiDAR: Parsing SICK Compact Format Binary Telegrams with Kaitai Struct

1 - What Is Kaitai Struct?

Whenever a sensor transmits data as a binary stream, someone has to write code that reads those bytes and turns them into meaningful values. The conventional approach is to write that code by hand, in whatever language the project uses, which leads to repetitive, platform-specific parsers that are hard to maintain and even harder to share across teams.

Kaitai Struct solves this by separating the description of a binary format from its implementation. You write a single .ksy file that declares what each field is, how large it is, and how to interpret it. From that one description the Kaitai Struct Compiler (ksc) generates a ready-to-use parser class in any supported language (C++, Python, C#, Java, JavaScript, Go, Rust, and more) without requiring you to write a single line of parsing code.

• Write once, use everywhere. One .ksy file produces parsers in a dozen languages.
• Self-documenting. The format definition is readable YAML; it also serves as the format specification.
• Tooling included. A browser-based IDE and a hex visualizer let you debug formats against real captured data without writing any code at all.
• Free & open-source. The compiler is GPLv3; all runtime libraries are MIT licensed.

2 - The SICK Compact Format

SICK LiDAR devices stream measurement data as UDP datagrams over the network. Each datagram carries exactly one compact telegram. Regardless of the payload type, every telegram is wrapped in the same four-field outer frame:

• Magic (field start_of_frame): four 0x02 bytes that mark the start of every compact telegram. A receiver uses this pattern to locate frames inside a raw byte stream.
• Telegram type (field telegram_type): a 32-bit integer that identifies the payload type. The value drives a switch in the Kaitai description so the correct parser is invoked automatically.
• Payload: type-specific data described by the individual .ksy files. The sections below cover the three active types.
• Checksum: CRC32 computed over start_of_frame + telegram_type + payload. Used by the receiver to detect corrupted datagrams.

The entry point for Kaitai is compact_frame.ksy. It imports the payload .ksy files and uses a switch-on: telegram_type expression to dispatch to the right sub-parser automatically:

3 - Scan Data (Primary Data)

The Scan Data telegram carries the raw distance and intensity measurements from one scan segment. A stream of these telegrams, one per sensor rotation segment, is assembled by the receiver into complete 360° scan frames. Each frame contains a point cloud that can be projected into 3D Cartesian coordinates using the azimuth and elevation angles embedded in the metadata.

The payload is structured as a header followed by one or more modules. Each module covers a contiguous angular range and carries its own metadata and the raw beam data for every measurement point inside that range.

Header fields

Module metadata fields (repeated per module)

Beam data fields (repeated per beam, per echo)

Kaitai description (complete)

4 - IMU Data

The IMU telegram delivers inertial measurements from the built-in Inertial Measurement Unit. It carries linear acceleration, angular velocity, and an orientation estimate as a quaternion, all time-stamped to the same clock as the scan data. This makes it possible to correct for sensor motion during data acquisition.

The IMU telegram is structurally simpler than the Scan Data telegram: it has no variable-length modules and no conditional fields. The payload is a flat sequence of fixed-size values.

Kaitai description (complete)

5 - Encoder Data

The Encoder telegram reports the current state of an external rotary encoder connected to the device. The encoder is typically attached to the axis of an external conveyor or rotating mount, so the sensor knows the precise angular position of the object being scanned at the time of each measurement. The telegram is emitted together with each Primary Data frame and carries a frame_number field that links it to the corresponding scan frame.

Kaitai description (complete)

The header fields (telegram_counter, transmit_timestamp, telegram_version, payload_length, sender_id) live in the shared header type below, imported from compact_header.ksy.

6 - Using Kaitai With This Format

The four steps below take you from zero to a working parser for any of the three telegram types described in this document. No programming language is assumed - Kaitai targets whichever language you already use.

ksc -t python formats/compact_frame.ksy
ksc -t csharp formats/compact_frame.ksy
ksc -t cpp_stl formats/compact_frame.ksy

// Open a captured binary file and wrap it in a Kaitai stream.
stream = open_binary("captured_frame.bin")
frame  = CompactFrame.parse(stream)

// Dispatch on telegram_type to access the typed payload.
if frame.telegram_type == 1       // Scan Data (Type 1)
    for each mod in frame.payload.module
        use mod.metadata.num_beams_per_scan
        use mod.metadata.frame_number
        for each beam in mod.beams
            for each line in beam.lines
                for each echo in line.echos
                    use echo.distance, echo.rssi

else if frame.telegram_type == 2  // IMU (Type 2)
    imu = frame.payload
    use imu.acceleration.x, imu.acceleration.y, imu.acceleration.z
    use imu.angular_velocity.x, imu.angular_velocity.y, imu.angular_velocity.z
    use imu.orientation.w, imu.orientation.x, imu.orientation.y, imu.orientation.z

else if frame.telegram_type == 4  // Encoder (Type 4)
    enc = frame.payload
    use enc.tick_count
    use enc.speed
    use enc.frame_number           // links to the matching Type 1 frame