Knowledge Article | Support Portal

Overview

1 - What Is Kaitai Struct?

Whenever a sensor transmits data as a binary stream, someone has to write code that reads those bytes and turns them into meaningful values. The conventional approach is to write that code by hand, in whatever language the project uses, which leads to repetitive, platform-specific parsers that are hard to maintain and even harder to share across teams.

Kaitai Struct solves this by separating the description of a binary format from its implementation. You write a single .ksy file that declares what each field is, how large it is, and how to interpret it. From that one description the Kaitai Struct Compiler (ksc) generates a ready-to-use parser class in any supported language (C++, Python, C#, Java, JavaScript, Go, Rust, and more) without requiring you to write a single line of parsing code.

Write once, use everywhere. One .ksy file produces parsers in a dozen languages.
Self-documenting. The format definition is readable YAML; it also serves as the format specification.
Tooling included. A browser-based IDE and a hex visualizer let you debug formats against real captured data without writing any code at all.
Free & open-source. The compiler is GPLv3; all runtime libraries are MIT licensed.

Try it without installing anything The Kaitai Web IDE runs entirely in your browser at ide.kaitai.io. You can paste a .ksy file, load a captured binary, and immediately inspect every parsed field in an interactive tree, no compiler, no runtime, no installation required.

Protocol

2 - The SICK Compact Format

SICK LiDAR devices stream measurement data as UDP datagrams over the network. Each datagram carries exactly one compact telegram. Regardless of the payload type, every telegram is wrapped in the same four-field outer frame:

Magic

4 bytes

0x02 0x02 0x02 0x02

Command ID

u32 le

1 / 2 / 4 / …

Payload

variable

type-specific data

Checksum

u32 le

CRC32

Magic: four 0x02 bytes that mark the start of every compact telegram. A receiver uses this pattern to locate frames inside a raw byte stream.
Command ID: a 32-bit integer that identifies the payload type. The value drives a switch in the Kaitai description so the correct parser is invoked automatically.
Payload: type-specific data described by the individual .ksy files. The sections below cover the three active types.
Checksum: CRC32 computed over magic + command ID + payload. Used by the receiver to detect corrupted datagrams.

The entry point for Kaitai is compact_frame.ksy. It imports the payload .ksy files and uses a switch-on: command_id expression to dispatch to the right sub-parser automatically:

compact_frame.ksy

meta:
  id: compact_frame
  endian: le
  imports:
    - type_1_primary_data
    - type_2_imu
    - type_4_encoder

seq:
  - id: magic
    contents: [0x02, 0x02, 0x02, 0x02]

  - id: command_id
    type: u4                     # 1 = Scan Data, 2 = IMU, 4 = Encoder

  - id: payload
    type:
      switch-on: command_id      # Kaitai picks the correct sub-parser here.
      cases:
        1: primary_data
        2: imu
        4: encoder

  - id: checksum
    type: u4                     # CRC32 over magic + command_id + payload.

Command ID	Payload type	.ksy file	Supported devices
`1`	Scan Data (Primary Data)	`type_1_primary_data.ksy`	picoScan120, picoScan150, multiScan100, LRS4000
`2`	IMU Data	`type_2_imu.ksy`	picoScan150, multiScan100
`4`	Encoder Data	`type_4_encoder.ksy`	picoScan150

Type 1

3 - Scan Data (Primary Data)

The Scan Data telegram carries the raw distance and intensity measurements from one scan segment. A stream of these telegrams, one per sensor rotation segment, is assembled by the receiver into complete 360° scan frames. Each frame contains a point cloud that can be projected into 3D Cartesian coordinates using the azimuth and elevation angles embedded in the metadata.

The payload is structured as a header followed by one or more modules. Each module covers a contiguous angular range and carries its own metadata and the raw beam data for every measurement point inside that range.

Header fields

Field	Type	Description
`telegram_counter`	u64	Counts every telegram sent since the device started. Starts at 1. Use to detect missing frames.
`timestamp_transmit`	u64 (µs UTC)	Device system time at transmission, in microseconds since 1970-01-01 00:00 UTC.
`telegram_version`	u32	Format revision. Version 3 is the base; version 4 adds a per-module `distance_scaling_factor`.
`next_module_size`	u32	Byte size of the first module. Used by the parser to advance through modules sequentially.

Module metadata fields (repeated per module)

Field	Type	Description
`segment_counter`	u64	Consecutive segment index since device start.
`frame_number`	u64	Full scan revolution counter. All modules with the same `frame_number` belong to one complete frame.
`sender_id`	u32	Device serial number. Useful when multiple sensors send to the same host.
`num_lines_in_module`	u32	Number of scan layers (rows) in this module.
`num_beams_per_scan`	u32	Number of beams per layer. All layers in a module have the same beam count.
`num_echos_per_beam`	u32	Number of echoes per beam (1 for single-echo, up to 3 for multi-echo devices).
`phi[]`	f32[] per layer	Elevation angle of each layer in radians. One value per `num_lines_in_module`.
`theta_start[] / theta_stop[]`	f32[] per layer	Start and end azimuth angles (radians) for the beams in each layer.
`distance_scaling_factor`	f32 (v4 only)	Multiply `distance_raw` by this value to get distance in mm. Enables sub-millimetre resolution or ranges above 65 535 mm.
`next_module_size`	u32	Size of the next module in bytes. Zero means this is the last module.
`data_content_echos`	u8 bitfield	Bit 0 = distance present, bit 1 = RSSI present. Guards conditional echo fields.
`data_content_beams`	u8 bitfield	Bit 0 = beam properties present, bit 1 = per-beam azimuth present.

Beam data fields (repeated per beam, per echo)

Field	Type	Description
`distance_raw`	u16	Raw distance. Multiply by `distance_scaling_factor` (v4) or treat as mm (v3). Present when `data_content_echos.distance` = 1.
`rssi`	u16	Received signal strength. Dimensionless, device-specific range. Present when `data_content_echos.rssi` = 1.
`theta`	u16	Per-beam azimuth, encoded as an integer. Convert: `angle_rad = (theta − 16384) / 5215`. Present when `data_content_beams.azimuth_angles` = 1.
`properties`	u8 bitfield	Bit 0 = reflector detected. Present when `data_content_beams.beam_properties` = 1.

Kaitai description (complete)

type_1_primary_data.ksy

meta:
  id: primary_data
  endian: le
  bit-endian: be

seq:
  - id: header
    type: header
  - id: module
    if: header.next_module_size > 0
    type: module
    repeat: until
    repeat-until: _.metadata.next_module_size == 0

types:
  header:
    seq:
      - id: telegram_counter
        type: u8
        doc: Counts all telegrams sent since the device was switched on.
      - id: timestamp_transmit
        type: u8
        doc: Sensor system time in µs since 1.1.1970 00:00 in UTC.
      - id: telegram_version
        type: u4
        valid:
          min: 3
          max: 4
      - id: next_module_size
        type: u4
        doc: Size of the first module to be read.

  module:
    seq:
      - id: metadata
        type: metadata
      - id: beams
        type: beam
        repeat: expr
        repeat-expr: metadata.num_beams_per_scan

  metadata:
    seq:
      - id: segment_counter
        type: u8
      - id: frame_number
        type: u8
        doc: Full revolution counter.
      - id: sender_id
        type: u4
        doc: Device serial number.
      - id: num_lines_in_module
        type: u4
        doc: Number of layers in this module.
      - id: num_beams_per_scan
        type: u4
        doc: Number of beams per scan per layer.
      - id: num_echos_per_beam
        type: u4
        doc: Number of echoes per beam.
      - id: timestamp_start
        type: u8
        repeat: expr
        repeat-expr: num_lines_in_module
        doc: Acquisition time of first beam per layer, in µs.
      - id: timestamp_stop
        type: u8
        repeat: expr
        repeat-expr: num_lines_in_module
        doc: Acquisition time of last beam per layer, in µs.
      - id: phi
        type: f4
        repeat: expr
        repeat-expr: num_lines_in_module
        doc: Elevation angles per layer (radians).
      - id: theta_start
        type: f4
        repeat: expr
        repeat-expr: num_lines_in_module
        doc: Start azimuth angles per layer (radians).
      - id: theta_stop
        type: f4
        repeat: expr
        repeat-expr: num_lines_in_module
        doc: Stop azimuth angles per layer (radians).
      - id: distance_scaling_factor
        if: _parent._parent.header.telegram_version >= 4
        type: f4
        doc: Scales raw distance values; enables sub-mm resolution above 65535 mm.
      - id: next_module_size
        type: u4
        doc: Size of next module; 0 = this is the last module.
      - id: reserved
        type: u1
      - id: data_content_echos
        type: data_content_echos
        doc: Flags indicating which echo fields are present.
      - id: data_content_beams
        type: data_content_beams
        doc: Flags indicating which beam fields are present.
      - id: reserved1
        type: u1
    instances:
      theta_start_deg:
        value: theta_start[0] * 180.0 / 3.141592653589
      theta_stop_deg:
        value: theta_stop[0] * 180.0 / 3.141592653589

  data_content_echos:
    seq:
      - id: reserved
        type: b6
      - id: rssi
        type: b1
      - id: distance
        type: b1

  data_content_beams:
    seq:
      - id: reserved
        type: b6
      - id: azimuth_angles
        type: b1
      - id: beam_properties
        type: b1

  beam:
    seq:
      - id: lines
        type: measurement_data
        repeat: expr
        repeat-expr: _parent.metadata.num_lines_in_module

  measurement_data:
    seq:
      - id: echos
        type: echo
        repeat: expr
        repeat-expr: _parent._parent.metadata.num_echos_per_beam
      - id: theta_3
        if: _parent._parent.metadata.data_content_beams.azimuth_angles and (_parent._parent._parent.header.telegram_version < 4)
        type: u2
        doc: Version 3 only — theta serialized before properties (see errata).
        doc-ref: https://supportportal.sick.com/trouble-shooting/multiscan-v111-twist-compact-data-format/
      - id: properties
        if: _parent._parent.metadata.data_content_beams.beam_properties
        type: beam_properties
      - id: theta_4
        if: _parent._parent.metadata.data_content_beams.azimuth_angles and (_parent._parent._parent.header.telegram_version >= 4)
        type: u2
        doc: Version 4 only — theta serialized after properties.
    instances:
      theta:
        value: '_parent._parent._parent.header.telegram_version >= 4 ? theta_4 : theta_3'
        doc: Azimuth angle raw uint16; convert with (theta - 16384) / 5215.
      theta_rad:
        value: (theta.as<f4> - 16384.0) / 5215.0
      theta_deg:
        value: theta_rad * 180.0 / 3.141592653589

  echo:
    seq:
      - id: distance_raw
        if: _parent._parent._parent.metadata.data_content_echos.distance
        type: u2
        doc: Raw distance, scaled by distance_scaling_factor.
      - id: rssi
        if: _parent._parent._parent.metadata.data_content_echos.rssi
        type: u2
        doc: Received signal strength (dimensionless, 0-65535).
    instances:
      distance:
        value: '_parent._parent._parent._parent.header.telegram_version >= 4 ? _parent._parent._parent.metadata.distance_scaling_factor * distance_raw : distance_raw'
        doc: Distance in mm.

  beam_properties:
    seq:
      - id: reserved
        type: b7
      - id: reflector
        type: b1

Use case: 3D point cloud reconstruction Given phi (elevation), theta (azimuth), and distance, a 3D Cartesian point is: x = d · cos(phi) · cos(theta), y = d · cos(phi) · sin(theta), z = d · sin(phi). The Kaitai-generated class makes all three values accessible as simple object attributes, no manual offset arithmetic required.

Type 2

4 - IMU Data

The IMU telegram delivers inertial measurements from the built-in Inertial Measurement Unit. It carries linear acceleration, angular velocity, and an orientation estimate as a quaternion, all time-stamped to the same clock as the scan data. This makes it possible to correct for sensor motion during data acquisition.

The IMU telegram is structurally simpler than the Scan Data telegram: it has no variable-length modules and no conditional fields. The payload is a flat sequence of fixed-size values.

Field	Type	Description
`telegram_version`	u32	Format revision. Currently always 1.
`acceleration.x / .y / .z`	3 × f32	Linear acceleration along the sensor X, Y, Z axes in m/s².
`angular_velocity.x / .y / .z`	3 × f32	Angular velocity around the sensor X, Y, Z axes in rad/s.
`orientation.w / .x / .y / .z`	4 × f32	Sensor orientation as a unit quaternion (w, x, y, z). Can be converted to roll/pitch/yaw using standard formulas.
`timestamp`	u64 (µs UTC)	Sensor time in microseconds since 1970-01-01 00:00 UTC. Shares the same time base as `timestamp_transmit` in Type 1.

Kaitai description (complete)

type_2_imu.ksy

meta:
  id: imu
  endian: le

seq:
  - id: telegram_version
    type: u4
    valid: 1

  - id: acceleration
    type: vector3f
    doc: Acceleration in m/s².

  - id: angular_velocity
    type: vector3f
    doc: Angular velocity in rad/s.

  - id: orientation
    type: quaternion
    doc: |
      Sensor orientation as (w, x, y, z). Convert to Euler angles:
        roll  = atan2(2(wx + yz), 1 − 2(x² + y²))
        pitch = asin(2(wy − zx))
        yaw   = atan2(2(wz + xy), 1 − 2(y² + z²))

  - id: timestamp
    type: u8
    doc: Sensor time in µs since 1970-01-01 UTC.

types:
  vector3f:
    seq:
      - id: x
        type: f4
      - id: y
        type: f4
      - id: z
        type: f4

  quaternion:
    seq:
      - id: w
        type: f4
      - id: x
        type: f4
      - id: y
        type: f4
      - id: z
        type: f4

Type 4

5 - Encoder Data

The Encoder telegram reports the current state of an external rotary encoder connected to the device. The encoder is typically attached to the axis of an external conveyor or rotating mount, so the sensor knows the precise angular position of the object being scanned at the time of each measurement. The telegram is emitted together with each Primary Data frame and carries a frame_number field that links it to the corresponding scan frame.

Field	Type	Description
`telegram_counter`	u64	Counts all telegrams since device start.
`timestamp_transmit`	u64 (µs UTC)	Sensor time at transmission in microseconds since 1970-01-01 00:00 UTC.
`telegram_version`	u32	Format revision. Currently always 1.
`payload_length`	u32	Payload size in bytes, not including the CRC32 trailer.
`sender_id`	u32	Device serial number.
`frame_number`	u64	Links this encoder snapshot to the Primary Data frame with the same `frame_number`.
`tick_counter_value`	u32	Current encoder tick count. Divide by the configured ticks-per-revolution to get the angular position.
`tick_counter_value_at_ref_1_signal`	u32	Tick count at the last rising edge of the reference signal 1 (zero-mark).
`tick_counter_value_at_ref_2_signal`	u32	Tick count at the last rising edge of the reference signal 2.
`speed_value`	f32	Encoder speed in the unit configured on the device (typically counts/second).
`tick_counter_value_timestamp`	u64 (µs UTC)	Time when the tick counter last changed.
`ref_1_timestamp`	u64 (µs UTC)	Time of the last reference signal 1 event.
`ref_2_timestamp`	u64 (µs UTC)	Time of the last reference signal 2 event.

Kaitai description (complete)

type_4_encoder.ksy

meta:
  id: encoder
  endian: le
  bit-endian: be

seq:
  - id: telegram_counter
    type: u8
  - id: timestamp_transmit
    type: u8
    doc: Sensor time in µs since 1970-01-01 UTC.
  - id: telegram_version
    type: u4
    valid: 1
  - id: payload_length
    type: u4
  - id: sender_id
    type: u4
    doc: Device serial number.
  - id: frame_number
    type: u8
    doc: Links to the corresponding Primary Data frame.
  - id: tick_counter_value
    type: u4
  - id: tick_counter_value_at_ref_1_signal
    type: u4
  - id: tick_counter_value_at_ref_2_signal
    type: u4
  - id: speed_value
    type: f4
  - id: tick_counter_value_timestamp
    type: u8
  - id: ref_1_timestamp
    type: u8
  - id: ref_2_timestamp
    type: u8

Getting Started

6 - Using Kaitai With This Format

The four steps below take you from zero to a working parser for any of the three telegram types described in this document. No programming language is assumed - Kaitai targets whichever language you already use.

1

Get the .ksy files

Obtain the .ksy format descriptions from your SICK contact or SDK distribution. You need compact_frame.ksy plus the payload files for the types you want to parse (type_1_primary_data.ksy, type_2_imu.ksy, type_4_encoder.ksy).

2

Try it in the browser first

Open ide.kaitai.io, drag compact_frame.ksy into the editor panel, then drag a captured .bin file into the hex view. Kaitai parses the binary immediately and shows every field in an interactive tree. This is the fastest way to verify that a captured frame is well-formed.

3

Generate a parser in your language

Download the Kaitai Struct Compiler (ksc) from kaitai.io/#download. Java 8 or later is required. Then run:

ksc -t python formats/compact_frame.ksy
ksc -t csharp formats/compact_frame.ksy
ksc -t cpp_stl formats/compact_frame.ksy

Replace python with any supported target: csharp, cpp_stl, java, javascript, go, rust, and more. The compiler reads all imported .ksy files automatically and writes one source file per type into the output directory.

4

Add the runtime and parse

Each target language has a small runtime library (install it from kaitai.io or the language's package manager). Add the generated source files and the runtime to your project, then parse:

// Open a captured binary file and wrap it in a Kaitai stream.
stream = open_binary("captured_frame.bin")
frame  = CompactFrame.parse(stream)

// Dispatch on command_id to access the typed payload.
if frame.command_id == 1          // Scan Data (Type 1)
    for each mod in frame.payload.module
        use mod.metadata.num_beams_per_scan
        use mod.metadata.frame_number
        for each beam in mod.beams
            for each line in beam.lines
                for each echo in line.echos
                    use echo.distance, echo.rssi

else if frame.command_id == 2     // IMU (Type 2)
    imu = frame.payload
    use imu.acceleration.x, imu.acceleration.y, imu.acceleration.z
    use imu.angular_velocity.x, imu.angular_velocity.y, imu.angular_velocity.z
    use imu.orientation.w, imu.orientation.x, imu.orientation.y, imu.orientation.z

else if frame.command_id == 4     // Encoder (Type 4)
    enc = frame.payload
    use enc.tick_counter_value
    use enc.speed_value
    use enc.frame_number          // links to the matching Type 1 frame

The same field names appear in every language, only the syntax differs. The Kaitai documentation at doc.kaitai.io shows the equivalent access pattern for each supported target.

LiDAR: Parsing SICK Compact Format Binary Telegrams with Kaitai Struct

Related Products

Table of Contents

1 - What Is Kaitai Struct?

2 - The SICK Compact Format

3 - Scan Data (Primary Data)

Header fields

Module metadata fields (repeated per module)

Beam data fields (repeated per beam, per echo)

Kaitai description (complete)

4 - IMU Data

Kaitai description (complete)

5 - Encoder Data

Kaitai description (complete)

6 - Using Kaitai With This Format

Was this article helpful?

LiDAR: Parsing SICK Compact Format Binary Telegrams with Kaitai Struct

Related Products

Table of Contents

1 - What Is Kaitai Struct?

2 - The SICK Compact Format

3 - Scan Data (Primary Data)

Header fields

Module metadata fields (repeated per module)

Beam data fields (repeated per beam, per echo)

Kaitai description (complete)

4 - IMU Data

Kaitai description (complete)

5 - Encoder Data

Kaitai description (complete)

6 - Using Kaitai With This Format

Thanks for your feedback!

Was this article helpful?