Skip to content

91. Denoiser

Customers often receive unstable data, which complicates the identification and detection of relevant scenarios for matching. Due to the varied structure and quality of the data, the process of converting it into a standardized object_list format is essential for ensuring consistency. The raw data is typically provided in CSV or Parquet format, and each customer requires a custom parser to convert it into the object_list format. This object_list is then serialized into a Google Protobuf file for further processing. The denoiser takes this parsed object_list as input and generates a stable, noise-free version of the data, ensuring better accuracy and reliability.

You can enable the denoiser to generate a user-readable report in HTML format that details the main actions performed on each object, and the reasons for these actions, such as identifying temporal gaps that required interpolation, splitting objects due to long temporal gaps, and filtering objects.

91.1 Examples of unstable data

Here are examples of unstable data, explained more clearly:

  • The vehicle's orientation is misaligned with the drive direction: The direction the vehicle is facing doesn't align with the direction it's moving in, resulting in inaccurate path tracking.
  • Erratic position changes due to unrealistic speed or acceleration: The vehicle’s position bounces or jumps inconsistently, as a result of non-physical changes in speed or acceleration, creating unrealistic movement.
  • Stationary vehicles change their yaw: When a vehicle is stationary, its yaw should remain constant. However, the yaw randomly changes, even though the vehicle is not moving.
  • Vehicles suddenly jump backwards: The vehicle’s position unexpectedly jumps to a point behind its previous location, as if it has moved backward, usually due to GPS or sensor errors.
  • Unrealistic object sizes: The size of detected objects, such as vehicles, becomes unrealistic, appearing much larger or smaller than their actual size due to errors in object detection.
  • Irregular timestamps between frames: The time intervals between consecutive data frames are inconsistent, with varying time differences instead of a fixed regular interval.

The denoiser improves the quality of the parsed data by filtering out instability and noise, resulting in a more reliable dataset for further analysis or processing.

91.2 Denoiser workflow

The following graphic illustrates the denoiser process workflow, showing how the object_list.pb file is processed into the sanitized denoise_object_list.pb file.

Figure 2: Denoiser workflow

At the end of the flow, the denoise_object_list.pb contains data that has been filtered, corrected, and refined to ensure it is accurate and ready for further analysis or processing.