9.1.3 Optimizing File Sizes

Integer Timestamps

Timestamps are formatted as float values by default. When floating point timestamps are logged during a longer running or high data rate capture, they take up a lot of disk space and are hard to compress. This results in large file sizes. Configuring the data logger to format timestamps as integers can result in smaller files.

To navigate to the timestamp configuration section, go to the log file configuration dialog:

The default configuration values for Integer change when timestamp format integer is first selected for a given data source or frame:
  • Encoding: Relative
  • Timestamps Resolution: 1
  • Resolution Unites: Microseconds
These are the two resulting modes and their functionality:
  • Relative: Each timestamp is converted to an integer and calculated as an offset to the first timestamp in the log file. Then, it is encoded according to the timestamp resolution and units values. For example, timestamp values of [33.000001, 33.000002, 33.000003] will be encoded as [0, 1, 2]

  • Delta: Each timestamp is converted to an integer and calculated as an offset to the previous timestamp in the log file. Then, it is encoded according to the timestamp resolution and units values. For example, timestamp values of [33.000001, 33.000002, 33.000003] will be encoded as [0, 1, 1]

Integer Timestamp Decoding Example

This code example shows how to decode integer timestamps as floating point values from a feather log:
import sys
import numpy as np
from pyarrow import feather as pf
import json

def read(filename): 
  table = pf.read_table(filename)
  df = table.to_pandas()
  print(df)

  timestamps = df['timestamp']
  meta = json.loads(table.schema.metadata[b'data-visualizer'])
  timestamp_meta = meta['timestamp']
  format=timestamp_meta['format']
  if format == 'integer':
    encoding = timestamp_meta['encoding']
    factor = float(timestamp_meta['resolution'])
    if encoding == 'delta':
      timestamps = np.cumsum(timestamps)*factor
    elif encoding == 'relative':
      timestamps = np.multiply(timestamps, factor)

    print('decoded timestamps\n{}'.format(timestamps))

if __name__ == "__main__":
  if (len(sys.argv) < 2):
    print("Usage: decode_timestamps.py <file_name>")
    sys.exit(1)

  filename = sys.argv[1]
  print("Reading " + filename + " as pyarrow table")
  read(filename)