Sensors and Measurements

Writing Sample

This page is an archive of technical writing I did for Project Aria. For the most up to date documentation go to Project Aria Docs.

Introduction

This page provides an overview of the sensors and measurements we use for Project Aria, covering:

• Project Aria device sensors
• Naming conventions for all the tools and IDs used for sensors
• Coordinate systems
• Time (inc time stamping and camera shutter types)
• GPS
• Units of measurement

Sensors

Project Aria sensor data is stored in VRS and can record with:

• 1 x 110 degree HFOV Rolling Shutter High Resolution RGB camera, up to 8MP
• 2 x 150 HFOV / 120 degree VFOV Global Shutter mono cameras for SLAM & hand tracking, 640 x 480 pix
• 2 x 80 degree DFOV Eye-tracking Global Shutter mono cameras with IR illumination, 320 x 240 pix
• 2 x IMU, Barometer and Magnetometer (one IMU is 1KHz and the other is 800Hz)
• 7 x 48 KHz spatial microphones
• GPS, Bluetooth and WiFi

Researchers can use different sensor profiles when collecting data. Sensor profiles allow them to choose which sensors record as well as what settings to use. Settings options include what camera resolution to use and whether the output is RAW (no encoding) or JPEG (compressed).

tip

Cameras on Project Aria devices are installed sideways. By default, images are reported and viewed as they were provided by cameras and will appear sideways.

Naming conventions for all tools

We use the sensor name set as the main query type for each sensor in all tools. The naming conventions for all sensors are:

• Cameras: camera-slam-left, camera-slam-right, camera-et-left, camera-et-right, camera-rgb
• IMUs: imu-left, imu-right
• Magnetometers: mag0
• Microphones: mic0, mic1, ..., mic6
• Barometer: baro0

Each sensor is associated with an instance-invariant name. For example, the left SLAM camera is named as "camera-slam-left". The name set of supported sensors can be fetched in Python3 with the deviceModel.getCameraLabels() command:

>>> deviceModel.getCameraLabels()
['camera-et-left', 'camera-et-right', 'camera-rgb', 'camera-slam-left', 'camera-slam-right']
>>> deviceModel.getImuLabels()
['imu-left', 'imu-right']`

note

Left/Right are relative to the left and right side of the glasses when the user is wearing the device. Another way to differentiate between the two sides is that the left SLAM camera is closer to the RGB camera.

Table 1: IDs Used for Sensors

StreamId	Stream	vrs::RecordableTypeId	VRS Instance ID	Calibration labels for coordinate transform	DataProvider API
211-1	Eye tracking camera	EyeCameraRecordableClass (211)	1	camera-et-left camera-et-right	getEyeCameraPlayer()
214-1	RGB camera	RgbCameraRecordableClass (214)	1	camera-rgb	getRgbCameraPlayer()
231-1	Microphones	StereoAudioRecordableClass (231)	1	mic0, mic1, mic2, ..., mic6	getAudioPlayer()
247-1	Barometer	BarometerRecordableClass (247)	1	baro0	getBarometerPlayer()
281-1	GPS	GpsRecordableClass (281)	1		getGpsPlayer()
283-1	Bluetooth beacon	BluetoothBeaconRecordableClass (283)	1		getBluetoothBeaconPlayer()
285-1	Time domain	TimeRecordableClass (285)	1		getTimeSyncPlayer()
1201-1	SLAM Camera Left	SlamCameraData (1201)	1	camera-slam-left	getSlamLeftCameraPlayer()
1201-2	SLAM Camera Right	SlamCameraData (1201)	2	camera-slam-right	getSlamRightCameraPlayer()
1202-1	IMU sensor 1 (1KHz)	SlamImuData (1202)	1	imu-right	getImuRightPlayer()
1202-2	IMU sensor 2 (800Hz)	SlamImuData (1202)	2	imu-left	getImuLeftPlayer()
1203-1	Magnetometer	SlamMagnetometerData (1203)	1	mag0	getMagnetometerPlayer()

RecordableTypeId

VRS files contain multiple streams, each associated with a device type, defined by a RecordableTypeId enum value. For Project Aria, each kind of sensor is defined as a device type. See Streams in VRS's Documentation for more information.

VRS instance ID

A unique ID number for each instance of a sensor type. The first instance of a sensor type has the number 1, the second sensor number 2, and so on.

StreamId

Unique identifier for a stream of data in VRS.

A StreamId identifies each instance of a sensor, device, algorithm, or "something" that produces a stream of records. A StreamId combines a RecordableTypeId that describes the type of sensor device, or other producer of records, and an instance id, to differentiate streams coming from different sensors of the same type.

Coordinate Systems

Applications like stereo vision and navigation usually handle 2D and 3D points in different spaces, and transformations need to be conducted between them. With Project Aria data, we attach a local R3 coordinate frame to each sensor.

Figure 1: Sensors and Sensor Directions on Project Aria Devices

Go to [Calibration Sensor Data] for more information and code snippets.

Time

Every signal (or Record in VRS terms) collected by sensors is stamped with a timestamp from a common clock. For Project Aria data, this is usually the local time clock. All records are sorted in monotonically increasing order in a VRS file. We use the following sensor-specific conventions on timestamps:

• For all cameras, the timestamp of a frame is the center of exposure time, a.k.a. the middle point of exposure interval.
• The RGB camera is a rolling shutter, with a readout time of 5ms (when recording at 1408x1408) or 15ms (when recording at 2880x2880) from first to last line. The recorded timestamp of an RGB frame is the center of exposure timestamp of the middle row of the image. SLAM and eye tracking cameras are global shutter and all of the pixels in the image are exposed simultaneously.
• IMUs, accelerometers and gyroscopes may have a time offset from the local time clock. This is due to internal signal processing in the IMU, which introduces a small time delay. These are estimated during calibration and stored in the JSON as TimeOffsetSec_Device_Gyro and TimeOffsetSec_Device_Accel respectively.

GPS

Horizontal and vertical accuracy are calculated using Android definitions.

Units of Measurement

The units for numerical values in the code and documentation are:

• Coordinates, location and distance in world space: meters (m)
• Coordinates in image space: pixels
• Timestamp and time intervals: seconds (s)
• Angles: radians (rad)
• Acceleration: m/s^2
• Angular velocity: rad/s
• Pressure: pascal (Pa)
• Temperature: celsius (°C)
• Magnetometer: Tesla (T)