BB.Camera.Frame (bb_camera v0.1.1)

Copy Markdown View Source

One independently decodable frame of media.

Deliberately not a BB.Message. Frames never travel over BB.PubSub - they go over the bounded interface BB.Camera.Hub serves, and nowhere else. Putting megabytes of image data through a fan-out registry with no backpressure is the specific failure mode this package exists to avoid.

Dimensions

width and height are the dimensions the frame is actually encoded at, which is not necessarily what the camera was asked for. A driver that can parse them out of the data - the SOF marker of a JPEG, say - should do so rather than echoing back its own request.

Timestamps

monotonic_time and wall_time are both integer nanoseconds, matching BB.Message's envelope. monotonic_time is the one to measure intervals with and may be negative; wall_time is the one to display. DateTime.from_unix!(frame.wall_time, :nanosecond) if you want a DateTime.

Stamp them with the moment the frame was captured when you know it - for a port-based driver, when the frame's last byte left the port - rather than leaving them to default to when the struct happened to be built.

Sequence

sequence identifies the frame within its capture session and must strictly increase within that session. Consumers are promised a frame strictly newer than the last one they received, and that promise rests entirely on this number. A driver with its own counter should set it; one that leaves it nil gets it stamped by the hub.

The hub forgets the last sequence it saw whenever capture is confirmed started, so a driver may restart its count at zero for each session or carry a single counter across restarts - whichever its capture backend makes natural. Neither is more correct, and the hub accepts both.

Frame rates

requested_frame_rate is what the camera was asked for, and measured_frame_rate is what the frames before this one actually did, over the most recent window the hub measured. Both are stamped by the hub, so a driver sets neither; both can be nil, the first if the camera never said what it was asking for and the second until a window has closed.

They are named fields rather than metadata keys because every frame has them whatever the driver is. metadata is for what only some drivers know - a JPEG's component count, say - and nothing should go in it that the framework puts on every frame.

Where the camera was

A camera on a gimbal is somewhere different in every frame, and the pose it had when a frame was captured cannot be recovered afterwards - by the time anyone asks, the joints have moved. So the hub stamps joint_configurations onto each frame as it passes through: every joint's configuration, read from the robot's state as the frame arrived.

attached_to says where the camera hangs - {:link, name}, {:joint, name} or :robot - which is what turns those configurations into a pose:

BB.Robot.Kinematics.forward_kinematics(
  robot,
  frame.joint_configurations,
  link_name
)

The transform is deliberately not computed here. It is a chain of 4x4 Nx products, which on a small target costs far more per frame than reading the configurations does, and most frames are never asked where they were taken. Snapshotting the cheap, perishable part and leaving the expensive, reproducible part to whoever wants it is the whole trick.

Every joint is captured rather than only the camera's own chain. It costs one more table read per joint and answers questions the camera's chain can't, such as where the gripper was in this picture.

This is the most recently known configuration, not the configuration at the shutter. Joint state is only as fresh as the last estimator update that wrote it. A joint reporting at 100Hz is tighter than a 30fps frame interval; a slow encoder is not, and nothing here can make it so.

Summary

Functions

Build a frame, validating it and stamping any timestamps left out.

Like new/1, but raises.

Types

t()

@type t() :: %BB.Camera.Frame{
  attached_to: {:link, atom()} | {:joint, atom()} | :robot | nil,
  data: binary(),
  format: BB.Camera.Format.t(),
  height: pos_integer(),
  joint_configurations: %{required(atom()) => number()},
  measured_frame_rate: float() | nil,
  metadata: map(),
  monotonic_time: integer(),
  requested_frame_rate: float() | nil,
  sequence: non_neg_integer() | nil,
  wall_time: integer(),
  width: pos_integer()
}

Functions

new(opts)

@spec new(keyword()) :: {:ok, t()} | {:error, term()}

Build a frame, validating it and stamping any timestamps left out.

new!(opts)

@spec new!(keyword()) :: t()

Like new/1, but raises.