One independently decodable frame of media.
Deliberately not a BB.Message. Frames never travel over BB.PubSub -
they go over the bounded interface BB.Camera.Hub serves, and nowhere else.
Putting megabytes of image data through a fan-out registry with no
backpressure is the specific failure mode this package exists to avoid.
Dimensions
width and height are the dimensions the frame is actually encoded at,
which is not necessarily what the camera was asked for. A driver that can
parse them out of the data - the SOF marker of a JPEG, say - should do so
rather than echoing back its own request.
Timestamps
monotonic_time and wall_time are both integer nanoseconds, matching
BB.Message's envelope. monotonic_time is the one to measure intervals
with and may be negative; wall_time is the one to display.
DateTime.from_unix!(frame.wall_time, :nanosecond) if you want a DateTime.
Stamp them with the moment the frame was captured when you know it - for a port-based driver, when the frame's last byte left the port - rather than leaving them to default to when the struct happened to be built.
Sequence
sequence identifies the frame within its capture session and must strictly
increase within that session. Consumers are promised a frame strictly newer
than the last one they received, and that promise rests entirely on this
number. A driver with its own counter should set it; one that leaves it nil
gets it stamped by the hub.
The hub forgets the last sequence it saw whenever capture is confirmed started, so a driver may restart its count at zero for each session or carry a single counter across restarts - whichever its capture backend makes natural. Neither is more correct, and the hub accepts both.
Frame rates
requested_frame_rate is what the camera was asked for, and
measured_frame_rate is what the frames before this one actually did, over
the most recent window the hub measured. Both are stamped by the hub, so a
driver sets neither; both can be nil, the first if the camera never said
what it was asking for and the second until a window has closed.
They are named fields rather than metadata keys because every frame has
them whatever the driver is. metadata is for what only some drivers know -
a JPEG's component count, say - and nothing should go in it that the
framework puts on every frame.
Where the camera was
A camera on a gimbal is somewhere different in every frame, and the pose it
had when a frame was captured cannot be recovered afterwards - by the time
anyone asks, the joints have moved. So the hub stamps
joint_configurations onto each frame as it passes through: every joint's
configuration, read from the robot's state as the frame arrived.
attached_to says where the camera hangs - {:link, name}, {:joint, name} or :robot - which is what turns those configurations into a pose:
BB.Robot.Kinematics.forward_kinematics(
robot,
frame.joint_configurations,
link_name
)The transform is deliberately not computed here. It is a chain of 4x4
Nx products, which on a small target costs far more per frame than reading
the configurations does, and most frames are never asked where they were
taken. Snapshotting the cheap, perishable part and leaving the expensive,
reproducible part to whoever wants it is the whole trick.
Every joint is captured rather than only the camera's own chain. It costs one more table read per joint and answers questions the camera's chain can't, such as where the gripper was in this picture.
This is the most recently known configuration, not the configuration at the shutter. Joint state is only as fresh as the last estimator update that wrote it. A joint reporting at 100Hz is tighter than a 30fps frame interval; a slow encoder is not, and nothing here can make it so.
Summary
Types
@type t() :: %BB.Camera.Frame{ attached_to: {:link, atom()} | {:joint, atom()} | :robot | nil, data: binary(), format: BB.Camera.Format.t(), height: pos_integer(), joint_configurations: %{required(atom()) => number()}, measured_frame_rate: float() | nil, metadata: map(), monotonic_time: integer(), requested_frame_rate: float() | nil, sequence: non_neg_integer() | nil, wall_time: integer(), width: pos_integer() }