Video annotation & interpolation¶

Modifier-key convention¶

The shortcut tables use meta for the platform-aware modifier classification-hotkeys.service.ts resolves at runtime:

• meta = Cmd on macOS.
• meta = Ctrl on Windows/Linux.

A binding written meta+i means Cmd+I on a Mac, Ctrl+I elsewhere. Where Shift/Alt are involved, the order is meta+shift+... / meta+alt+... — actual key-press order doesn't matter.

Two scopes¶

• VIDEO scope — only active when a video classification is open: frame transport, keyframe management, interpolation, occurrence navigation, merge.
• BASE scope — always active in both video and image classifications: bbox selection, classification/sequence navigation, save, in-app shortcut help.

Both scopes are active together during video annotation, so a video user has every binding below available.

What "object" means here¶

The frontend uses object for what the AI worker calls a detected object (a multi-frame track of one individual). An object has a sparse bboxes array indexed by frame number — most entries null (no observation on that frame), some filled with [x, y, w, h]. A contiguous run of non-null entries is an occurrence: the animal entered the frame, did something, and exited.

A single tracked individual can have multiple occurrences — e.g. an animal that walks behind a tree and re-emerges produces two non-null runs with a null gap between them.

How interpolation works¶

Interpolation is explicit, never automatic — the frontend only fills bboxes when you press one of the two interpolation hotkeys.

The math is linear independently on x, y, w, h (interpolateBbox in resource-annotator.component.ts:135-148). The fill algorithm (fillInterpolatedBboxes at resource-annotator.component.ts:1521-1576) is piecewise: it collects every existing non-null bbox in the active range as a knot, then interpolates only the null slots strictly between consecutive knot pairs. Existing bboxes are preserved — never overwritten — even if they sit between two knots that would otherwise imply a different value.

Concretely, if the selected object's bboxes array looks like:

frame:  0   1    2   3   4   5    6    7    8   9   10
bbox:  K0   .    .   .   .  Km    .    .    .   .   K1
                             ^                        ^
                             └─ existing intermediate knot
                                                      └─ keyframe
                                                         (or current
                                                          frame)

…and you trigger meta+i with the keyframe at frame 0 and the current frame at frame 10, the algorithm:

1. Collects the three knots K0 (frame 0), Km (frame 5), K1 (frame 10).
2. Linearly fills frames 1-4 between K0 and Km.
3. Linearly fills frames 6-9 between Km and K1.
4. Leaves Km itself untouched.

This is what the source comment calls "fill gaps between existing knots" (resource-annotator.component.ts:1534-1537). Adding a manually-edited bbox in the middle of a previously-interpolated run is therefore how you correct interpolation drift: the new bbox becomes a knot, and re-running interpolation re-shapes the adjacent segments around it. There's no separate "promote to keyframe" gesture — any non-null entry is a knot.

Validation:

• The frame range must be at least 2 frames wide, otherwise: "endFrame - startFrame must be ≥ 2" (resource-annotator.component.ts:1529-1532).
• At least two knots must exist in the range — typically one at the keyframe, one at the current frame (resource-annotator.component.ts:1546-1551). Error: "Need a bbox at both the keyframe and the current frame."

Two flavours of interpolation¶

The keyframe is a separate marker from the bboxes themselves, toggled by meta+k: pressing on an unmarked frame sets it; pressing on the already-marked keyframe clears it. meta+shift+k always clears, regardless of state.

Occurrence-aware navigation¶

Four navigation actions move the playhead relative to the selected object's runs of non-null bboxes (resource-annotator.component.ts:1815-1908). All four require exactly one object selected — zero or more than one is an error.

The "occurrence"/"run" model (resource-annotator.component.ts:1842-1851):

isRunStart(bboxes, f) = bboxes[f] != null && (f === 0 || bboxes[f - 1] == null)
isRunEnd(bboxes, f)   = bboxes[f] != null && (f === bboxes.length - 1 || bboxes[f + 1] == null)

Frame transport, save, selection¶

Track lifecycle: keyframes, deletion, merge¶

In-app help¶

meta+/ opens the Hotkeys dialog, auto-rendered from classification-hotkeys.service.ts for the current build.

What happens to your edits¶

When you press meta+s (or autosave triggers), the frontend:

1. Walks every object's full bboxes array — every non-null entry, whether placed manually, from the AI tracker, or filled by a prior interpolation run — and emits one row per non-null frame.
2. Encodes the rows into the AggregateFramesPayload msgpack format (see Frame timeseries), zstd-compresses, and uploads the blob with the rest of the classification payload (classification-view.component.ts:935-992).

Two consequences:

• Interpolation is materialised before save. The backend never re-interpolates — whatever shape your timeline has when you press save is the shape stored. Re-running interpolation later requires editing in the frontend again.
• No per-frame attribute overrides. Attribute values (species, sex, age, behaviour, …) are stored once per object, not per frame — there's no per-frame attribute editing surface. Per-frame AI metrics (confidence, distance) are read-only, sourced from the AI pipeline.

Troubleshooting¶

Next steps¶

• Frame timeseries — the storage architecture behind these edits
• Configure trackers