Frame hypertable¶

See Frame timeseries for why the msgpack file, not the hypertable, is the durable source of truth — the hypertable is a derived, rebuildable index.

There's no on/off switch¶

ObjectFrameObservation is a TimescaleDB hypertable unconditionally — created by migrations 0095_object_frame_observation_model, 0096_migrate_bboxes_to_timeseries, and 0096b_enable_timescaledb_compression, none of which read any settings/env var to decide whether to run. There is no USE_FRAMES_HYPERTABLE flag or equivalent anywhere in the codebase (checked the full source tree and all three migrations directly) — TimescaleDB is a hard requirement of the database backend, not an opt-in feature.

What is on-demand, per project, is whether the hypertable's snapshot has been populated yet — that's what the rest of this page covers.

Populating a project's snapshot¶

Each ClassificationProject has its own ProjectHypertablePopulation row (status Pending / Populating / Ready / Failed), visible in the Expert admin under Media classification → Project hypertable populations. It's created lazily the first time you trigger a populate run, and readers consult its status before querying ObjectFrameObservation for that project.

# Default 1 fps, dispatched to Celery
python manage.py populate_project_hypertable --project 42

# Higher resolution, run inline (blocking) instead of via Celery
python manage.py populate_project_hypertable --project 42 --fps 5 --sync

# Every project, async
python manage.py populate_project_hypertable --all-projects --fps 1

This reads each classification's existing frames_msgpack file and writes (downsampled to the requested fps) rows into ObjectFrameObservation. Refuses to start if any targeted project is already Populating — wait for it to finish, or reset its status manually if it's stuck.

Tuning the hypertable¶

configure_hypertable inspects and tunes the shared TimescaleDB hypertable (chunk interval, compression). Safe to re-run — it only applies changes that differ from the current state.

# Read current config (status, dimensions, compression, chunk distribution)
python manage.py configure_hypertable --show

# Change the chunk interval for new chunks (existing chunks unchanged)
python manage.py configure_hypertable --chunk-interval "30 days"

# Replace the active compression policy
python manage.py configure_hypertable --compress-after "60 days"

# Re-tune segmentby for newly compressed chunks
python manage.py configure_hypertable --compress-segmentby "project_id,classification_id,dynamic_attrs_id"

# Combine + dry-run
python manage.py configure_hypertable \
    --chunk-interval "30 days" \
    --compress-after "14 days" \
    --dry-run

Rebuilding msgpack files from source data¶

If a classification is missing its frames_msgpack file, or it's gone stale relative to the underlying data, regenerate it with rebuild_classification_msgpacks. Images are rebuilt from the dyn_attrs JSON columns directly; videos need the project's hypertable already populated, and are skipped unless you pass --include-videos:

# Single project, only classifications missing a msgpack file (default)
python manage.py rebuild_classification_msgpacks --project 42

# Force-rebuild everything in a project, including videos
python manage.py rebuild_classification_msgpacks --project 42 --force --include-videos

# Single classification
python manage.py rebuild_classification_msgpacks --classification 12345

# Every project, dry-run (counts only, no file writes)
python manage.py rebuild_classification_msgpacks --all-projects --dry-run

--project / --classification / --all-projects are mutually exclusive and one is required.

Verify it worked¶

python manage.py configure_hypertable --show

Or check a project's status in Media classification → Project hypertable populations (should be Ready), and query ObjectFrameObservation directly to confirm row counts match expectations for your dataset.