AI pipeline architecture¶

The AI pipeline turns an uploaded resource into detections, species classifications, and (optionally) per-frame distance and pose estimates. It spans all three TRAPPER services — Expert backend, AI Manager, and AI Worker coordinator — plus the GBIF-keyed species table.

Services involved¶

End-to-end flow¶

sequenceDiagram
    participant U as User / Upload
    participant E as Expert backend
    participant M as AI Manager
    participant W as AI Worker

    U->>E: Upload resource (CS wizard or trapper-tools)
    E->>E: Create Resource, attach to Collection
    E->>M: POST /api/prediction_jobs/ (auto-submit or manual re-run)
    M->>M: Create PredictionJob + ResourceBatch rows, dispatch download_images task
    M->>W: send_task run_batch_prediction (runtime Celery queue)
    W->>M: GET resource files (origin_image URLs)
    W->>W: Run inference (detection, classification, pose/depth)
    W->>M: POST save_results (msgpack blob, zstd)
    M->>M: Store detected_objects FileField, refresh job status
    M->>E: Results available (Expert polls or webhook)
    E->>E: Unpack msgpack, resolve species, create Classification rows (type=AI)
    E->>E: Rebuild frames_msgpack blob (hypertable indexed later, on demand)
    E->>E: Privacy blurring (if blur_humans / blur_vehicles)

Stages¶

1. Resource ingestion¶

Resources arrive via the Citizen Science Upload wizard or trapper-tools. They are attached to a Collection; the Collection is linked to a ClassificationProject via a ClassificationProjectCollection row.

2. Job submission¶

When a new resource is ingested (or a pipeline re-run is triggered manually — see Run & re-run the AI pipeline), the Expert creates an AI Classification Job and submits it to the Manager via POST /api/prediction_jobs/. The submission carries:

• The job type (detection, classification, distance_calibration, distance_estimation)
• The prediction_model_id (the Manager's UUID for the selected AI provider)
• The merged job_config (from the ClassificationProject's prediction_config field, shaped by trapper-schemas — see Classification project: Prediction Job Configuration)
• Resource URLs and sample IDs

3. Batching and dispatch (Manager)¶

The Manager creates a PredictionJob row (tracking overall status: Initial → In Progress → Done / Failed / Canceled) and one or more ResourceBatch rows — each batch contains at most batch_size resources. A Celery download_images task is queued per batch.

Each batch progresses through stages:

CREATED → RESOURCE_DOWNLOADING → RESOURCE_DOWNLOADING_FINISHED → PREDICTION → FINISHED

and statuses: CREATED → IN_PROGRESS → SUCCEEDED / FAILED / CANCELED.

4. Inference (Worker)¶

The Worker pulls trapperai_runtime.run_batch_prediction from its runtime-specific Celery queue (queue name derived from AIRuntime.name). It:

1. Downloads resource files from the Manager (unless pass-through mode is active).
2. Runs the configured predictor (detection, classification, depth, or pose) against each resource.
3. Returns results as a msgpack blob (zstd-compressed, base64-encoded) per resource via POST /api/resource_batches/<id>/save_results/.

The msgpack format is documented in Msgpack detection-blob contract.

5. Result ingestion (Expert)¶

The Expert receives the results and, for each resource:

1. Unpacks the msgpack blob — extracts the objects array (detected bounding boxes with confidences, labels, frame indices).
2. Resolves species — maps the model's category labels to the project's Classificator species table using GBIF keys (gbifSpeciesKey). Labels with no mapping are dropped (if skip_missing_labels is set) or raise an error.
3. Creates Classification rows (classification_type=AI) — one per resource, linked to the resource's Collection and the ClassificationProject.
4. Creates ClassificationDynamicAttrs rows — one per detected object per frame, carrying observation_type, species, bounding box, confidence, and any extra per-frame metrics (distance, keypoints).
5. Rebuilds the msgpack blob — injects the newly-assigned DB IDs (ClassificationDynamicAttrs.id) into the msgpack objects so the frontend can reference specific detections by their database primary key.
6. Updates the frames hypertable — on the next populate_project_hypertable run, the per-frame detection data is indexed into the TimescaleDB hypertable for analytics queries (see Frame timeseries).

6. Privacy blurring¶

If the ClassificationProject has blur_humans or blur_vehicles enabled, a post-processing step rewrites the original media file to obscure the detected regions. blur_backup preserves an unblurred copy. blur_humans_and_vehicles_immediately applies blurring without waiting for human approval — irreversible unless backup is on.

7. Sequence building¶

A Celery task (celery_build_sequences) groups resources into sequences — ecological events — based on a configurable time window per collection (default 5 minutes). It is not a scheduled (beat) task: it fires when a collection is attached to a classification project, from the bulk Build sequences admin action, and after Citizen Science / collection uploads. Sequences must be built before annotators can step through bursts in the classify view. See Create a classification project, step 5.

Job types¶

Five capabilities flow through the same pipeline skeleton with different predictor logic. Four are submitted as distinct job types; pose estimation is bundled inside a detection job.

Detection¶

The detection job is the entry point for every pipeline except distance calibration, which runs independently against reference images (see Distance calibration). The Worker runs an object detector against each resource and a tracker to link detections across frames into continuous tracks (for video). The output is a set of DetectedObject tracks — one per individual animal, person, or vehicle — with bounding boxes indexed by frame.

Available detectors:

All detection models output three observation_type classes: Animal, Human, Vehicle.

Tracking:

For video resources, a tracker links frame-by-frame detections into multi-frame tracks. The tracker is configured per-job via prediction_config.detection.common.tracker. Available options include botsort / bytetrack (via Ultralytics) and boxmot:<name> (via the BoxMOT library). See Configure trackers for the full list and configuration knobs.

Classification¶

The classification job runs a species classifier on cropped detections from a prior detection run. It requires detection to have completed first — the classifier receives the bounding-box crops, not the full frame, so detection must have run and its results must be stored.

Available classifiers:

The table lists the classifiers bundled in the trapper-schemas model manifest — the set the AI Manager can register out of the box via sync_models_from_schemas. The Worker additionally ships a DeerAI (EfficientNet/ONNX, deer-specific) predictor class, but it has no manifest entry, so it can't be registered through the normal sync; treat it as experimental.

The classifier output is a species label plus a confidence score per detected object. The Expert resolves the label to a Species row via its gbifSpeciesKey, creating ClassificationDynamicAttrs rows with the species attached.

Pose estimation¶

Pose estimation runs inside the detection job — not as a separate submission. When a pose_ai_model is configured on the ClassificationProject, the Worker passes each tracked detection crop through the pose predictor immediately after the tracker, adding 39-keypoint skeletal estimates to the msgpack output.

Available pose models:

Keypoints are stored as per-frame data in the frames msgpack blob and surfaced in the classification frontend as an overlay on the bounding box.

Distance calibration¶

Before the pipeline can estimate real-world distances, it must be calibrated against a set of reference images — photographs taken at known distances. The calibration job:

1. Runs a monocular depth model against the reference images.
2. Fits an affine or piecewise linear function mapping predicted depth values to known metric distances.
3. Stores the resulting CalibrationModel JSON on the PredictionJob row.

The calibration is done once per deployment or camera position — it encodes the camera's specific geometry and lens characteristics. See Distance estimation for the full workflow.

Distance estimation¶

With a calibration in place, the distance estimation job runs the depth model on all resources in the collection and converts raw depth maps to metric distances for each detected animal. The pipeline:

1. Runs the depth model to produce a per-pixel depth map.
2. Uses FastSAM to segment the animal within its bounding box, deriving a representative depth value for the object.
3. Applies Kalman filter and Hampel filter smoothing to remove outliers across the track's frame series.
4. Writes the per-frame distance series into the frames msgpack blob.

Available depth models:

Runtime selection¶

The AI Worker coordinator manages multiple runtimes — isolated Python environments, each targeting a specific hardware backend and predictor library. The coordinator selects the appropriate runtime automatically based on detected hardware. Detection/classification models and depth models run in separate runtimes, so each hardware lane has a parallel depth variant:

flowchart LR
    HW{"Detected<br/>hardware"} --> N["NVIDIA GPU"]
    HW --> A["Apple Silicon"]
    HW --> CPU["CPU<br/>(incl. Ampere ARM64)"]
    HW --> H8["Hailo-8"]

    N --> NR["PyTorch CUDA / ONNX Runtime CUDA<br/><i>depth: PyTorch Depth CUDA</i>"]
    A --> AR["PyTorch MPS<br/><i>depth: PyTorch Depth MPS</i>"]
    CPU --> CR["PyTorch CPU / ONNX Runtime CPU<br/>(Ampere AIO variants are docker-only)<br/><i>depth: PyTorch Depth CPU</i>"]
    H8 --> HR["Hailo-8 Runtime (.hef)<br/><i>depth: Hailo-8 Depth Runtime</i>"]

Each runtime runs in an isolated process (or Docker container for docker_only runtimes) with its own virtual environment. Runtimes are defined in runtimes.yaml and the coordinator selects the highest-priority compatible runtime at job dispatch time.

Job chaining¶

Job types have dependencies that determine execution order:

detection ─────────────────────────────────────────────────► (results in Expert)
              └─► classification (needs detection bboxes)
              └─► pose estimation (runs inside detection job)

distance_calibration ──────────────────────────────────────► CalibrationModel stored
              └─► distance_estimation (needs calibration + detection bboxes)

The Expert enforces these constraints: a classification job will not be submitted if no detection results exist for the resource; distance_estimation requires both a completed distance_calibration and existing detections.

Configuration layers¶

The pipeline's behaviour is controlled at three levels, from broadest to narrowest:

See Classification project: advanced configuration for the full field reference.

Species resolution¶

The pipeline does not store raw model label strings. Instead, every classifier output is mapped to a GBIF taxon key — a stable, globally unique species identifier from the GBIF Backbone Taxonomy.

When the Expert ingests classification results:

1. It looks up each model label in the Species table by gbifSpeciesKey.
2. If a match is found, the species is attached to the ClassificationDynamicAttrs row.
3. If no match is found, behaviour depends on skip_missing_labels (see above).

This design means you can swap classifiers (e.g. upgrade from DeepFaune v1.2 to v1.4) without remapping species identifiers — as long as the new model's labels resolve to the same GBIF keys, the existing Classificator and any previously-approved classifications remain valid.

See Taxonomy & GBIF for how species are imported and managed.

Observability¶

See also¶

• Classification model — the data model behind AI Classification rows and ClassificationDynamicAttrs
• Msgpack detection-blob contract — wire format between Worker and Expert
• Frame timeseries — how per-frame detection data is indexed for analytics
• Taxonomy & GBIF — how species labels are imported and mapped to GBIF keys
• Distance estimation — the calibration + estimation workflow in detail
• Configure trackers — ByteTrack, BoT-SORT, BoxMOT options and configuration
• Register & sync AI providers — adding models to the Manager and syncing to the Expert
• Classification project: advanced configuration — all prediction_config fields