This page is the single entry point for building with UIFY. It is self-contained for the common path — an agent or human can complete ~90% of typical tasks without leaving this document. Links go out only for deep material (EKF-on-manifold derivation, CLAP internals, per-platform camera backend quirks).

If something in this document contradicts a reference doc, the code wins; file an issue or fix the doc.

What UIFY is

A Rust engine that turns camera input into a time-indexed stream of geometric primitives, with optional real-time transport to other programs. Five built-in trackers, all implementing one trait:

Every tracker produces Sample<G, C> values — value, tangent-space covariance, confidence, host-monotonic timestamp.

60-second quick start

# Development shell (Nix). Installs Rust toolchain, pre-commit hooks.
nix develop

# Build the workspace.
make

# Run the full test suite (unit + property + synthetic GT).
make test

# Lint (clippy warnings = errors).
make lint

# Build the CLAP plugin bundle.
make clap

# Format everything.
treefmt

The one abstraction

Every tracker is this:

pub trait Tracker {
    type Geometry;
    type Covariance;
    type Measurement<'a>;

    fn step(&mut self, measurement: Self::Measurement<'_>)
        -> Result<Option<Sample<Self::Geometry, Self::Covariance>>, TrackerError>;

    fn reset(&mut self);
}

Different trackers differ only in Geometry and Covariance. The pipeline around them — detector, associator, filter, smoother, sink — is the same shape.

Minimal pipeline (point tracker)

use nalgebra::{Matrix2, Matrix4, Vector2, Vector4};
use uify_core::{Sink, Tracker};
use uify_point::{PointMeasurement, PointTracker2D};

let mut tracker = PointTracker2D::new(
    Vector4::zeros(),               // initial state (px, py, vx, vy)
    Matrix4::identity(),            // initial covariance
    Matrix4::identity() * 1e-3,     // process noise (per second)
    Matrix2::identity() * 1e-3,     // measurement noise
);

loop {
    let frame = camera.next_frame()?;                  // runtime::camera::Camera
    let detection: Vector2<f64> = inference.infer(&frame)?;
    let measurement = PointMeasurement { t: frame.t, position: detection };
    if let Some(sample) = tracker.step(measurement)? { // Sample<Vec2, Mat2>
        sink.write(&sample)?;                          // any Sink<Vec2, Mat2>
    }
}

End-to-end example

examples/pipeline/ ties it all together: synthetic camera → constant inference → PointTracker2D → RingbufSink → RingbufSource → OscPoint2DSink → UDP. Run it with:

cargo run -p uify-pipeline-example

A DAW-targeted gesture pipeline (HandLandmarker → GestureFSM → ParameterMap → CLAP plugin params) is a future addition; the gesture FSM will live as a consumer of tracker samples, not part of any tracker, so the same hand tracker can drive non-DAW consumers too.

Latency + accuracy rules of thumb

• Audio thread never allocates, never locks, never blocks on I/O. It reads rtrb and interpolates in the tangent space to the block time.
• Every sample has a host-monotonic timestamp. Latency is measured, not guessed.
• Every filter lives on the correct manifold. Do not average quaternions componentwise. See architecture-manifolds.
• Every sample carries covariance. Consumers should gate on uncertainty.

Glass-to-parameter target: ~15–25 ms baseline, ~8–12 ms with distilled models + zero-copy capture. Below ~8 ms requires a 240 Hz camera.

Gotchas

• CoreML / DirectML / NNAPI have different f16 / f32 tolerances. The synthetic GT suite runs per-backend; do not compare numeric outputs across backends with ==.
• CLAP hosts reload plugins during scan. Global state in static mut is a bug waiting to happen — keep all mutable state in the Plugin instance.
• The GHC RTS is not involved. If you find yourself writing FFI glue for a garbage-collected runtime on the audio thread, stop.

Extending

• Add a tracker: new crate under crates/uify-<name>/, implement Tracker, register in uify-clap-plugin if it should drive parameters.
• Add a transport: new crate under crates/uify-transport-<name>/. Consume Sample<G, C> via a Sink impl.
• Add a platform: implement the camera::Backend trait and feature-gate in uify-runtime.

See architecture-extension for the full guide.