Layer diagram

                     ┌──────────────────────────────┐
                     │  Consumers                   │
                     │  (DAW plugin, external apps) │
                     └──────────────────────────────┘
                                   ▲
                   ┌───────────────┴────────────────┐
                   │   Transports                   │
                   │   OSC │ MIDI/MPE │ shm │ FFI   │
                   └───────────────┬────────────────┘
                                   ▲
                   ┌───────────────┴────────────────┐
                   │   Trackers                     │
                   │   point │ bbox │ plane │ face  │
                   │                      │ roto    │
                   └───────────────┬────────────────┘
                                   ▲
                   ┌───────────────┴────────────────┐
                   │   Core                         │
                   │   Tracker trait, manifolds,    │
                   │   filters, pipeline            │
                   └───────────────┬────────────────┘
                                   ▲
                   ┌───────────────┴────────────────┐
                   │   Runtime                      │
                   │   camera │ inference │ ringbuf │
                   └────────────────────────────────┘

Dependency direction points up. Core has zero I/O dependencies and is safe to run anywhere (including a real-time audio thread). Runtime owns all the OS and GPU integrations. Trackers compose core primitives and use runtime services through traits.

No tracker ever imports another tracker. No transport ever imports a tracker directly — it consumes Sample<G, C> via a generic Sink<G, C> trait.

Crate layout

crates/
  uify-core/            # no I/O; pure math + traits
  uify-point/           # point trackers (ball, generic feature)
  uify-bbox/            # bounding boxes (AA + oriented)
  uify-plane/           # homography + PnP → SE(3)
  uify-face/            # landmarks + SE(3) + blendshapes
  uify-roto/            # contour / mask tracker
  uify-runtime/         # cameras, inference, ring buffer
  uify-transport-osc/   # OSC
  uify-transport-midi/  # MIDI / MPE
  uify-transport-shm/   # shared memory
  uify-clap-plugin/     # CLAP (AU/VST3 via clap-wrappers)
  uify-ffi/             # C ABI cdylib + generated header

Pipeline

See pipeline. Every tracker is:

Source → Detector → Associator → Filter → Smoother → Sink

Each stage is a trait with a single step() method. Stages compose; a detector can be swapped without touching the filter.

Thread model

See threading. Three threads, strict roles, one lock-free bridge.

• Vision thread: camera capture → GPU inference → manifold filter → ring buffer push. Can allocate. May block on GPU.
• Audio thread (host-owned): reads the newest sample from rtrb, interpolates in the tangent space to the block time, writes parameter updates. No allocations. No locks. No I/O.
• UI thread: viewport, configuration. Reads a snapshot of tracker state; never the hot path.

Latency

See latency-budget. Glass-to-parameter:

Accuracy

See manifolds. Non-negotiables:

• Camera intrinsics calibration for every 3D tracker.
• Sub-pixel refinement on every point-like tracker.
• Tangent-space EKF for every group-valued tracker.
• RTS smoother for offline passes.
• Synthetic Blender GT suite with property tests.

Extension strategy

See extension. TL;DR: add a tracker by adding a crate that implements Tracker; add a transport by adding a crate with a Sink<G, C> impl; add a camera backend by implementing the camera::Backend trait in uify-runtime.