Introduction
Helios is a live open-vocabulary vision demo — a natural-language chat beside a live camera feed that draws real-time labeled boxes around whatever you ask for.
Helios is a live vision demo: a natural-language chat beside a live camera feed. Tell it what to look for in plain language ("anything someone could trip over") and the feed draws real-time labeled bounding boxes around matching objects. It combines open-vocabulary detection (YOLOE, via Ultralytics) with natural-language understanding (Claude), running locally on a CUDA GPU.
Helios is a demo, not a production safety system
Helios is built to show open-vocabulary vision working live on one prepared machine. It is not a deployable monitoring, compliance, or life-safety system: it makes no reliability guarantee, it is not certified against any safety standard, and its PPE and danger-zone modes must never be relied on to protect a person. Treat every box it draws as a demonstration of a detector's output, not as an assurance about the scene.
The two panes
Helios is a self-contained app with two side-by-side panes:
- Chat pane — you type natural language describing what to detect.
- Live feed pane — the machine's camera with real-time bounding boxes and labels around every instance of the currently-active concepts.
Capture, inference, and annotation all run locally in one tight loop. Frames arrive at the browser pre-annotated as an MJPEG stream, so the page is a plain <img> with no per-frame JavaScript and no frame-versus-box sync bugs.
Why the pipeline works: reasoning and detection are decoupled
Claude does the reasoning — it turns a vague phrase into a concrete noun list. YOLOE only does visual pattern-matching on concrete nouns. Helios never asks the vision model to be smart.
That split is what makes vague prompts feel magical (Claude is the strong link) while keeping detection fast and local (YOLOE does what it is built for). The Claude expansion is a one-time step per prompt change — roughly 0.5–1.5 s — not per frame, so it never enters the frame hot path.
- Expected strengths: common concrete objects (people, bags, bottles, laptops, chairs, cables, cups) and vague or abstract prompts that Claude expands well.
- Expected strain: fine-grained or rare nouns, abstract concepts that survive into the detector, too many concepts at once, and confidence-threshold flicker.
Concepts are capped (default 8) at concrete, visually-groundable nouns, ordered most to least relevant.
No fallbacks, no degraded modes
Helios requires a CUDA-capable NVIDIA GPU and has no CPU path. If a CUDA device, camera, model, or API key is unavailable, the app hard-fails with a clear full-pane error card rather than silently dropping to a lesser path. There is also no literal-text shortcut for the chat: the chat is genuinely Claude-interpreted natural language, never a word-split of what you typed.
This is a demo principle, not a production-reliability principle — Helios optimizes for a great live experience on a known, prepared machine, and would rather show an honest error than a "works but worse" mode. See Reference for the three fatal error states and their exact text.
Scope
In scope
- Open-vocabulary real-time detection from text concepts (bounding boxes, labels, confidence).
- Genuine natural-language to concept-list mapping via Claude.
- Local camera capture and a smooth local inference loop, on a single-page UI.
- Three presenter-selected demo modes and object counting.
- Device portability: any one prepared CUDA-capable machine, tuned by config rather than code.
Out of scope
- Pixel-perfect segmentation masks.
- Cloud or hosted inference for the vision loop (a per-frame network round-trip in the live path).
- Recording, persistence, and mobile.
- Running on multiple machines at once, or distributed inference — portability means one selected machine per session.
- Multiple simultaneous cameras (one selected camera per session).
- Voice input; text chat only.
- Any fallback or degraded mode.