Building FireSight at LA Hacks: Bringing a Drone’s View to Smart Glasses

Description and use

FireSight is a prototype that links drone video, AI analysis, and smart glasses to deliver real‑time hazard alerts to a wearer on the ground.

Problem

Aerial video contains a lot of information, but someone still has to interpret it and communicate what matters to the person on the ground.

Outcome

FireSight demonstrates a two‑way system that delivers real‑time hazard alerts from a drone to smart glasses, with wearer location reporting and SOS functionality, all within a prototype architecture.

Workflow

• 1. Capture video and telemetry from a DJI drone via Beagle Eyes. 2. Push paired image‑telemetry bundles to a backend. 3. Backend ingests data, runs OpenCV and Gemini vision‑language‑model analysis. 4. Generate hazard annotations and send structured alerts to the glasses. 5. Wearer receives alerts on smart glasses, can acknowledge or request help. 6. Wearer’s GPS and telemetry are reported back to the backend and displayed on the command dashboard. 7. Dashboard shows live video, annotations, wearer status, SOS and man‑down alerts, and a mini‑map. 8. System handles stale frames, duplicate rejection, and bounded buffering to keep information fresh.

Evidence

We wanted to connect aerial footage, AI analysis, a command dashboard, and smart glasses so that useful information could reach someone without requiring them to stop and check another screen.

Source

The backend uses Python and FastAPI, with OpenCV, NumPy, and Google’s Gemini SDK for image processing and analysis.

Source

We added SOS and a man‑down check… Holding a temple sends an SOS, which appears on the dashboard with the wearer’s reported position.

Source

The glasses do not provide reliable head‑facing compass orientation, so we use geographic directions such as northeast rather than claiming a hazard is directly to the wearer’s left or right.

Source