[Hackathon 3/3] Cal Hacks 13.0 — Drigil, a Job Coach in Smart Glasses

Hackathon Series 3/3 — Cal Hacks 13.0

My third hackathon that October was Cal Hacks 13.0, held on October 23–25, 2026, at San Francisco’s Palace of Fine Arts. The official event site describes 2,000+ hackers building over 36 hours.

Our project, Drigil, is a smart-glasses job coach for workers with intellectual and developmental disabilities (IDD). I developed the hardware pipeline connecting live video streams to image analysis, along with the emergency call system.

Devpost project · GitHub repository · Project website · Cal Hacks

Project Demo

The Drigil demo submitted on Devpost.

Watch on YouTube

1. The Problem — Needing Someone Beside You

Learning a new job often takes more than hearing instructions once. A person may need the task broken into small steps, feedback on mistakes, and a way to ask for help. A job coach or supervisor cannot always be beside them.

Drigil aims to repeat a supervisor’s demonstration at the point of work, then reduce guidance as the worker gains skill. The goal is to support independent work by adjusting the amount of help.

Our demo uses a café task: putting sleeves on cups. It guides the worker through finding the cup, picking up the sleeve, fitting it, and checking the result.

2. The Service — Teach Once, Coach on the Job

A supervisor uploads a narrated task video. Claude organizes it into 4–8 steps, each with an action, a description of correct completion, likely mistakes, and a correction. With consent, Cartesia clones the supervisor’s voice so the worker hears familiar guidance.

During work, the system analyzes camera frames alongside the current task. It distinguishes completion, work in progress, and a possible mistake, then provides the next instruction or a short correction. The worker can ask questions aloud or request human help.

Repeated success reduces support through full instruction → short cue → alerts only → silence. Recent errors can increase support again. SpacetimeDB shares progress with the supervisor’s interface.

3. My Work — Turning Live Video into Input for AI

Before AI can interpret a task, the scene needs to reach it reliably. My hardware work focused on connecting the live camera stream to analyzable frames.

Camera and Video Transport

The prototype uses a Raspberry Pi 5 and a Logitech Brio 100 camera. The Pi serves the camera’s MJPEG stream, and the receiving side obtains frames for the application’s visual-analysis flow. USB-C connects the Pi and host.

Seeing an image once is only the starting point. The pipeline also needs consistent frame delivery, a way to detect interruptions, and a clear relationship between the image being analyzed and the worker’s current action.

The repository includes tools for measuring frame rate, frame intervals, and decoding time, as well as stream health checks, recording, and synchronization checks. These support validation of the connection between camera input and image analysis.

Separating Video Input from Model Inference

The hardware supplies the stream; the model interprets the task state. The browser blurs faces before sending frames, and the edge application supplies task context when requesting analysis from Claude.

Connecting a camera to a Pi does not mean all AI inference runs on that Pi. This implementation uses cloud models for live visual judgment. On-device inference remains a future improvement.

4. My Work — The Wristband Emergency Call System

Spoken guidance does not eliminate every situation where a worker gets stuck. When explaining the problem aloud or finding a screen control is difficult, a touch on the wristband provides a physical route to the supervisor.

The hardware uses an Arduino Uno Q, a touch sensor, and the board’s 13×8 LED matrix. Touch events travel over USB serial and connect to calls or notifications through the application’s /wrist/press API.

This is an in-app request for supervisor support, not a connection to external emergency services. The ordinary help button first contacts a coworker and later escalates to a supervisor; a long wristband touch goes directly to the supervisor.

The system limits repeated inputs of the same kind and associates requests with an active work session. Detection and delivery are separate states so that device feedback can acknowledge what actually happened.

Recovery and Physical Validation

A serial connection needs to handle cable removal, reconnection, and competing processes trying to open the port. The application discovers the device and reports its status. The repository records real-board checks for short touches, long holds, and automatic recovery after reconnection.

Those notes also record a sensor input that remained HIGH longer than expected and the need to recheck it when worn. Firmware input checks and end-to-end delivery to the supervisor are separate validation tasks.

5. The Team’s Implementation — Guidance, Memory, and Human Help

The hardware connects to the team’s coaching, speech, and shared-state systems.

Repeated difficulty can trigger an instruction revision, but the supervisor reviews and approves changes before they take effect. The coaching flow also withholds uncertain judgments and uses repeated observations and time intervals to reduce incorrect corrections.

6. The Main Constraint — Timeliness and Incorrect Corrections

A late response about an old frame can give the wrong instruction after a worker has moved on. Video timing and the task step associated with a model response therefore need to be managed together.

Devpost reports the team’s comparison on 30 labeled frames from the cup-sleeve task. That is a limited demo evaluation, not a general accuracy claim across workplaces and users. This article focuses on the input and call paths and the design choices intended to reduce errors rather than presenting those numbers as overall product performance.

Moving beyond the prototype requires user testing of wearability, changes in camera viewpoint, sensor reliability, network latency, and privacy handling.

7. Award — 2nd in Claude for Public Benefit

Drigil placed 2nd in Anthropic’s Claude for Public Benefit track at Cal Hacks 13.0.

With Sarath Sundar H and Shraddha Khanapur, I helped build a service that observes work and connects it to support. My hardware pipeline and call system became useful through their integration with the models, web application, and realtime backend.

8. Looking Back at Three Hackathons

At SF Hacks, we built an app for sharing campus mobility barriers. At the ExecuTorch Hackathon, we built a device that interprets context using a board’s NPU. At Cal Hacks, we built a smart-glasses job coach.

Across all three, I kept returning to how an AI judgment becomes an action a person can use. For Drigil, that meant both connecting the camera to analysis and bringing a human into the loop when AI guidance was insufficient.

As a hardware developer, this was an opportunity to build and check the path from physical input to user feedback, beyond the behavior of an individual sensor or board.

Hackathon Series

1. SF Hacks 2026 — GatorAccess Campus Accessibility Navigator
2. ExecuTorch Hackathon — IDK Button
3. Cal Hacks 13.0 — Drigil (this article)

Based on the Drigil Devpost and public implementation records. My contribution covered the live video/image-analysis hardware and emergency call system.