[Hackathon 2/3] ExecuTorch Hackathon — IDK Button, Context-Aware Interpretation on an NPU

Hackathon Series 2/3 — ExecuTorch Hackathon

This is the second of three hackathon stories. After campus accessibility at SF Hacks, we built a physical button for revisiting the context of something you just heard.

I joined the IoT track of the ExecuTorch Hackathon, starting on October 17, 2026, at Founders, Inc. in San Francisco. We built IDK Button — The button that knows what is being said using Arduino VENTUNO Q, Qualcomm Hexagon NPU, and the Qualcomm AI platform. I was project lead and handled the NPU pipeline, model conversion, quantization, inference, and validation.

The submission record gives the final submission time as October 18 at 1:00:02 PM PDT. This article is dated by the event’s start date, 2026-10-17.

GitHub repository · Submitted demo

1. The Problem — Hearing Words Without Understanding the Context

Recognizing every word in a conversation does not guarantee understanding. Words such as ground, port, and bank have several meanings; expressions such as make it depend on the surrounding conversation.

IDK Button uses recent conversation to help explain the current sentence. It transcribes speech, interprets it with context, asks for clarification when the meaning is ambiguous, and lets users save translations and useful vocabulary.

A physical button and rotary knob provide an alternative to retyping a sentence on a phone. The user listens, requests an interpretation, makes any necessary choice, and reviews the result.

2. Team and My Contribution

I prepared and validated the paths for running models on the board. Speech recognition and language inference use different runtimes, so model preparation, inputs, outputs, and application calls had to be connected separately.

Satyam handled the backend, hosting, and SQLite storage. Albert built the physical device, enclosure, and firmware. Together these formed board inference → application processing on the Mac → physical-device and dashboard feedback.

3. Architecture — The Board and the Mac

ASR and LLM inference run on the board’s Qualcomm NPU; the dashboard, backend, and dictionary lookup run on the Mac. Model inference does not require internet access, but the complete application uses both devices.

Recognized text goes to FastAPI on the Mac, which requests context analysis and translation from Qwen on the board. USB and ADB port forwarding connect these services. A 127.0.0.1 address alone does not identify the physical execution location; the forwarding path matters.

4. My Work — NPU Inference, Conversion, and Quantization

Whisper Through ExecuTorch

Speech recognition uses the precompiled whisper_qnn_fp16.pte artifact. The runtime loads it with ExecuTorch’s _load_for_executorch(), loads libqnn_executorch_backend.so, and checks that QnnBackend is registered.

Library paths, including ADSP_LIBRARY_PATH, point to the Qualcomm Hexagon runtime libraries. In the submitted architecture, Whisper encoder and decoder inference execute on Hexagon through the QNN/QAIRT HTP path. Audio preprocessing and token selection remain separate orchestration work.

My task extended beyond loading an artifact: checking backend availability, calling the model methods, and passing usable results to the next stage.

Qwen Through a Separate Runtime

Context analysis and translation use unsloth/Qwen3.5-2B-GGUF with Q4_0 quantization. Unlike Whisper’s .pte path, Qwen is served through GenieX’s llama.cpp-based engine and Qualcomm NPU runtime path, with --compute npu selected.

The two preparation and execution paths are distinct:

• Whisper: ExecuTorch AOT artifact, QNN backend, FP16 execution.
• Qwen: GGUF Q4_0 model, served through GenieX for context and translation.

Whisper’s FP16 execution and Qwen’s 4-bit quantization should not be conflated. Their file formats, runtimes, and calling conventions differ.

Token Decoding and Alternative Transcriptions

The ASR code calls encoder and decoder methods in a token-level loop. Alongside greedy decoding, it can branch at an uncertain position using a second-choice token to produce another transcription candidate.

No-speech probability and candidate score margins help flag uncertain recognition. Users can compare candidates before translating. An alternative is not guaranteed to be correct, but it exposes a possible recognition error instead of hiding it.

5. Freezing Context and Resolving Meaning

Context at the Moment of the Button Press

New speech must not silently change a request that is already being processed. The app therefore separates the live transcript from a frozen snapshot captured when the button is pressed.

The code maintains a bounded 30-second audio buffer and sentence history. Recent sentences supply context while the target sentence is handled separately. Listening can continue after the request, while the frozen context remains unchanged.

Three Dictionary Stages

Known expressions use reviewed rules first. The model handles cases that need additional context rather than replacing every dictionary lookup.

Vote Share Is Not Translation Accuracy

A small model’s self-reported confidence is not a reliable accuracy score. The repository instead samples choices among candidate meanings repeatedly and reverses their order to examine consistency and reduce ordering bias.

The resulting number is the model’s vote share, not the probability that a meaning is correct. A model can consistently choose the wrong answer. Clarification remains part of the flow, and uncalibrated translation_confidence is left empty.

6. Connecting the Physical Device and Dashboard

Albert’s hardware uses a button, rotary knob, and LEDs to communicate state. The board’s 8×13 LED matrix distinguishes recording, ASR, LLM processing, and result selection. Knob input is reflected in the dashboard.

Results provide three perspectives: natural translation, literal translation, and a short contextual explanation. These are different views of the same sentence, not confidence tiers. Saving and reviewing sentences and words connects to Satyam’s backend and SQLite layer.

7. What Validation Needed to Establish

Artifact loading, backend execution, response format, and semantic correctness need separate checks. Successful HTTP requests and valid JSON do not prove that a translation conveys the intended meaning.

The AI handoff records actual board outputs and their limitations. For example, translating You're grounded. as punishment can still miss the specific restriction on going out implied by the context.

The repository includes mocked model-response tests for APIs, context selection, dictionaries, storage, and hardware/transcript state handling. These complement actual board checks. Model weights, the Qualcomm SDK, and the complete runtime are not included, so cloning the repository alone does not reproduce the NPU environment.

8. What I Took Away

Running models on a device was a substantial part of building the product. Conversion, quantization, runtime configuration, and the board-to-Mac request path all had to work before a button press could produce an interpretation.

As project lead, my NPU work met Satyam’s application and storage layer and Albert’s hardware and firmware. The goal was to make inference available in a form users could review and act on.

Previous: Hackathon 1/3 — SF Hacks 2026, GatorAccess

Next: Hackathon 3/3 — Cal Hacks 13.0, Drigil