← Build Log

$ cat minilingo voice assistant.md

MiniLingo Voice Assistant

MiniLingo is a self-contained, hybrid voice assistant built from scratch on an ESP32-S3 microcontroller. It pairs instant offline wake-word and command recognition with cloud AI pipelines (Whisper STT, Cerebras LLM, and local/cloud TTS) to deliver responsive conversational intelligence on resource-constrained embedded hardware.

Source Code↗

Spec sheet

type
iot / embedded
stack
ESP32, C/C++, ESP-IDF, FreeRTOS, Embedded Systems, Edge AI, IoT
status
● active · since 2026
repo
https://github.com/Chrimo02/SysProg/tree/main/MiniLingo
minilingo voice assistant — preview
MiniLingo Voice Assistant preview
fig. 01 · interface previewscale 1:1 · rev 2026

Overview

MiniLingo is a standalone desktop voice assistant engineered on a single ESP32-S3 microcontroller (16 MB Flash, 8 MB octal PSRAM) paired with an I2S microphone, audio amplifier, and an SPI touch display. Most consumer smart assistants rely entirely on cloud roundtrips—incurring high latency, continuous operational costs, and privacy tradeoffs. MiniLingo solves this through an architectural hybrid: common actions (timers, volume adjustments, hardware temperature readings) run completely offline on-chip with zero latency, while open-ended queries are passed through an optimized cloud AI pipeline.

Key Features

  • Hybrid Edge/Cloud Architecture: Instant on-device wake-word detection (supporting runtime switching between “Alexa”, “Mycroft”, and “Jarvis”) and offline speech command matching, escalating only conversational questions to cloud providers.
  • Interactive Touchscreen GUI: Built with LVGL v9, presenting a live chat stream with speech bubbles, responsive status bars, an animated listening/speaking microphone widget, and touch volume controls.
  • LLM Tool Calling on Embedded: Integrates Cerebras LLM function calling to convert arbitrary natural language requests (e.g., setting a 7-minute timer) into direct hardware actions.
  • Dual-Tier Speech Synthesis: Primary local Piper TTS server over LAN for sub-second spoken replies without quota limits, backed by Google Gemini TTS as an automated cloud fallback.
  • On-Device Provisioning: Built-in Wi-Fi captive setup portal allowing runtime configuration of credentials and API keys directly from a smartphone without re-flashing.

Architecture & Tech Stack

  • Hardware: ESP32-S3 (N16R8), INMP441 MEMS microphone (I2S RX), MAX98357A Class-D amplifier (I2S TX), ILI9341 2.8“ TFT display & XPT2046 touch controller (shared SPI2 bus), and DS18B20 1-Wire temperature sensor.
  • Firmware & RTOS: ESP-IDF v5.x with FreeRTOS pinned dual-core execution (Core 1 dedicated to the real-time audio front-end; Core 0 running LVGL rendering, network tasks, and atomic event synchronization).
  • On-Device Voice AI: Espressif esp-sr framework (WakeNet9 neural models & MultiNet7 speech recognition FSTs).
  • Cloud & Networking: Groq Whisper Large v3 Turbo (STT multipart uploads), Cerebras gpt-oss-120b(OpenAI-compatible chat completions), and mbedTLS for encrypted communications.

Highlights & Learnings

  • Internal SRAM Budgeting & TLS Starvation: Internal SRAM is severely constrained (~512 KB) and heavily contested by the esp-sr models, Wi-Fi drivers, and task stacks. Offloading mbedTLS buffer allocations directly to external PSRAM and enforcing strict sequential HTTP lifecycles proved critical to avoid starving the ESP32 hardware AES accelerator during cloud calls.
  • Acoustic Signal Processing & Silence Guards: Whisper tends to hallucinate plausible text when receiving ambient noise or wake-word reverb. Introducing a voice activity detection (VAD) silence guard (>1000 ms validated speech threshold) cleanly eliminated false cloud transcripts while keeping the acoustic pipeline robust.