On-board language model inference, served over Wi-Fi.
Infron open-sources ESP32 LLM Service
Infron has open-sourced esp32-llm-service, a project that lets an ESP32-S3 serve a small language model over Wi-Fi while running inference on the board itself. The bundled TinyStories FP32 model has 260,032 parameters and continues short English stories. The project demonstrates the complete path from local model weights to a callable streaming service.
How it works
The model weights, tokenizer, inference runtime and KV cache all stay on the ESP32-S3. Computation runs in a native C Transformer; MicroPython handles tokenization, sampling and the network service. The Mac only builds and deploys the software over USB, then acts as a client. Once installed, the board can serve requests on an independent power supply.
Clients connect over Wi-Fi through a documented Chat Completions-compatible API subset. It supports JSON responses and Server-Sent Events, or SSE, which send generated text incrementally. The API also includes model listing, Bearer authentication, usage reporting and stop strings. A separate login-based MicroPython TCP shell provides file operations, system information and Python execution, including while inference is running.
Performance on the ESP32-S3
On the board, the repository’s October 21, 2026 hardware record shows about 30 tokens/s. A five-token prompt plus a 32-token output took 1.071 seconds, or 29.88 tokens/s, and that rate includes prompt processing and sampling. Seen from the Mac, the full HTTP request took about 1.70 seconds. A separate 128-token generation reached about 25.67 tokens/s. Test setup: ESP32-S3 at 240 MHz, 16 MB Flash, 8 MB Octal PSRAM, MicroPython 1.29.0.
Scope and limitations
This release focuses on short English story continuation rather than general chat or Chinese question-answering. It is designed as an experimental local service instead of a production endpoint for the public internet. API compatibility covers the documented subset: message text is concatenated rather than interpreted by chat role; tools and the Responses API remain outside the supported feature set. Total context is 512 tokens, output is capped at 256 tokens, and HTTP requests are handled one at a time.
Build it yourself
Want to reproduce it? Start with the hardware identification guide, then follow the pinned downloads, build, deployment and validation steps in the repository. Guides in English and Chinese cover memory variants and recovery procedures. Explore the code, star the repository if it is useful, and share implementation questions or reproduction findings on GitHub.
FAQ
Does the ESP32-S3 run the model entirely on-device?
Yes. The model weights, tokenizer, inference runtime and KV cache stay on the board. A Mac is needed for building and deployment, but it is not part of the inference path after installation.
What API does esp32-llm-service expose?
It provides a documented Chat Completions-compatible subset over Wi-Fi, with standard JSON responses or SSE streaming. It does not implement the full Chat Completions or Responses API feature set.
How fast is it?
In the repository’s October 21, 2026 hardware record, a five-token prompt and 32-token output took 1.071 seconds, or about 29.88 tokens/s, on an ESP32-S3 with 16 MB Flash and 8 MB Octal PSRAM.
Is this a general-purpose chatbot?
No. The bundled TinyStories model is intended for short English story continuation. The project is an experimental local service and is not designed as a production endpoint for the public internet.
Where can I find setup and recovery instructions?
Start with the repository README and the hardware guide. The repository includes English and Chinese setup guides, plus recovery procedures for supported memory variants.
