How it works
The model supplies language. The manuals supply facts.
Shuka is not a chatbot with a system prompt. It is a retrieval pipeline in which a small language model is only ever allowed to paraphrase passages it can cite — and is switched off entirely when there is nothing to cite.
The pipeline
Four stages, all on one laptop.
STAGE 01
Embed the question
A 25 MB sentence-embedding model (MiniLM, quantized ONNX) turns the farmer's question into a 384-dimension vector. CPU, ~100 ms.
STAGE 02
Search the manuals
Exact cosine search over 1,308 pre-embedded passages from 8 manuals. At this scale brute force is single-digit milliseconds — no vector database, no approximation, nothing to tune.
STAGE 03
Write from sources only
The top passages, with page provenance, are handed to Llama 3.2 1B (Q4_K_M via llama.cpp) under instructions to answer only from them and cite by number — and never to invent a dosage.
STAGE 04
The guardrail
If no passage clears the relevance floor, generation never starts. The refusal is a feature: it fires on poultry medicine, weather forecasts and loan questions in our evaluation.
Design decisions
Choices a reviewer would ask about.
Why retrieval, not fine-tuning
Citations and refusals can't be fine-tuned in.
A fine-tuned 1B model still can't tell you which page its dosage came from, and still answers when it shouldn't. Grounding gives page-level citations, a refusal mechanism, and a corpus you can audit — and updating knowledge means re-indexing documents, not retraining.
Why a 1B model
Small enough to leave room for knowledge.
On an 8 GB machine the model must share memory with the OS, the embedding model and the index. At 0.77 GB on disk and 1.39 GB peak RSS measured, the whole system uses a fifth of the contest's memory budget — headroom we'll spend on a 3B upgrade only if the evidence says it pays.
Why llama.cpp
The same runtime the judges run.
The app binds llama.cpp (via node-llama-cpp) and loads the identical GGUF file the contest's profiler measures — one artifact, no gap between what we demo and what gets scored.
Engineering findings
What the constraint taught us.
Hybrid laptop CPUs starve generic llama.cpp builds. On a 2 P-core + 8 E-core i5, CPU-only decode managed 3–6 tokens/s and a full RAG prompt took over 85 seconds to prefill — efficiency-core stragglers hold the whole tensor graph hostage. The fix wasn't a bigger chip: the same laptop's integrated GPU, driven over Vulkan, prefills 27× faster and decodes at 14–15 tokens/s. Shuka auto-selects the iGPU and falls back to CPU. "Integrated graphics" is in the contest's hardware definition for a reason — it's the most underused compute in the African laptop fleet.
RAG changes what you must benchmark. Grounded prompts are long by design — four retrieved passages ahead of every question. Benchmarks that only time short prompts flatter the wrong thing, so our published harness times prefill and decode separately, on both backends, and ships every raw record in the repository.
The stack, exactly
- inference llama.cpp · node-llama-cpp
- model Llama 3.2 1B Instruct · GGUF Q4_K_M
- embeddings MiniLM-L6-v2 · ONNX int8
- index exact cosine · 1,308 × 384-d
- app Node.js · zero-framework web UI
- network after setup none
What's next
The roadmap is more corpus, not more model.
Next
Nigeria-first sources
NAERLS extension bulletins — the national reference — are indexed the day their licensing is cleared. The pipeline is ready; the documents are staged.
Then
Hausa, Yoruba, Pidgin
The corpus stays English; the conversation shouldn't have to. Multilingual question understanding is the highest-value upgrade for the people at the front of the queue.
Measured, then shipped
A 3B model — if it earns it
Memory headroom allows it. It ships only if the published eval shows the accuracy gain is worth the throughput cost on the standard laptop.
Named, not hidden
What real scale takes
Companies serving millions of farmers run field agents, voice channels, financing and packaged installers a cooperative can use without a developer. A prototype is not that, and this page won't pretend otherwise — the distance is the roadmap: a one-file installer, trained desk operators through ADP offices and agro-dealer networks, and a first field pilot to replace our assumptions with observations.