ChatJEVs assembles English one word at a time out of nothing but bounded decisions — multiple choice, ratings, and yes/no probabilities. No component of the system can generate text. We describe six architectures spanning 10 to 521 decision units, what each one adds, and where the whole approach stops working.
TypeSafe's Jev is a decision model. You give it state and a set of questions; it returns calibrated probabilities over the answers. It has exactly three question types — a Choice over at most 255 labelled options, a Score on a scale you define, and a Noul, one statement with one probability. It cannot emit a token.
We ask whether a sufficiently large network of such units develops useful language behaviour anyway. The answer word is sampled from a Choice over candidate words; everything else in the architecture exists to make that one Choice a good one. Nothing is trained — Jev is frozen. What varies between V0 and V5 is only the software around it.
Can a network of calibrated decision models produce language when every unit only makes a bounded decision?
A single word is produced by a short pipeline of batched requests. Only the last stage picks a word; everything before it decides what kind of word belongs there and what the reply so far means.
plan speech act and length once per reply
route which banks are worth running V3+
banks meaning · grammar · memory · concepts
memory slots updated in ordinary code, outside Jev
decode form → bucket → word + an <end> option
critics judge the top candidates V4+
sample a word falls out of the distributionEach is the same engine with more units, more state, and more second-guessing. Every version names the specific weakness the next one attacks.
| Version | Units | Per word | Vocab | What it adds | What it still cannot do |
|---|
The first four versions all improve the state the decoder sees. V1 adds working-memory slots and feeds the previous step's most confident answers back in, making it recurrent. V2 adds 46 concept units — small questions about animacy, size, motion, polarity — so meaning is represented as a pattern across units rather than in any one of them. V3 adds routing units that decide which banks run at all, which memory slot the decoder attends to, and how much context is relevant: attention, decided rather than computed.
All four share one flaw. Every word is a single 255-way guess, committed instantly, with no lookahead and no way back. A bad word at position two poisons everything after it, permanently.
The decoder stops deciding and starts proposing. Its six best candidates go to
a panel of eight critics, all forty-eight questions in one request: is this
grammatical here, would a fluent writer really write it, is it relevant, does it
advance the reply, does it assert anything false, is there still a way to finish
the sentence after it, does it repeat, does it match the tone. The panel's
verdict is blended with the decoder's own probability
(p0.4 × critics0.6) and the word is sampled
from the result.
The bet: asking "is cat right here?" is what a decision
model is built for. "Pick one of 255" is not.
Three drafts at rising temperature, each written quickly by the V1 architecture. A panel scores all three on eight whole-reply dimensions and votes head to head. The winner goes through a repair pass: every position is marked keep, replace or cut in a single batched request, and replaced positions are decoded again with the full critic panel. Cheap drafts, expensive judgment — and the only version that can fix its own second word.
The decoder gets 255 slots and fills them in priority order: words matching the
role and inflection the syntax units settled on; then words copied from the
conversation, which with a small vocabulary is most of what keeps a reply on
topic; then frequent words filling the remaining budget so a wrong guess
upstream is recoverable; then <end>, so the decoder can
decide the reply is finished rather than being told.
Each option is described by naming it — "The next English word is 'cat' (noun, singular)". An earlier build described options by part of speech instead, which made all 34 nouns in a candidate list read identically. The decoder was choosing blind, and output quality was correspondingly random.
Two instruments, because forced choice alone cannot see what V4 and V5 do.
Cloze is 90 minimal pairs across nine categories: agreement, attraction, binding, case, tense, quantifier, recall, selection and plausibility. Every distractor fills the same syntactic slot as the answer, so no item can be won by picking the only auxiliary or the only noun. In the recall items, every option appears somewhere in the passage, so copying a word from the prompt decides nothing. Option counts differ by category, so results are reported against each item's own chance level and with a 95% interval — at ten items per category, one flip moves a score ten points, and the interval says so.
Generation writes replies to eight probes and grades them in ordinary Python — no Jev in the scoring loop. Eight checks: did it produce anything, does it stutter, does it repeat, does the number on the verb match the noun, is the article right before a vowel, is there a verb at all, did it merely echo the prompt, did it stop by itself rather than hit the cap. This is the half that measures what the architecture is actually for.
It had 40 items, and 23 of them could be answered by picking the only
option of the right part of speech — including every single item in the
category it claimed was the hardest. It measured lazy distractors. The new
suite ships with a self-audit (chatjevs audit) that fails the
build if any item is solvable that way.
chatjevs.db. Nothing here is estimated; if no benchmark
has been run, the chart says so.None of the figures in this section have been run. They are reasoned estimates from the architecture's known constraints, and should be read as a prediction of what would happen, not a result. An earlier draft of this note put ARC and MMLU at 40–60%. That was wrong, for the reason in the next paragraph.
Jev is a System One model: it decides over state you supply, it does not store facts. That single property sets the ceiling. A benchmark whose answer is not present in the prompt gives it nothing to decide between, so the result is chance — not "somewhat below a real LLM", but chance. Assuming otherwise was the mistake in the earlier draft.
From there the picture is structural. Public LLM benchmarks split cleanly into two groups, and neither group measures ChatJEVs:
Expect every bar below to end on its own dashed chance tick. That gap is exactly why the internal benchmark exists: it is the only instrument that isolates what the architecture contributes.
| Benchmark | Format | Chance | Projected | Lift | Binding constraint |
|---|
A word here is not a token — it is a whole round of the network. Every word costs two to six HTTP requests and between ten and several hundred answered questions. This is the dominant practical constraint on the project and the reason the UI exposes a word cap.
python -m chatjevs serve # the app, at 127.0.0.1:8000
python -m chatjevs audit # check the benchmark itself
python -m chatjevs bench --arch v2 # fills the charts above
python -m chatjevs compare --archs v0,v1,v2,v3,v4,v5 --limit 2Benchmark results are written to chatjevs.db on this machine, and
Figures 2 and 3 read from it directly. Responses are cached by
(model, state, questions), so re-running a benchmark is free.