imajevTyped decisions from photos and app state
imajev is a family of open models (2B, 4B and 9B parameters) that answer typed questions about photos and application records. You send up to two photos, a record, and questions with the options you allow; each answer is a probability over those options plus an explicit unknown. There is no text generation: one forward pass per question, on your own machine. The request shape is the one TypeSafe's Jev uses, so an existing Jev client works by changing its base URL.
| Model | ImajevBench | JevBench hard | Hard ECE, calibrated | Latency p50 | Base download |
|---|---|---|---|---|---|
| imajev-2b | 71.7% | 60.4% | 0.123 | 238 ms (83 raw) | 4.6 GB |
| imajev-4b recommended | 82.4% | 70.3% | 0.116 | 350 ms (96 raw) | 9.3 GB |
| imajev-9b | 82.1% | 69.4% | 0.092 | 316 ms (96 raw) | 19.3 GB |
ImajevBench v2.0-lite test split, 279 items. JevBench public hard split, 111 text-only items, our same-protocol run, as served (4 option rotations, calibration file). Latency: one JevBench hard item as served, raw single pass in brackets, on one H100 under load, serial. Base download: the pinned Qwen3.5 weights.
1What it does
Application code often needs a small, typed judgement about a photo: does it match the record, which field does it contradict, is the part damaged, did anything change between two photos. A general chat model can answer, but its answer is free text that has to be parsed, and it rarely says when it cannot tell.
imajev answers only in the options you supply, returns a probability for each, and has a trained unknown for missing or contradictory evidence. That lets your code act on a threshold, abstain, or send the case to a person. Three question types cover most uses: noul (yes or no), choice (one of 2 to 254 options) and score (2 to 10 ordered levels).
2Examples
Six requests from the demo playground, answered by imajev-4b. The action line is ordinary application code applied to the probabilities; the model never writes text. The playground's own checks pass on this model.
3Results
Both benchmarks were run with the same protocol for every system. JevBench is text-only; ImajevBench adds photos, written records, and items whose correct answer is unknown.
predict() API and shown apart because its interface differs.Table
Table
Not #1 on JevBench. JevK5 and Eikos-4B (73.9%) lead every imajev size on the hard split, by 3 to 4 items of 111; the 4B (70.3%) is above Hopper (67.6%). On the previous adapter the gap was mostly in items that judge the quality of an answer.
Jev-Omni on ImajevBench. 78.5% through its own API. imajev-4b and 9B lead by about 4 points, which is not significant (p ≈ 0.25 on the previous adapters; not re-run). The difference comes from the 21 items whose answer is unknown, which Jev-Omni cannot abstain on; on answerable items it is slightly ahead and better calibrated.
Reasoning is a different interface. A frozen Qwen3.6-35B-A3B with its reasoning turned on scores 97.3% on JevBench hard, at seconds and thousands of tokens per decision.
4Calibration
Each model ships a calibration.json with one temperature, fitted by NLL on 150 template-generated JevBench-style items (none from JevBench); the explicit unknown is kept, with no offset. Temperature scaling never changes an answer, only how confident the model says it is.
| ECE, raw → calibrated | 2B | 4B | 9B |
|---|---|---|---|
| JevBench hard, served (4 rotations) | .176 → .123 | .164 → .116 | .187 → .092 |
| MMLU, text only | .141 → .038 | .150 → .035 | .123 → .031 |
| Typed decisions, 2,000 | .128 → .039 | .149 → .047 | .159 → .026 |
| SST-5, 2,210 | .221 → .105 | .220 → .020 | .211 → .029 |
| Photo-only checks, 823 | .023 → .120 | .038 → .062 | .027 → .053 |
The four lower rows were measured on the previous adapters and temperatures and are not re-run for the shipped versions. The last row is the exception. On photo-only verification the raw probabilities are already well calibrated and the temperature softens them too much. If most of your traffic is photo-against-record checks, serve without --calibration or fit a temperature on your own sample (section 5.5).
5For developers
How to run it, call it, act on its answers, pick a size, check it on your own data, and reproduce every number on this page.
5.1 Run a server
One command serves one model on POST /v1/systemone. MLX on Apple silicon, PyTorch elsewhere; no network calls at inference.
5.2 Request and response
Questions are keyed by your own names. A choice lists its options as keys of criteria, each with an optional description; a score lists its levels in order. Images go as multipart files, or as data URLs in an images array when you send JSON.
- probabilities
- For
choiceandscore: a distribution over your options that sums to 1, given that the model answers. - noul
- P(yes) with half of the unknown mass added, as in Jev. For the plain yes-probability subtract
unknown_probability / 2. - unknown_probability
- The mass on imajev's trained unknown option: missing, contradictory or out-of-scope evidence.
- abstained
truewhen unknown was the single most likely outcome.- confidence
- How concentrated the option distribution is, scaled down by the unknown mass.
- request limits
- Up to 2 images, a state string or JSON object up to 32 KB, 1 to 8 questions, 2 to 254 options, 2 to 10 score levels, English.
5.3 Act, abstain or escalate
The probabilities are the product. A common pattern: act automatically above a threshold, send the case to a person when the model says unknown or is unsure, and log everything. Choose the threshold from your own error costs.
5.4 Choosing a size
| Model | Use it when | ImajevBench | MMLU (previous version) | Unknown items right |
|---|---|---|---|---|
| imajev-2b | latency or memory is the constraint | 71.7% | 59.8% | 5 / 21 |
| imajev-4b | the default for almost everything | 82.4% | 74.5% | 14 / 21 |
| imajev-9b | knowledge-heavy text questions, when memory is not tight | 82.1% | 79.2% | 15 / 21 |
5.5 Check it on your own data
- Measure first. Collect a few hundred of your own requests with the right answers, including cases where the right answer is "cannot tell". Send them through the server and compare accuracy at your threshold. The harness in
src/imajev_benchdoes this for records in ImajevBench's format and reports cluster confidence intervals. - Fit your own temperature. The evaluators in
scripts/write one JSONL row per question with its logits.scripts/v1_text/fit_temperature_calibration.py rows.jsonl calibration.jsonfits one temperature per question type and option count, and the server loads the result with--calibration. - Compare on ImajevBench. The benchmark's public records, images and scorer are on Hugging Face. Run your own model and send the row.
5.6 Reproduce the numbers
Every figure on this page is rebuilt from files in the repository by python site/build_site.py. Benchmark predictions, per-run manifests and scores are under results/ and bench/. ImajevBench rows are rescored with python -m imajev_bench score and the paired tests with imajev_bench.stats.paired_cluster_test. JevBench runs keep every raw response.
6Data, training and compute
Each model is a LoRA (rank 16) on the language layers of Qwen3.5 plus a small readout over option codes; the vision encoder stays frozen. Teachers are open-weight models only. No JevBench items, no Jev outputs and no paid-API outputs were used in training.
| Stage | Data |
|---|---|
| 1 · Image and text | About half a million human-labelled decisions from licence-checked public datasets: 300k on images (21 sources) and 200k text-only (15 sources). |
| 2 · New sources | About 416k decisions on six photo and text sources the models had not seen, labelled by the 9B and kept only when two option orders agreed; then 72k photo-vs-record and two-photo decisions. The 2B and 4B were trained on this stage; the 9B, which produced its labels, went from stage 1 to stage 3. |
| 3 · Hard questions | About 23k hard questions: documents and questions written and answered by open-weight teachers, kept only when the teachers agreed, plus licensed human reasoning sets. Stage 1 had cut the 9B's JevBench hard score from 64.9% to 42.3%; this stage took it to 68.5%. The shipped adapters are a weight-space average of two adapters: that one and a soft-target continuation on 39,515 further rows; served, the 9B scores 69.4%. |
6.1 Models
All three use the same adapter: a LoRA with rank 16, α 32, no dropout and no bias, on every language-model projection (q, k, v, o, gate, up, down and the DeltaNet in_proj_qkv, in_proj_z, out_proj). The vision encoder is frozen and has no LoRA. Base weights run in bfloat16; the LoRA and readout weights are stored in float32.
| imajev-2b | imajev-4b | imajev-9b | |
|---|---|---|---|
| Base model, pinned revision | Qwen3.5-2B @15852e8c | Qwen3.5-4B @851bf6e8 | Qwen3.5-9B @c2022362 |
| LoRA parameters | 15,630,336 | 30,474,240 | 40,108,032 |
| Decision readout (bias-free, float32) | 255 × 2048 | 255 × 2560 | 255 × 4096 |
| Readout parameters | 522,240 | 652,800 | 1,044,480 |
| Total trainable parameters | 16,152,576 | 31,127,040 | 41,152,512 |
| Adapter file, float32 | 62.6 MB | 122.0 MB | 160.5 MB |
| Readout file | 2.1 MB | 2.6 MB | 4.2 MB |
| Base weights to download | 4.6 GB | 9.3 GB | 19.3 GB |
| Shipped calibration temperature | 1.65 | 1.72 | 1.75 |
All three base models are Apache-2.0. Each supplied option is bound to one of 255 single-token codes; the readout maps the hidden state at a decision position to logits over those codes plus unknown, and a softmax after the calibration temperature gives the probabilities. One prefill per request, one forward pass per question, no text generation. Requests over 4,096 tokens are refused, not truncated.
6.2 Data by stage
A decision is one typed question answered about one request; a record is one request, which can hold up to eight decisions. No training data comes from JevBench, from Jev, or from any paid API.
| Stage | What it contains | Count |
|---|---|---|
| 1 · Image and text | 36 licence-admitted sources in 466,135 records: 200,000 decisions from 15 text sources (24,832 of them carry an unrelated photo as an irrelevance control), 300,000 from 21 image sources, and 4,000 photo-vs-listing contradictions from Amazon Berkeley Objects. Held out from the same manifest: dev 30,026, calibration 25,217, test 35,528 decisions. | 504,000 train |
| 2 · New sources | The calibrated 9B labelled 616,964 candidate decisions on sources the models had not seen and 475,305 were kept (77.0%): the answer had to be the same under two option orders with top probability at least 0.6 (or unknown at least 0.5). New photo and text sources 366,967 → 271,036 (73.9%); PD12M photos 179,997 → 144,871 (80.5%); photo-vs-record and natural pairs 70,000 → 59,398 (84.9%). Of 71,630 photo-vs-record and two-photo decisions, 45,999 check a photo against a written record, 8,469 compare two natural photos, and 17,162 compare a reference photo with an edited copy, labelled by construction. For the 2B each target was blended 0.5 : 0.5 with the untuned 2B's own distribution. | 475,305 kept of 616,964 |
| 4B first run | The 4B was trained once on a combined mixture instead of stages 1 and 2 in sequence: 504,000 stage-1 decisions with their original labels, 296,482 stage-2 teacher decisions with unknown targets capped at 15%, and 66,372 photo-vs-record and pair decisions. 17.15% of its targets are unknown. No base-model blend. | 866,854 train |
| 3 · Hard questions | Two rounds of documents and questions written by open-weight teachers and kept only when independent answerers agreed, plus 8,532 items from 10 licensed human reasoning sets in round 1. Hard labels only. Then a soft-target continuation: the round-2 teacher rows relabelled with Qwen3.6-35B-A3B probability distributions, 9,880 new hard, judge and programmatic rows, the Eikos decisions strict slice (10,570, open-weight teachers only) and 5,000 image-replay rows. | 14,112 + 7,812 + 39,515 soft train records |
Per-source counts and licences for all 36 stage-1 sources are in section 2 of the full technical specification. For 16 of the 21 image sources the recorded licence covers the annotations only; those photos stay under their upstream terms and are not redistributed. Stage-2 photo sources: PD12M (CC0-1.0), Wikimedia Commons (CC-BY-4.0, CC-BY-3.0, CC0-1.0) and Open Images (CC-BY-2.0).
| Stage 3 | Round 1 | Round 2 |
|---|---|---|
| Writer | Qwen3.6-27B @6a9e13bd | Qwen3.6-27B @6a9e13bd, reasoning on for hard families |
| Answerers | Qwen3.6-27B (reasoning), gpt-oss-20b @6cee5e81 | the two, plus Qwen3.6-35B-A3B @995ad96e (reasoning) |
| Keep rule | both answerers agree with the intended answer | all three agree |
| Documents written | 4,462 | 2,699 |
| Questions / kept | 13,386 / 9,368 | 8,097 / 4,852 |
| Dropped | 4,013 disagreement, 5 JevBench overlap | 1,883 disagreement, 1,362 near-duplicates, 0 JevBench overlap |
| Human reasoning items | 8,532 from ARC-Challenge, CommonsenseQA, CosmosQA, DROP, GSM8K, QASC, StrategyQA, TruthfulQA and WANLI, at most 800 training rows per set; MMLU-Pro 800 for evaluation only | none new |
| Training records | 14,112 (of 17,898) | 7,812: 3,598 new + 4,214 replayed from round 1 |
Round 2 also holds out a 622-record calibration fold from three domains kept out of training (telecom, hospitality, nonprofit grants).
6.3 Training by model
Every stage used the same trainer (scripts/train_decision_lora_torch.py, PyTorch and PEFT, 4 GPUs): cross-entropy on float32 readout logits over the listed options, soft cross-entropy where a record carries a target distribution, no label smoothing; AdamW with weight decay 0 and betas (0.9, 0.999); linear warm-up, then cosine decay to 10% of the peak rate; gradient clipping 1.0; seed 0. Each step is 2 micro-batches per GPU × 4 GPUs, each at most 40 examples under a padded-token budget. Stage 1 and the 2B's first run kept the checkpoint with minimum dev loss; later stages kept the best mean accuracy over the reasoning dev set and the JevBench-style dev set (6.5).
| Model | Stage | Started from | Hardware | Epochs | Peak LR | Steps kept / total | Step time |
|---|---|---|---|---|---|---|---|
| imajev-2b | initial run (594,214 records, earlier image mixture) | Qwen3.5-2B | 4×H100 | 1 | 2e-4 | 2,614 / 2,614 | 1.3 h |
| imajev-2b | 1 | initial run | 4×H100 | 1 | 1e-4 | 2,495 / 2,495 | 1.3 h |
| imajev-2b | 2, first pass | stage 1 | 4×H100 | 0.4 | 5e-5 | 1,100 / 1,866 | 1.0 h |
| imajev-2b | 2, pairs pass | first pass | 4×H100 | 1 | 5e-5 | 1,361 / 1,361* | not recorded |
| imajev-2b | 3, round 1 | stage 2 | 4×H200 | 2 | 3e-5 | 250 / 256 | 8 min |
| imajev-2b | 3, round 2 | round 1 | 4×H100 | 2 | 2e-5 | 80 / 220 | 9 min |
| imajev-2b | 3, soft-target continuation | round 2 | 4×H100 | 2 | 2e-5 | 350 / 620 | 22 min |
| imajev-4b | first run | Qwen3.5-4B | 4×H200 | 0.5 | 1.5e-4 | 1,900 / 2,595 | 1.9 h |
| imajev-4b | 3, round 1 | first run | 4×H200 | 2 | 3e-5 | 250 / 303 | 12 min |
| imajev-4b | 3, round 2 | round 1 | 4×H100 | 2 | 2e-5 | 266 / 266 | 22 min |
| imajev-4b | 3, soft-target continuation | round 2 | 4×H100 | 2 | 2e-5 | 260 / 747 | 62 min |
| imajev-9b | 1 | Qwen3.5-9B | 4×H200 | 1 | 2e-4 | 3,100 / 3,508 | 2.6 h |
| imajev-9b | 3, round 1 | stage 1 | 4×H200 | 2 | 3e-5 | 300 / 404 | 22 min |
| imajev-9b | 3, round 2 | round 1 | 4×H100 | 2 | 2e-5 | 260 / 365 | 20 min |
| imajev-9b | 3, soft-target continuation | round 2 | 4×H100 | 2 | 2e-5 | 510 / 1,018 | 65 min |
Step time is summed optimizer-step time. *Chosen by hand on probe results. Each shipped adapter is the element-wise average (LoRA and readout) of the round-2 adapter and the kept checkpoint of the soft-target continuation; the continuation was trained with soft-target cross-entropy, a rationale loss (weight 0.3, up to 192 tokens) and option permutation. Gradient checkpointing only for the 9B in stage 3. The 9B's stage 1 crashed eight times in its first 980 steps (a periodic diagnostic timer in the trainer, since removed) and resumed from 20-step checkpoints; the 2B's initial and stage-1 runs had similar resumes. No data was skipped.
6.4 Compute and cost
About $676 of rented GPU time for the whole project (RunPod, 17 pods, 21 to 24 September 2026; $499.07 for the first three stages, about $177 for the soft-target continuation), every run included: data labelling and generation, all training, all evaluations, the competitor benchmark runs and aborted pods.
| Use | Hardware | Cost |
|---|---|---|
| 9B stage 1 | 4×H200 | $71.67 |
| 2B stage 2 (labelling, blending, training) | 4×H100 | $95.43 |
| 4B first run, stage-3 round-1 data generation, round-1 training of all three models, evaluations | 4×H200 | $129.36 |
| Stage-3 round-2 data generation, round-2 training, a frozen 35B reference check, competitor benchmark runs | 7×H100 | $99.88 |
| 2B initial run and stage 1 with their evaluations (three pods) | 4×H100 | $92.69 |
| ImajevBench base-model and generation rows | 1×H100 | $4.91 |
| Release calibration check | 1×H100 | $2.32 |
| Soft-target continuation: relabelling, training of all three models, evaluations, weight averaging | 8×H100 | ≈ $177 |
| Short-lived and aborted pods | various | $2.81 |
| Total | ≈ $676 |
6.5 Evaluation sets
| Set | What it is | How it is kept out of training | Size |
|---|---|---|---|
| ImajevBench v2.0-lite test | photo, state and photo+state typed decisions in 89 evidence clusters, 21 with an unknown answer; AI-generated images | built after training; an after-the-fact check found no shared images or questions (byte hashes, no perceptual hash) | 279 |
| JevBench public splits | hard / original / easy, text-only typed decisions | 8-gram overlap lint on all teacher data; a re-check found no row sharing 2 or more 8-grams | 111 / 72 / 48 |
| MMLU-1000 | official MMLU test split, 17 to 18 per subject, also sent with an unrelated photo | registered as never-train; 589 matching MMLU-Pro items removed from stage 3 | 1,000 |
| Irrelevance panel | the 2,000 MMLU rows plus 489 ABO and 334 VizWiz controls that score each donor photo with its own relevant question | donors from the test partition; no panel image in training | 2,823 |
| Typed-decisions test | synthetic workflow decisions (400 records × 5); gold is agreement with that dataset's teacher model | evaluation only | 2,000 |
| SST-5 | official Stanford Sentiment Treebank test sentences | evaluation only, not redistributed | 2,210 |
| Reasoning dev set | 6,000 from the typed-decisions train split plus 240 hand-written items in 8 families | not trained on, but used to pick checkpoints in stage 2 and stage-3 round 1, so its scores are not independent of selection | 6,240 |
| JevBench-style dev set | template-generated items in 10 families, none from JevBench | not trained on; used to pick round-2 checkpoints and to fit the shipped calibration temperatures | 150 |
| State probe / pairs probe | photo-vs-record and two-photo questions from the training templates | 10 of the 200 state-probe items reuse photos seen in training; the pairs probe shares no images | 200 / 60 |
| Held-out photo sources | MM-UPD, TUBench, CountQA, LIVE in the Wild, Fashionpedia, MuirBench, plus 177 real two-photo pairs | sources never used in training; no shared images | 4,989 |
6.6 Checks
- JevBench. Every stage-3 teacher question was linted against the public JevBench files (62,915 reference 8-grams); a question sharing two or more was dropped (5 in round 1, 0 in round 2). A re-run over both training manifests found no row at the threshold.
- Evaluation panels. An audit gate fails any build that places MMLU, SST-5, typed-decisions test or any upstream test row outside the test partition. A re-check over the training partitions of all six training manifests found no question or long state shared with MMLU-1000, typed-decisions test, SST-5 or the reasoning dev set.
- ImajevBench. Nothing excludes it by construction. An after-the-fact comparison of all 923 benchmark images and every record's text against all training partitions found no overlap. Training images were re-encoded, so a byte-hash check would miss the same photo saved differently; no perceptual-hash check was run.
- Licences. Every training source is admitted with a licence receipt; 16 image sources are admitted for their annotations only, with photos under upstream terms and not redistributed.
The complete record, including the per-source table, is the technical specification in the repository.