Test a cheap sorting AI on your own cases first

Who pays

This is a playbook, not a product. You use it before selling decision-model work so your accuracy claims hold up on your own labeled cases.

How to build it

1. Gather and label real decisions

   Collect 50-100 of your real decisions in Google Sheets or Supabase. Label the correct answer for each one.

2. Pick models to compare

   Pick two or more deciders to compare, such as Jev, Cloudflare Clef or Clef-flash, Perplexity pplx-decider, or a local Kev or Laya run through llama.cpp /v1/systemone.

3. Test short and long prompts

   Run each sample twice: once with a short question and once with your full production prompt. Long prompts made models worse in one test.

4. Score accuracy and cost

   Use Claude Code to write a small script that scores accuracy, escalation rate, and cost per correct decision for each decider. Count any fixed extra tokens per request.

5. Check against a custom-trained model

   Compare the results against a small fine-tuned model for your task, such as Qwen3-4B. Only build client work on a decider that wins on your own data.

Sources on X

• _Pranjal · Oct 7, 2026

  In the poster's own test on 2,496 real pairs, the decision models were asked to keep 95% of relevant items. Used without any task training, they still sent 70% to 99% of traffic to the expensive judge. The poster's own fine-tuned Qwen3-4B sent only 32%. Full production prompts made the models worse, but the poster still rates them as genuinely interesting for short routing, moderation, tool selection, and fixed-option decisions.

• ziyacivan · Sep 26, 2026

  The poster's own measurement routed Claude Code turns through Jev for 3 days, and main-chat routing saved only about $0.03. 73% of tokens were tool-loop steps that a per-turn decision can't reach, and the savings came from routing subagents. This was one machine, and costs were estimated from list prices.

• felix94123 · Sep 25, 2026

  The poster's own benchmark on 362 fresh items found Jev and Kev-4B within 2 points of accuracy on every task. Jev was ahead on paraphrase detection. The two have the same list price, but Jev counts a fixed ~257 extra input tokens per request, so short requests cost up to 12x more.

• 0xTreff · Oct 2, 2026

  Existing accounts kept working. This was reported as of 2026-10-02.

• FareaNFts · Oct 5, 2026

  An article comparing Jev, Clef, Strands Decider, Glide, and pplx-decider notes that almost every speed and accuracy number in this category is the company's own benchmark. It advises running 50 of your real decisions through two models before switching.