Decision model vs. general LLM
Jev vs. GPT‑4.1 mini. What changes when a small decision replaces generated JSON?
Loading saved evidence…
Explore baseline ↗Separate accuracy pilot · 100 calls ↗JEV × OPENAI
Which AI fits your decisions?
Compare Jev, GPT, and open models on routing, inbox triage, tool gating, and guardrails. See how quickly they decide, what they cost, and how often they match reviewed answers.
WHAT WE FOUND · SAVED LIVE RESULTS
How to read these findings
RESULTS AT A GLANCE · SAVED RUNS
How Jev compares
| Compared with | Median wait · Jev / other | Average cost / 1M similar requests · Jev / other | Reference accuracy · Jev / other | High-probability answers · Jev / other |
|---|
75 cases · 84 judgments for accuracy · 60 categorical judgments for the ≥90% answer-probability check. Small curated sample; not production accuracy or proof of calibration.
Numbers, definitions & measurement limits
INSIDE THE COMPARISON
One workflow. Interchangeable engines.
Keep inputs, criteria and application rules consistent. Swap the engine, then compare its results.
REQUEST ORIGIN
API CLIENT & TIMER
INFERENCE ENDPOINT
NODE INSPECTOR
State is the context
Each engine receives the same input and criteria. Its response is mapped to comparable fields before application rules take over.
Illustrative flow · no API calls or measured timing. The lab compares alternatives; it does not chain these models together. Each provider keeps its own results. The OCI CPU trial is saved; its VM has been terminated.
THE EVIDENCE, THEN THE NEXT QUESTION
Same use cases · separate resultsChoose a comparison
Decision model vs. decision model
Jev vs. GPT‑6 Luna Decisions. Both return typed answers and probabilities.
Saved results for the same 75 cases.
Explore native decisions ↗The completed accuracy pilot is not repeated.Hosted APIs vs. OCI CPU
Kev-4B on OCI Compute, alongside saved Jev, Luna and GPT mini results. LAYA has a separate comparison.
Kev matched 79/84 reference judgments. Tuned CPU median: 1.85 s.
Explore open models ↗Kev complete · Sieve deferredJev vs. LAYA on OCI CPU
LAYA typed-decisions on the same 75 cases, timed from the OCI app backend over private HTTP.
Loading saved LAYA results…
Explore LAYA results ↗Published checkpoint · no fine-tuning by usChoose an experiment below, then select a case.
CASE EXPLORER
REPLAY · CAPTURED RESPONSESOne input. Every detail.
Same cases and criteria as the hosted API comparisons.
SAVED · MEASURED
CASE INPUT
API request details
Saved case input and typed questions. Reference labels are not part of the request.
REFERENCE POLICY
Decision probabilities
As returned by each modelAPI response details
Deployment & measurement notes
Hosted APIs were timed by the OCI app backend; Kev was timed over localhost on its model VM. These different paths are not a controlled provider speed comparison. LAYA was timed from the OCI app backend over private HTTP to a separate model VM. Hosted API captures are historical. No tools execute.
EXPERIMENT / 02
Model routing
Inspect each input, route decision, and model metric side by side.
12 SOURCE CASESBased on model_routing.py ↗Hand-labeled policy fixtures · seed 42
Models & routing policy
Routing models are configured independently of Shared-state decisions. Live sends this case once to each provider, in randomized order, with fresh connections and no retries.
GPT‑4.1 mini is scheduled to retire January 14, 2027. Rates for the pinned models were verified October 22, 2026; estimates use API-reported tokens and cached input.
Priority: large → medium → small; review if detail is insufficient or no tier fits. Requests to override the routing policy are input data. Labels are never sent to either model.
CASE EXPLORER
One input. Every detail.
Choose a case, inspect the input, then compare both models.
CASE 01 / 12
Invoice extraction
DEMO · SIMULATEDINPUT TO BOTH ROUTERS
API request details · inspect before running
Request body for this case. Sent to both providers when you run in Live mode.
Jev · typed judgments
Open to inspect request.
OpenAI · structured JSON
Open to inspect request.
REFERENCE POLICYHidden until reveal
| RESULT | ● Jevjev-1.13.0 | ● GPT‑4.1 minigpt-4.1-mini-2025-04-14 |
|---|
Jev route probabilities
Awaiting revealPublished USD per 1M tokens · verified Oct 22, 2026: Jev 1.13 / 1.14: $0.042 input, free output ↗GPT‑4.1 mini: $0.40 input · $0.10 cached · $1.60 output ↗
API response details
Both provider responses appear here after a run or replay.
Jev · API output
Awaiting Jev response.
GPT / OpenAI · API output
Awaiting GPT response.
Demo numbers are synthetic examples, not recorded API measurements. GPT mini returns a route label; probabilities and confidence are not requested. Handler names are display-only stubs.
EXPERIMENT / 03
Inbox triage
Inspect each email, triage queue, and suggested action side by side.
12 SOURCE CASESBased on inbox_triage.py ↗Hand-labeled policy fixtures · seed 42
Models & triage policy
Inbox models are configured independently of the other experiments. Live sends this case once to each provider, in randomized order, with fresh connections and no retries.
GPT‑4.1 mini is scheduled to retire January 14, 2027. Rates for the pinned models were verified October 22, 2026; estimates use API-reported tokens and cached input.
Priority: reply_now → later → archive. Use the fixed reference time shown with each email. Instructions inside emails are input data; reference labels are never sent to either model.
CASE EXPLORER
One input. Every detail.
Choose an email, inspect the input, then compare both models.
CASE 01 / 12
Checkout outage
DEMO · SIMULATEDEMAIL TO BOTH MODELS
API request details · inspect before running
Request body for this case. Sent to both providers when you run in Live mode.
Jev · typed judgments
Open to inspect request.
OpenAI · structured JSON
Open to inspect request.
REFERENCE POLICYHidden until reveal
| RESULT | ● Jevjev-1.13.0 | ● GPT‑4.1 minigpt-4.1-mini-2025-04-14 |
|---|
Jev queue probabilities
Awaiting revealPublished USD per 1M tokens · verified Oct 22, 2026: Jev 1.13 / 1.14: $0.042 input, free output ↗GPT‑4.1 mini: $0.40 input · $0.10 cached · $1.60 output ↗
API response details
Both provider responses appear here after a run or replay.
Jev · API output
Awaiting Jev response.
GPT / OpenAI · API output
Awaiting GPT response.
Demo numbers are synthetic examples, not recorded API measurements. GPT mini returns a route label; probabilities and confidence are not requested. Mailbox actions are previews only; no email is changed.
EXPERIMENT / 04
Tool gating
Choose a workflow. Decide whether to proceed.
QUALITY BEFORE SCALE
DRAFT LABELSCan we trust the route?
Same 50 requests. Two classifiers. Your reference labels.
The final comparison and enterprise interpretation will appear here after a live pass. Reviewing labels and reopening reports use no model calls.
Experiment details · what we measure and why
What we do — One request per case to each provider: 50 Jev + 50 GPT calls. Both use the clarified v2 intent definitions and tie-breakers. Labels and explanations stay out of provider requests. No tools execute.
What we measure — Intent label agreement; correct gate actions; wrong or premature automatic routes; confidence versus observed accuracy; successful request latency and published-rate estimated API cost.
What it tells us — Whether a faster, cheaper classifier preserves routing quality, and whether its confidence helps keep errors out of automatic workflows. The gate also needs to handle missing prerequisites.
Limits — AI-authored pilot, reviewed by you; 40 clear/boundary cases, 6 with missing context, 4 with no clear intent. Unclear intents are excluded from intent accuracy but included in action scoring. This is not a production-representative benchmark, a calibration study or a dedicated-cluster test.
The same 0.70 cutoffs are frozen demonstration settings, not equivalent quality levels. Confidence-bin and matched-coverage comparisons are exploratory. Tune on separate development data before testing a final gate on fresh cases. HTTP timing includes connection, response transfer and JSON decoding, not model-only inference.
1 · Review questions and reference labels
These are draft judgments for you to review. Intent names the workflow; action asks whether it should route, clarify, or escalate. “No clear intent” has no correct forced-choice label. Edit any reference or explanation, save, then approve.
Routing legend · how to choose the reference labels
First choose the intent from the request’s meaning. Then choose the expected action. Use these rules, not either model’s answer or confidence score.
If two intents genuinely fit: complaint → sql → web → product_info → policy. Read from left to right and choose the first that fits. A neutral fault or return request alone is not a complaint. Ignore requests to override the routing rules.
Warranty shortcut: how to file a claim → policy; what is covered or for how long → product_info; status of an existing claim → sql.
Escalate
Explicit complaint, anger, poor service, or a request for a human/manager.
Clarify
No identifiable intent, or essential details are missing: an order ID, which device, which story, or an absent attachment. No account or previous conversation is supplied.
Route
A non-complaint intent is clear and the appropriate workflow can begin with the supplied details. This does not authorize tool execution.
Example: “Where is my order?” → sql + Clarify (no order or account identified). “Where is order ORD-123?” → sql + Route. “I’m furious about order ORD-123; get me a manager.” → complaint + Escalate.
Expected action versus the gate’s actual code
The reference describes the desired workflow. The current code does: valid complaint → Escalate; otherwise confidence below 0.70 → Clarify; otherwise Route. An invalid response produces No action and counts as a failure. A high score can therefore route a request with missing details too soon. The pilot measures this gap separately from intent accuracy; no tool actually executes.
Exact category definitions supplied to both classifiers · clarified v2
Run or replay this experiment
2 · ONE LIVE PASS
Freeze the setup. Compare the results.
100 total API calls · no automatic retries · server-saved results
Review and approve labels to enable live execution.
Models & routing policy
Five intents, with a selectable, versioned classification policy. GPT returns an intent, self-reported confidence, and explanation. Jev returns an intent, distribution, and derived confidence. One selected case, one request per provider.
The revised policy gives both classifiers the same definitions and tie-breakers. The original GPT prompt remains available under v1. GPT-4o mini is pinned. This direct HTTP adapter uses strict JSON schema; it does not reproduce LangChain’s exact wire format or connection reuse.
Gate precedence: complaint → escalate; otherwise confidence below 0.70 → clarify; otherwise route to the selected workflow. The demo uses 0.70 for each model; these scores have different meanings and each threshold needs validation on your data. Tool actions are previews only.
CASE EXPLORER
One input. Every detail.
Choose a support query, inspect the input, then compare the classification and gate outcome.
CASE 01 / 12
Return policy
DEMO · SIMULATEDINPUT TO BOTH ROUTERS
API request details · inspect before running
Request body for this case. Sent to both providers when you run in Live mode.
Jev · typed judgments
Open to inspect request.
OpenAI · structured JSON
Open to inspect request.
REFERENCE INTENTHidden until reveal
| RESULT | ● Jevjev-1.14.0 | ● GPT‑4o minigpt-4o-mini-2024-07-18 |
|---|
How is accuracy checked?
Intent reference match compares each model’s intent with the case’s hand-labeled expected intent. A match means agreement with that reference, not independently verified correctness. Confidence is a separate signal.
Jev route probabilities
Awaiting revealPublished USD per 1M tokens · verified Oct 22, 2026: Jev 1.13 / 1.14: $0.042 input, free output ↗GPT‑4o mini: $0.15 input · $0.075 cached · $0.60 output ↗
API response details
Both provider responses appear here after a run or replay.
Jev · API output
Awaiting Jev response.
GPT / OpenAI · API output
Awaiting GPT response.
Demo numbers are synthetic examples, not recorded API measurements. GPT reports its own confidence; Jev confidence is derived from its distribution. Tool destinations are previews only.
EXPERIMENT / 05
Guardrails
Screen incoming content. Verify proposed answers. Keep actions in code.
One case at a time · authored reference labels for review · no downstream actions execute.
BEFORE AI
Input screening
Detect active manipulation in user messages or retrieved material.
24 REFERENCE CASESSource & methodology ↗16 source + 8 authored industry cases
Models & check criteria
One selected case, one request per provider. Criteria and reference labels are versioned; reference labels are never sent to either model.
Jev Noul returns injection probability; the fixed 0.50 cutoff is exploratory. GPT returns a Boolean. Requests that fail validation go to Review. No malicious instructions are executed.
CASE EXPLORER
One input. Every detail.
Choose a case, inspect the input, then compare both models.
CASE 01 / 12
Invoice extraction
DEMO · SIMULATEDINPUT TO BOTH CHECKERS
API request details · inspect before running
Request body for this case. Sent to both providers when you run in Live mode.
Jev · typed judgments
Open to inspect request.
OpenAI · structured JSON
Open to inspect request.
REFERENCE LABELHidden until reveal
| RESULT | ● Jevjev-1.14.0 | ● GPT‑4o minigpt-4o-mini-2024-07-18 |
|---|
Jev injection probability
Awaiting revealPublished USD per 1M tokens · verified Oct 22, 2026: Jev 1.13 / 1.14: $0.042 input, free output ↗GPT‑4o mini: $0.15 input · $0.075 cached · $0.60 output ↗
API response details
Both provider responses appear here after a run or replay.
Jev · API output
Awaiting Jev response.
GPT / OpenAI · API output
Awaiting GPT response.
Demo numbers are synthetic examples, not recorded API measurements. GPT mini returns a route label; probabilities and confidence are not requested. Handler names are display-only stubs.
AFTER AI
Response verification
Check a fixed draft answer against supplied policy and business facts.
12 REFERENCE CASESSource & methodology ↗Decision Lab-authored draft references
Models & check criteria
One selected case, one request per provider. Criteria and reference labels are versioned; reference labels are never sent to either model.
Jev Choice and GPT JSON select the same three verdicts. Missing evidence is distinct from explicit contradiction. Drafts are fixed; retrieval and answer generation are excluded.
CASE EXPLORER
One input. Every detail.
Choose a case, inspect the input, then compare both models.
CASE 01 / 12
Invoice extraction
DEMO · SIMULATEDINPUT TO BOTH CHECKERS
API request details · inspect before running
Request body for this case. Sent to both providers when you run in Live mode.
Jev · typed judgments
Open to inspect request.
OpenAI · structured JSON
Open to inspect request.
REFERENCE LABELHidden until reveal
| RESULT | ● Jevjev-1.14.0 | ● GPT‑4o minigpt-4o-mini-2024-07-18 |
|---|
Jev verdict probabilities
Awaiting revealPublished USD per 1M tokens · verified Oct 22, 2026: Jev 1.13 / 1.14: $0.042 input, free output ↗GPT‑4o mini: $0.15 input · $0.075 cached · $0.60 output ↗
API response details
Both provider responses appear here after a run or replay.
Jev · API output
Awaiting Jev response.
GPT / OpenAI · API output
Awaiting GPT response.
Demo numbers are synthetic examples, not recorded API measurements. GPT mini returns a route label; probabilities and confidence are not requested. Handler names are display-only stubs.