Jev on agent tool-call risk
A small, honest benchmark on one realistic job: sorting an agent's tool calls into risk classes before they run. The set is hand-labelled and deliberately mixed in difficulty, several backends are run against an identical task definition, and the write-up is clear that this measures one-shot classification, not the multi-step workflows the vendor's comparisons are drawn from.
- agents
- tool-calls
- risk-classification
- reproducible
- From
- themsquared
- License
- Apache-2.0
- Language
- Python
- GitHub stars
- 1 (2026-09-26)
- Published
- 2026-09
- Listed here
- 2026-09-26
Sources checked
Also in this category
Evaluations- Evaluations
Jev out-of-distribution calibration test
Asks the question that matters for a model sold on calibration: does the calibration survive on a task the model cannot have seen? It pairs public benchmarks with rule-generated tickets whose labels never appear in the text, and reports error against a noise floor rather than in isolation. The useful conclusion is that miscalibration differs by question type, so one temperature for the whole model is the wrong unit.
scienthoon7 stars2026-09-19 - Evaluations
PriorBench: pre-registered evaluation of Jev
The methodologically strictest evaluation listed here: fifty predictions were written down and timestamped before any data was collected, and the repository reports how many of them were falsified as prominently as how many held. The finding that travels furthest concerns the confidence value, which the authors found flat across most of its range and informative only at the very top.
PriorBench0 stars2026-09-20 - Evaluations
LocalJev model bake-off (2026-09-18)
Not a test of Jev, but of what you give up by running a compatible server on local open models. The same engine is run against five installed checkpoints on three public gold-labelled tasks, twice: once on short inputs, and once with the same target buried in thousands of words of unrelated text. The second condition is where the smaller models come apart.
GitHub Next2026-09-18