Scorecards run on 2 November 2026
Which model is best for your task? Only a test says.
Three tasks, scored the same way on the same day. On one, the newest GPT wins. On two, a free open model trained on the team's own data beats it. The method is the product; the answer changes with the task.
Each bar is one model option, measured on a fixed test set the team keeps as a plain file. The green bar is the winner for that task. Cost is per 1,000 answers; "0" means it runs on the team's own machine with no per-call fee.
What Ownbench is
Ownbench is a service by Housedevinci for teams that run a model they did not build. Most of them never know if the model they have is still the best for their task, or the cheapest, because the test that would tell them lives inside the vendor's tools. We give them that test, and the move.
- Your test set. We take 200 of your real cases and turn them into a test file with the right answers. You own it, as a plain file.
- Your scorecard. We run every option on it: your current model, open models trained on your data, the newest closed models with a prompt. Accuracy, speed, cost, side by side. The three cards below are what it looks like.
- The move. We move you to the winner, on your own machines or on the host you choose. If the winner is the model you already have, we say so.
- The re-score. Every time a vendor retires a model, raises a price or ships a better one, we run your test again and tell you if it is time to move. That is the part that lasts.
The first scorecard is free. The move is a fixed price. The re-score is a small monthly fee. We started with the teams whose fine-tuned OpenAI models stop on 14 January 2027, because they have a date.
Bank support messages, sorted into 77 intents
Public data, the shape of job many teams tuned GPT for: a short customer message in, one intent label out. 200 test cases, 2,000 training examples.
Share of the 200 test messages labelled right.
Verdict: move to the newest GPT with a prompt.Training a small model helped, but not enough. The open model wins only on speed and on cost.
Show the full numbers
| Option | Right of 200 | 95% range | Invalid answers | Median time | Cost per 1,000 |
|---|---|---|---|---|---|
| Small open model (4B), prompt only | 115 | 51 to 64% | 7 | 0.9 s | 0, own hardware |
| Same model, trained on 2,000 examples | 138 | 62 to 75% | 22 | 0.3 s | 0, own hardware |
| Newest GPT, prompt only | 170 | 79 to 89% | 0 | 1.5 s | about 1.05 USD |
Plain recipes, rewritten in a structured recipe language
A real team's task, from the training pairs the team keeps in a public repository: a recipe in, the same recipe in the Cooklang format out. 100 test recipes from one country, 2,000 training pairs from the others. Scored without the team; nothing of theirs leaves the test machine.
Share of the reference ingredients found in the output.
Verdict: own the model.A free model trained on the team's data finds more of the ingredients than the newest GPT and costs nothing per call. GPT is faster per recipe on a laptop; on a server the gap closes.
Show the full numbers
| Option | Valid output, of 100 | Ingredients found | Step count within one | Median time | Cost per 1,000 |
|---|---|---|---|---|---|
| Small open model (4B), prompt only | 57 | 48% | 10 | 21 s | 0, own hardware |
| Same model, trained on their pairs | 100 | 92% | 68 | 31 s | 0, own hardware |
| Open 8B model, trained on their pairs | 100 | 95% | 56 | 39 s | 0, own hardware |
| Newest GPT, prompt only | 100 | 80% | 46 | 10 s | about 7.92 USD |
Venue descriptions, turned into vibe tags
A real team's task, from the 2,046 training pairs the team keeps in a public repository: a venue's text in, a short description and a set of tags out. 200 test venues. Same rules as the recipe task.
Tag score: how well the model's tags overlap the team's own tags, from 0 to 1.
Verdict: own the model.The trained 8B model matches the team's tags far better than the newest GPT with a prompt, and every one of its answers came back in the right shape.
Show the full numbers
| Option | Valid shape, of 200 | Tag score | Median time | Cost per 1,000 |
|---|---|---|---|---|
| Small open model (4B), prompt only | 200 | 0.24 | 3.6 s | 0, own hardware |
| Same model, trained on their pairs | 200 | 0.38 | 3.6 s | 0, own hardware |
| Open 8B model, trained on their pairs | 200 | 0.54 | 11 s | 0, own hardware |
| Newest GPT, prompt only | 200 | 0.31 | 6.1 s | about 3.44 USD |
What the three say together
The newest closed model wins the general task. A free model trained on the team's own data wins the team's own task.
- Nobody can know which case they are in without the test.
- The test set, kept as a plain file, is the asset. Every future move is then one afternoon.
- We have no model to sell, so the verdict can be "stay", and on the first card it is.
How the cards were made
- A fixed test set, drawn with a fixed seed, never seen in training.
- The same prompt for every "prompt only" line, the same answer-reading rule for every line.
- Small open model: a 4 billion parameter instruct model in 4-bit, trained with LoRA on a laptop (Apple M1 Max), 4 to 60 minutes per task, under 15 GB of memory.
- Open 8B model: Llama 3.1 8B Instruct, LoRA trained on a hosted GPU for a fixed 4 USD per task, then tested on the laptop.
- Newest GPT: the newest general model the API listed that day, prompt only, output limited to the answer.
- Costs from the providers' public price lists that day. Ranges by the Wilson method.
- Every run has a log with the exact commands, times, memory and failures.
Limits, stated plainly
- The recipe and venue cards were scored from the teams' public training data, without the teams. Their own tuned models were not available, so the "your current model" line is missing; on a real engagement it is the first line.
- The bank card is public data, not a customer.
- On the venue card the small model's training went unstable after 300 steps; its line uses the step-300 checkpoint, which is why the 8B line is the fairer "trained" comparison.
- The newest GPT refused a fixed-answer setting and spent hidden reasoning tokens on some calls; that affects its time and cost, and it is in the logs.
- Times for the open models are on a laptop; a server is several times faster.
- The two teams' data, trained models and test sets were deleted after the cards. Only these numbers remain.