🔍 Read the full analysis: A Practical Guide To Post-Training Agents With Database Grading on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub and uses executable checks of final database states as training rewards. In one 50-step GRPO run, Qwen3.8-27B’s average success rate rose from 51.70% to 60.78%, while tasks passed in all 20 attempts increased from 38 to 107; tasks solved at least once fell from 453 to 446.
Microsoft and Hugging Face say ThinkingBox-Training is now available on GitHub, offering a way to post-train agents using executable checks of the database records they leave behind, as detailed in the original analysis. In one reported 50-step GRPO run, Qwen3.8-27B achieved a higher average success rate on a 507-task benchmark, but solved slightly fewer tasks at least once, showing a reliability gain alongside a small decline in task coverage.
The teams evaluated the model on 507 benchmark tasks, each run 20 times. The authors report that successful attempts rose from 5,242 of 10,140 (51.70%) before training to 6,163 (60.78%) after training. Tasks passed in all 20 attempts increased from 38 (7.50%) to 107 (21.10%), a measure the report labels Observed 20/20.
Coverage moved in the other direction. The number of tasks solved at least once across their 20 attempts, labeled pass@20, decreased from 453 (89.35%) to 446 (87.97%). That is seven fewer tasks solved at least once. The figures come from the authors’ reported run; they do not establish how the system would perform in production.
For training, the team used a pool of ThinkingBox-format practice tasks separate from the evaluation benchmark. The report says the pool was selected using estimated difficulty for the base model, then kept fixed through training. Each task’s executable check assessed whether the expected records were present in the backend after the agent finished. The authors say this rewards the outcome rather than a prescribed sequence of tool calls.
Why Final Database State Matters
For agents that can change business records, a plausible conversation or a sequence of seemingly reasonable tool calls does not prove that the requested action was completed correctly. A check of the final database state can score whether required records are present and whether the resulting state matches the task’s expected outcome. This gives developers a concrete training signal for tasks where the consequence is a changed booking, account, or other operational record.
The reported results also show why average success alone is not enough to judge an agent. The increase in tasks passed on every recorded attempt suggests more repeatable performance on some benchmark tasks, while the fall in pass@20 indicates that the trained model did not solve as many distinct tasks at least once. Teams weighing the approach would need to consider both consistency and coverage, as well as whether the benchmark resembles their own workflows.
As an affiliate, we earn on qualifying purchases.
From Database Benchmark to Training
ThinkingBox was initially presented as a way to evaluate agents based on the database records left at the end of a task. The benchmark described by the authors has 507 tasks, each involving a backend, MCP tools and a simulated user with private context. Executable checks compare the final state with the expected result, rather than requiring the agent to reproduce a particular human procedure.
The training report extends that grading approach into post-training. The team says it used tasks in the same format to train Qwen3.8-27B with GRPO for 50 steps, then evaluated a selected checkpoint on a separate benchmark. The authors describe the evaluation as out of distribution because its domains, databases and most MCP tools were not encountered during training. That description refers to the benchmark setup, not evidence of performance in live operations.
In one illustrative hotel-booking task, a traveler requested a room downgrade that required manager approval. The authors say the untrained agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved the booking and payment state in all 20 attempts, and the expected records matched in 19. This is an example from one task, not a general success rate for hotel-booking agents.
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”
— Microsoft and Hugging Face, in their joint post
Limits of the Reported Run
The reported evidence covers one 50-step training run. The supplied material does not provide results across different random seeds, training-pool selections or run lengths, so it is unclear how stable the reported gains are. It also does not give a full comparison with alternative reward designs or training methods, limiting conclusions about which part of the approach drove the change.
The authors report a seven-task decline in pass@20 but do not establish why coverage fell. Nor does the supplied account report trials in a live production setting or provide details on the training process’s cost. Production reliability, broader generalization and reproducibility remain open questions; benchmark results alone do not answer them.
Reproduction and Broader Testing
With the stack available on GitHub, researchers and developers can inspect the implementation and attempt their own training runs. The next useful evidence would include independent replications and results across more models, task pools, seeds and training durations, including whether the reliability and coverage tradeoff persists.
Before deployment decisions, teams would also need evaluations tailored to their intended workflows. Those tests should check for correct final records as well as incomplete or unauthorized changes, and report both repeatability and the range of tasks an agent can handle. No production deployment results or further milestones are specified in the supplied account.
Key Questions
What is ThinkingBox-Training?
It is a post-training stack that Microsoft and Hugging Face say is available on GitHub. It uses executable checks of an agent’s final database state as rewards during training.
How did Qwen3.8-27B perform after training?
In the reported 50-step run, its average success rate on a 507-task benchmark rose from 51.70% to 60.78%. Tasks passed in all 20 attempts rose from 38 to 107, while tasks solved at least once fell from 453 to 446.
What does pass@20 mean in this report?
It is the number and share of benchmark tasks the model solved at least once across 20 attempts. It fell from 453 tasks (89.35%) before training to 446 (87.97%) afterward.
Do the results show the system is reliable in production?
No. The supplied results are from a benchmark evaluation and a single reported training run. The source does not provide live production results, independent replications or enough information to determine performance across different training setups.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
