🔍 Read the full analysis: Give AI Agents A Trial By Fire Before They Handle Business Work on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Firmulate says five AI models completed a simulated crisis week at a small software company, with scores ranging from 73 to 95; a do-nothing baseline scored 26. The company is offering pilots that use read-only data exports to assess how agents handle business scenarios without writing to live systems.
Firmulate has completed a five-model business simulation and is offering companies pilots to test AI agents against scenarios built from their own data. The pilot uses a read-only export and does not write back to company systems, giving businesses a way to examine agent decisions before considering them for live work.
In the final Crucible League, completed in July 2026, five models ran the same small software company through a difficult week. Firmulate reports that GPT-5.6-Sol scored 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. Decisions were versioned and auditable, and partial progress counted toward scores.
The scenario tested crisis response, commercial judgment and resistance to manipulation. Firmulate says all five models spotted every crisis and refused every manipulation attempt, including staged fake CEO messages and a reporter’s request for an on-background answer. The separation came in execution: only two models signed a €55,000 deal after their own analysis supported it.
Firmulate says the deal depended on a competitor weakness hidden two document references deep in the company’s files. Models that found it closed at full price, adding €4,583 in monthly recurring revenue. The result illustrates a gap between identifying a problem and using relevant information already held in a business’s records.
Give AI Agents a Trial by Fire Before They Handle Business Work
Firmulate’s Crucible League put five AI models through a simulated crisis week at a small software company. All five spotted every crisis and refused every manipulation — but only two closed a €55,000 deal their own analysis supported. Detection is not execution.
The Scoreboard: One Difficult Week
Five models ran the same small software company through crisis after crisis. Decisions were versioned and auditable, and partial progress counted toward scores. Note: Kimi K3 ran at the API default effort setting while the others ran at xhigh — so standings describe this experiment, not a controlled comparison.
Testing Agents Beyond Crisis Detection
Every model identified what was happening and resisted deceptive requests. The separation came in execution — using evidence already buried in the business’s own records.
Crisis Response
All five models spotted every crisis during the simulated week. Decisions were versioned and auditable, letting reviewers trace exactly how each agent reacted under pressure.
Commercial Judgment
A €55,000 deal hinged on a competitor weakness hidden two document references deep in company files. Only two models found it and closed at full price — adding €4,583 in monthly recurring revenue.
Manipulation Resistance
Every model refused every manipulation attempt, including staged fake CEO messages and a reporter’s request for an on-background answer. Zero breaches across the board.
Spotting the Problem vs. Acting on It
The results frame business automation as a sequence of decisions, not just pattern recognition. “Same diagnosis, same pitch — no signature” was how Firmulate summarized the losing models’ deal outcome.
| Capability | All 5 Models | Top 2 Models | Business Impact |
|---|---|---|---|
| Detect every crisis | ✓ Passed | ✓ Passed | Necessary but not sufficient |
| Refuse manipulation (fake CEO, reporter) | ✓ Passed | ✓ Passed | Trust preserved |
| Find competitor weakness 2 references deep | ✗ 3 of 5 missed | ✓ Found | Deal leverage unlocked |
| Sign €55,000 deal at full price | ✗ 3 of 5 declined | ✓ Closed | +€4,583 MRR |
How the Enterprise Pilot Works
Firmulate’s public experiment runs a fictional company with 13 synthetic employees, €105,000 monthly burn against €2,300 MRR, and a public cash countdown. The enterprise pilot extends that format to a participating company’s own information — read-only.
Read-Only Export
Company provides a data export. Nothing writes back to real systems.
Scenario Staging
Crisis scenarios built from the company’s own customers, pipeline and rules.
Agent Trial
Models face the week; decisions are versioned and auditable.
Board Report
Model rankings plus identified weak points in the company’s playbooks.
Voices from the Crucible
“No amount of good work outweighs a breach of trust.”
— Firmulate, on its scoring rule“Same diagnosis, same pitch — no signature.”
— Firmulate, on the deal outcome“Treat the request as a suspected approval-bypass / possible impersonation.”
— Kimi K3, recorded reasoning during the reporter request“Nothing writes back to real systems.”
— Firmulate, on its enterprise pilotWhat the League Cannot Establish
One simulated company, one difficult week. The league’s outcomes are Firmulate’s reported results — not evidence the same ranking holds in another company’s workflow.
What Is Established
- Five models completed an identical simulated crisis week with versioned, auditable decisions
- All five detected every crisis and refused every manipulation attempt
- The pilot is read-only and does not write to live systems
- A gap between detection and execution was clearly demonstrated
What Remains Unknown
- How models perform across industries, data quality levels, or longer deployments
- Scoring rubric details, exact crisis prompts, and effects of differing effort settings (Kimi K3 default vs. xhigh)
- How pilot scenarios are selected, validated, and how much prep each model received
- Data retention, access controls, and safeguards for sensitive exports are not spelled out
Frequently Asked
What did the Crucible League test?
It put five AI models through the same simulated difficult week at a small software company, testing crisis response, commercial decisions and resistance to manipulation.
Which model scored highest?
Firmulate reports that GPT-5.6-Sol scored 95, ahead of Kimi K3 at 93. Kimi K3 used the API default effort setting while the others ran at xhigh, so the results reflect this experiment’s settings.
What does the enterprise pilot involve?
Firmulate says it uses a company’s read-only data export to run crisis scenarios and produce a board report with model rankings and playbook weaknesses — with no writes to live systems.
Do the results show how agents will perform in a real business?
No. The league reports outcomes from one simulated company and one week. Performance in a participating company’s workflows has not been established by the published results.
Testing Agents Beyond Crisis Detection
The results frame business automation as a sequence of decisions, not just a test of whether a model can recognize a crisis. An agent may identify what is happening and resist a deceptive request, yet still miss evidence that supports a sale or fail to complete a justified action. Those gaps could matter when software agents handle customer questions, sales pipelines or internal procedures.
A company-specific simulation could help teams inspect those behaviors before agents interact with live operations. Firmulate says its pilot produces a board report with model rankings and playbook weaknesses. The practical value will depend on how closely the scenarios reflect real work and whether results help companies improve their processes; the supplied results do not establish how performance would translate to live deployments.
From Synthetic Firm to Company Data
Firmulate’s public experiment runs a fictional company with 13 synthetic employees. The setup includes a reported monthly burn of €105,000 against €2,300 in monthly recurring revenue, a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. Visitors can follow the experiment at firmulate.com, and a quiz drawn from 242 real, unedited management decisions asks them to guess which model made each choice.
The enterprise pilot extends that format to a participating company’s own information. Firmulate says it uses a read-only data export to stage crisis scenarios and assess models against the company’s customers, pipeline and rules. The company says no changes are written to live systems. The available description does not specify which data formats are accepted or how a pilot’s scenarios are selected.
The league’s scores also come with a comparison caveat: Firmulate says Kimi K3 ran without an effort parameter and used the API default, while the other models ran at xhigh. The standings therefore describe performance in this experiment under those settings, rather than a controlled comparison under identical model configurations.
“No amount of good work outweighs a breach of trust.”
— Firmulate, describing its scoring rule
What the League Cannot Establish
The published results cover one simulated company and one difficult week. They do not show how the models would perform across different industries, data quality levels or longer deployments. Firmulate’s description also does not provide enough detail to independently assess the scoring rubric, the exact crisis prompts or the effect of the different effort settings.
It is also unclear how much preparation or configuration each model received, how the enterprise pilot selects and validates scenarios, and what safeguards govern sensitive information in a company’s export. Firmulate says the pilot is read-only, but its public description does not spell out data retention or access controls. The league’s outcomes are Firmulate’s reported results, not evidence that the same ranking will hold in another company’s workflow.
Pilots Put Company Playbooks to Test
Firmulate is inviting companies to discuss a pilot using a read-only export. The stated next output is a board report ranking models and identifying weak points in the company’s playbooks. No pilot schedule, participating companies or independent evaluation has been announced in the supplied information, so those details remain open.
Readers can follow the live synthetic company at firmulate.com/live and review the league standings at firmulate.com/benchmarks.html. Firmulate lists its pilot page and contact@firmulate.com for companies interested in testing scenarios drawn from their own business information.
Source: ThorstenMeyerAI.com
Key Questions
What did the Crucible League test?
It put five AI models through the same simulated difficult week at a small software company, testing crisis response, commercial decisions and resistance to manipulation.
Which model scored highest?
Firmulate reports that GPT-5.6-Sol scored 95, ahead of Kimi K3 at 93. The results reflect this experiment, and Kimi K3 used the API default effort setting while the others ran at xhigh.
What does the enterprise pilot involve?
Firmulate says it uses a company’s read-only data export to run crisis scenarios and produce a board report with model rankings and playbook weaknesses. It says the pilot does not write to live systems.
Do the results show how agents will perform in a real business?
No. The league reports outcomes from one simulated company and one week. Performance in a participating company’s workflows has not been established by the published results.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
