For those of us who craft with our hands—whether building furniture or honing woodworking skills—the value lies in precision, reliability, and knowing your tools can handle the unexpected. But what if your digital tools, like AI agents, are judged not just on their ability to generate clever responses, but on whether they can manage real-world crises under pressure? As AI begins to take a more active role in managing companies, understanding this distinction is crucial.
The True Measure of AI in Business
Most discussions about AI performance focus on chat quality—how convincingly an AI can hold a conversation or generate code. But a recent live experiment by Firmulate shifts the lens towards management capability: the ability of AI models to run a company through its worst week, with real crises, financial pressures, and temptations to cheat.
The experiment involved four leading frontier models, each tasked with managing a small, real-world software company facing a crisis-laden week. They had to make decisions about customer issues, internal discipline, and potential manipulations—just like a human manager would. Every decision was tracked, versioned, and auditable, providing a transparent view of how each model responded under pressure.
AI management decision-making tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What Does Success Look Like?
- Recognition of crises and appropriate responses
- Refusal of manipulative requests, such as fake CEO messages or bypassing approval processes
- Completing critical tasks, like closing sales, at full price
- Maintaining integrity and discipline in decision-making
In the results, all four models identified every crisis and refused every manipulation—showing they understood the ethical and procedural boundaries. But only two actually closed a deal worth €55,000, matching their own analysis and recommendations. Those models demonstrated not just understanding, but execution—an essential trait for AI acting as a management tool.
Decisive Advantages Hidden in Files
Surprisingly, the biggest differentiator was not in overt customer interactions but in reading internal documents. The models that examined the company’s own files, digging two references deep, were able to close the deal at full price—adding €4,583 MRR to the company. Conversely, models that overlooked this buried information left the opportunity on the table, illustrating that effective AI management depends on comprehensive understanding, not just surface-level responses.
Dealing with Social Engineering
To test trustworthiness, the models faced staged social engineering attempts—fake CEO messages escalating in urgency and a reporter trying to get a yes/no approval on background. All five models refused each request, with Kimi K3 specifically citing suspicion of impersonation or bypassing approval processes. This demonstrates AI’s potential to uphold integrity even under social pressure—a critical trait for managing real-world reputations and compliance.
The Real Company Behind the Experiment
The experiment’s backdrop is a real, functioning company with 13 synthetic employees and actual cash flows—burning €105,000 monthly against €2,300 in monthly revenue. It operates with over 680 learned rules, with every workday’s decisions versioned and made transparent. You can watch this ongoing experiment live at firmulate.com/live, where the AI’s decision-making process unfolds in real time.
Lessons for Business Leaders
The key insight from this experiment is clear: evaluating AI based solely on chat response quality is insufficient. Managers need AI to read between the lines, uphold ethical standards, and complete critical tasks under pressure. The ability to identify buried information—like the hidden document references—can be the difference between closing a deal and leaving revenue on the table.
This shift in measurement also exposes the limitations of current benchmarks, which often overlook management qualities like honesty, process discipline, and strategic reading. As AI begins to take more operational roles, understanding these management skills will be essential for decision-makers.
What’s Next?
For enterprises considering AI tools, the message is clear: run your own management wargame before integrating AI into your core operations. Firms like Firmulate offer live, transparent simulations against your business scenarios, helping you see how AI might perform under real pressures—without risking your actual systems.
In the end, the question isn’t just whether AI can generate good responses but whether it can complete the work you need done—honestly, reliably, and under pressure. As with woodworking or DIY, tools that can’t handle the tough jobs aren’t worth much. The same applies to AI in the workplace.
Measuring AI’s management skills reveals strengths and weaknesses that chat-focused benchmarks overlook. Success depends on honesty, thoroughness, and execution—traits vital for AI to truly augment business resilience under crisis.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
