Running AI in production: the pilot is the easy part
An AI pilot proves little. Production starts where others build on the output. How we take AI systems into production, with six questions to ask before you go.
With AI systems, the pilot is the easy part. Production starts where other people build on the output or it leaves the company, and that's where you find out whether a system holds up. The usual success story goes: the pilot works, the team likes it. Then nobody asks who'll look after it a year from now, what happens when the model answers differently after an update, or when anyone decides on its future. That's how you get a pilot graveyard, a pile of projects that never make it into supported operation. At SUMAX we've been moving the whole company onto AI since 2023, together with the team, and we settled on an order early: the foundation for everyone first, then tools for individual tasks.
One good run proves little.
A good run tells you little about a new application. Data sources change, a model gives different results after an update, and the business task keeps moving. So whoever decides on the launch also decides on upkeep, review and fallbacks.
The threshold is where others build on the output or it leaves the company. From there on, an application needs output that can be checked, a fallback for when it fails, and a fixed date to decide whether to expand it, rework it or end it. Decide on production at the same moment you decide on the pilot.
Production starts where other people build on the output or it leaves the company.
We start with tasks where the team spots errors right away.
A narrow workflow teaches you the most. The input stays manageable, the output can be checked, and feedback turns into concrete changes. Along the way you see what data and what expert review a task needs.
Big projects with lots of people involved hide all that. When a broad system delivers poor results, it's hard to tell whether the data, the task or the way people use it is to blame. At SUMAX, process knowledge used to be scattered, and that knowledge belongs in one place before a tool builds on it. That's why we start with tasks a team knows well and whose errors it catches right away. For your first step into production, pick a workflow where your team sees a mistake at first glance.
Foundation first, then tools.
Our transformation started with the foundation for everyone: users, permissions and access. Through the permissions, security was part of it from day one. Tools for individual tasks came after that. Every AI call runs through our own layer between the applications and the models, which keeps the model swappable and books every call with its cost. As of this revision (October 2026), that's a five-figure number of calls a month, and by then we'd replaced several purchased tools with our own.
I led that foundation and built it with the team. Mostly it helps me judge what I can reasonably ask of teams and vendors, and where a promise is unrealistic. For leadership, something other than the architecture matters, though: if you build on a shared foundation, you depend on it. When a central service goes down, several workflows go down with it. Build users, permissions and access for everyone first, and plan for the foundation failing before many workflows rely on it.
Whoever passes on an AI result owns it.
Whether output is usable is for the team that works with it to decide. Pass an AI result on unchecked and you become a pass-through, and the mistake lands on the next person in line. For an application to become a real working tool, you need sign-offs people can trace, documentation a new colleague can follow and a set channel for feedback.
At SUMAX the tools get built with the people who use them. We rewrote the role descriptions, and training uses real tasks from each team's own area. As of this revision, the large majority of our people use our own AI tools every month. As COO, I free up the time for this. AI saves routine work, but checking takes time, and if you don't plan for it you get unchecked output or overloaded teams. Put review time in the plan before the application goes into daily use.
The handoff shows whether output holds up.
The next person has to see what's already been checked, which sources are behind it and which decision is still open. A clear handoff takes care of that, with or without an extra application.
An example: research produces a list of possible measures. For the colleague who's supposed to turn it into a plan, the list is only useful once sources, uncertainties and priorities are visible. Without that, she starts the research over and the time saved is gone.
Only pass on an AI result with its sources, uncertainties and open decisions visible.
Write decisions down before the next model ships.
Why was a particular solution chosen? Which alternatives were on the table, and which constraint applied at the time? A list of existing features won't give you that reasoning. A useful decision log has the recurring requirements, the rejected alternatives and the expected limits of the solution.
When a new model or an off-the-shelf product comes along, the log shows which assumptions still apply and whether switching is worth it. An earlier decision then gets measured against the conditions it was made under. An application that runs every day still needs that critical look, and one that hardly anyone uses anymore should be switched off, even if it once took a lot of work. Write the reasoning down while you still remember it.
Six questions before you go to production.
The first question decides whether the rest apply.
- Does anyone outside the pilot team build on the output, or does it leave the company?
- Can the output be checked, and does the reviewer know how to spot an error?
- Is there a fallback if the model, the data source or the shared foundation goes down?
- Is there a date to decide whether to expand, rework or end it?
- Is it written down why this solution was chosen and which alternatives were dropped?
- Can the next person in line see what's been checked and what's still open?
A no on question 1 means it's still a trial. A no on any of the others means it's too early for production.
A trial is allowed to be messy.
For short trials that disappear after a week, the full effort would be overkill. Put a log, a fallback and a review date on every experiment and you smother the team's curiosity. The rules kick in the moment others build on a result or it leaves the company.
I decide on production together with the pilot, not after it has proven itself. Take the pilot your team praises most right now and run it through the six questions.