How banks judge an AI engineering partner beyond the POC

To judge an AI engineering partner beyond the proof of concept, a bank scores each candidate on eight capabilities that decide whether a pilot can run in regulated production: a governed methodology, event-driven architecture, a production track record, domain-driven design, regulatory experience, a vendor-agnostic orchestration layer, human-in-the-loop governance, and measurable cost and delivery outcomes. The evidence to ask for under each one is set out below, with a comparison table for scoring a shortlist.

In production a system runs on real transaction volumes alongside legacy platforms, and it has to produce an evidence record for every decision. When an autonomous agent sits on the path of a payment flow or a credit decision, the bank evaluates the accuracy of the model together with the auditability and governance of the system around it. The partner question moves from "can we build this?" to "can we run this?".

Why partner selection changes after the proof of concept

During a pilot, teams optimise for rapid iteration and visible results in a contained setting. A partner who has shown a working pilot has shown that the model can do the task. The capabilities below test whether the same partner can build and run the system around the model for as long as the bank depends on it.

Claims about speed and savings need a baseline. The DX study of more than 400 companies reports a median 7.76% increase in pull request throughput as AI adoption rose, and says a 5-15% throughput gain is what the current generation of AI coding tools is delivering (DX, 2026). A partner's outcome figures are best read against numbers of that kind, and against baselines measured before the work started.

For UK banks, model risk expectations also reach the supply chain. PRA SS1/23 expects vendor models to be validated to the same standard as in-house ones, for banks, building societies and PRA-designated investment firms with internal model approval for regulatory capital. A partner who can show how their systems are tested and evidenced gives your validation function the material it needs. For agent-based systems more broadly, see AI agents and agentic systems.

What methodology should the partner work to?

A governed methodology puts AI inside the software lifecycle while human engineers keep responsibility for the architecture and the constraints it sets. Bugni's AI-native engineering page describes engineering pipelines that continuously test system behaviour, enforce policy and evaluate AI components.

Ask a candidate for the following:

• The points where humans review AI-generated output before it affects live systems.
• The architectural boundaries inside which AI agents operate.
• The same governance rules applied to every deployment, so that faster delivery keeps the same controls.
• A mapping of their approach to the NIST AI Risk Management Framework, which covers how to govern, map, measure and manage AI risk, and to ISO/IEC 42001:2023 for AI management systems.

Your own teams will shift as well. They review formal specifications and constraints and spend less time reading raw code.

Why does event-driven architecture matter beyond the pilot?

Event-driven architecture is a design pattern where decoupled services communicate by emitting and reacting to discrete state changes. It gives intelligent systems a way into live payment and screening flows without the bottlenecks of batch processing. Event-driven architecture also supports the evidence a bank needs.

Ask for evidence of these:

• Event sourcing that keeps an append-only record of each decision and of the data the model saw when it decided.
• Sanctions and fraud checks that run in parallel, so that compliance checks do not hold up the main transaction flow.
• A stated approach to eventual consistency, so that balances and credit limits stay accurate across services.
• Event streams that let you update a model and roll back to a previous version if anomalies appear.
• Latency measured at volume. One comparative study reports a p99 latency below 200 ms at 100,000 events per second, and notes that event-driven designs support stateful, event-by-event risk analysis (Parnerkar, 2025).

Distributed event-driven systems carry operational overhead, so ask which observability tooling lets your teams monitor the event streams.

How do you test a partner's production track record?

A pilot shows that a concept works for a month. Production longevity shows that an architecture holds up under real data and changing rules for years. Reference checks are the most direct test.

• Ask for specific systems that went live at least three years ago and remain in production, and speak to the engineers who inherited the code.
• Ask what upkeep those systems needed, and whether they absorbed new requirements without major re-architecture.
• Ask about systems that were decommissioned soon after launch, and what caused it.
• Ask how the platform will take on agentic workflows later without re-architecture.

Why does domain-driven design matter for AI systems?

Domain-driven design aligns the technical architecture with the business domains it serves. In banking that means services that mirror domains such as payments or credit decisioning. Domain-driven design supports legacy modernisation for banks adopting a phased approach (Deloitte, 2026), and its ideas are being applied to AI coding (threedots.tech, 2026).

Look for these practices:

• Bounded contexts, so that a sanctions screening service stays separate from an account management service and a fault in one does not cascade.
• A ubiquitous language carried through specifications and event schemas, so that AI agents work from the same terms as the business.
• Context mapping that lets modern services run alongside legacy mainframes during phased modernisation.
• Domain models that respect regulatory boundaries in the code.

Your business stakeholders will spend time with the partner up front to settle the domain language before code is written.

What regulatory experience should a partner show?

Ask the partner to name the instruments they have built to, and what each one required of the system.

Add three requests. Ask for event logs that show who or what made each decision, so the action cannot be denied later. Ask for the model validation and resilience documentation your own submissions will need. Ask whether compliance documentation is an automated output of the engineering pipeline.

Why should the orchestration layer be vendor-agnostic?

A vendor-agnostic orchestration layer separates your core systems from the specific models and screening providers you use today. The model market changes quickly, and the layer keeps your options open.

• Ask whether you can swap a sanctions screening provider or a language model through configuration, without re-architecture.
• Ask who owns the orchestration layer and the evidence layer, and whether you can run them without the partner.
• Ask about open standards. Anthropic donated the Model Context Protocol to the Agentic AI Foundation, a directed fund under the Linux Foundation, on 9 December 2025.
• Ask whether services talk to your internal orchestration layer and not directly to a vendor's API.

Building your own orchestration layer takes more upfront architectural work than buying a tightly coupled product, so ask for a costed comparison of the two routes.

How should human-in-the-loop governance work?

Autonomy in banking runs along a spectrum, from human review of every action, through semi-autonomous operation inside tight contracts, to broader autonomy where the evidence and reversal paths already support it. Human-in-the-loop governance sets explicit human oversight at defined risk thresholds. Moody's survey of 600 risk and compliance professionals found that 42% regard human oversight as mandatory (Moody's, November 2025), and PwC advises that banks embed human oversight at key AI decision points (PwC, 2025).

Ask the partner to show:

• Configurable thresholds that route a decision to a human reviewer, based on signals such as risk score, disagreement across repeated model samples, a failed verifier check, or transaction value. An LLM reports no calibrated confidence, so a threshold on a confidence score alone is weak.
• Escalation paths that pause the workflow, gather the relevant context and present it to the reviewer.
• Reviewer sign-off written into the audit trail, with the data the reviewer saw.
• Reviews that are real. Under UK GDPR Articles 22A to 22D, a rubber-stamp review does not count as human involvement in a significant automated decision.

Thresholds set too low flood reviewers and cancel the speed gain, so ask how the partner tunes them using historical data. A human-in-the-loop design guide covers the routing in detail.

How do you measure cost and delivery outcomes?

An engineering partnership has to make financial sense. Ask for a total cost of ownership (TCO) model that covers running costs as well as build costs, including ongoing compliance maintenance.

• Ask for documented baselines. A good partner measures your current deployment frequency and lead time for changes before writing code.
• Ask how they measure change failure rate alongside throughput, so that speed is read together with stability.
• Compare any throughput claim with published data such as the DX figures above.
• Ask for post-implementation metrics against the baseline, in place of qualitative accounts.

Bugni's published results include commercial customer onboarding reduced from 10 days to under 12 hours at a major UK bank (screening platform), and roughly 20 microservices taken from blank sheet to production in four months at a UK challenger bank (credit decisioning platform). More detail sits in the case studies.

How the eight capabilities compare

Use this table to score a shortlist. Each row names the evidence to request and what the capability supports.

Common pitfalls when judging a partner

Scoring on demonstration quality. A polished pilot shows the model working. The eight capabilities show whether the system can run.

Accepting outcome figures without baselines. A multiplier with no measured starting point cannot be checked. Ask for the baseline and the method.

Treating compliance documentation as a final step. An engineering pipeline that emits documentation as it runs keeps it in step with the system as built.

Coupling the orchestration layer to one provider. A tightly coupled layer ties model and screening choices to a single supplier. Ask how a swap would work before you need one.

Where to start

Send each shortlisted partner the questions in the sections above, and score the answers in the table. Then test the leading candidate on one workflow before committing to the wider programme.

Further reading

• AI agents and agentic systems
• AI-native engineering
• Event-driven architecture
• Domain-driven design
• Human-in-the-loop AI design guide
• Case studies