AI Operating Model: CoE vs Federated vs Platform Team
CoE, federated or platform team? How to choose an AI operating model by maturity, with decision rights, funding flows and EU AI Act Article 4 duties.
On 2 February 2025, Article 4 of the EU AI Act made every organisation that deploys AI responsible for its staff's AI literacy. On 27 July 2026 the Digital Omnibus, Regulation (EU) 2026/1744, softened the duty to "take measures to support" literacy rather than ensure it, but kept it. National authorities have supervised it since 2 August 2026. A duty to build competence across a workforce is an organisational design question in disguise: who teaches, who decides what is enough, who pays?
The numbers ask the same question. McKinsey's November 2025 State of AI survey found that 88% of organisations use AI in at least one function, but only 39% report any enterprise-level EBIT impact. That gap is rarely about models. It is about how the organisation decides, funds and runs AI. This article is for the people who have to settle that: CIOs, CDOs, CTOs and the heads of their AI teams. We compare the three operating models every enterprise chooses between, show which fits at which stage of maturity, and argue that the real answer is a sequence, not a choice.
TL;DR / Key takeaways
- An operating model is a set of decision rights, not an org chart. Write down who sets priorities, picks models, signs off risk, builds, runs and pays. Those six answers are your operating model.
- Start central, then hand delivery over. In our experience a Centre of Excellence wins for the first 12–18 months and becomes a queue after that. Microsoft's own adoption framework says the CoE should move from control to an advisory role.
- Platform teams scale, but only with real demand. Shared gateways, evaluation and guardrails pay off once several business units are building. Before that you are building a platform nobody uses.
- Federated needs AI-literate business units. Without them, ten teams buy ten tools and nobody sees the combined risk.
- The governance spine is the same in every model. You need an AI inventory, Article 4 literacy measures, named oversight owners, the works council in Germany and one accountable executive. A missing spine is a compliance gap, whichever topology you pick.
Download: Editable draw.io file · Open in diagrams.net · Chart data (CSV)
Why this decision is on every CIO's desk now
Pilots stopped being enough. McKinsey describes most organisations as still experimenting or piloting. About one-third have begun to scale AI across the enterprise, and about 6% are "high performers" that attribute more than 5% of EBIT to AI. BCG's October 2024 study of 1,000 executives in 59 countries found that 74% of companies had yet to show tangible value from AI. Its "70-20-10" principle puts 70% of the effort into people and processes and only 10% into algorithms. These are operating-model problems.
The work moved from chat to agents. One team can roll out a chat assistant. An agent that reads your CRM and calls internal APIs for a user needs identity, tool governance, evaluation and an owner in the business unit whose process it changes. That cross-cutting capability is why the enterprise agent platform became a reference-architecture topic in 2026. Someone has to own it.
Regulation turned literacy and oversight into duties. Article 4 asks for literacy measures. Article 26 asks deployers of high-risk systems to give human oversight to people with the "necessary competence, training and authority". Annex III obligations now apply from 2 December 2027, so Article 26 can wait for most use cases. Article 4 cannot. Both need a named owner somewhere in your operating model.
The three models, and what actually differs
We compare the models by asking who holds each decision. The numbers match the callouts in the hero diagram.
① The centralised Centre of Excellence
One central team holds the AI talent, sets strategy, and builds and runs the use cases. Business units bring requests and a steering board prioritises them.
Where it wins. Early on, talent is scarce and standards do not exist. Ten good people in one team get the first use cases live faster than ten people spread across ten units, and you have one place to answer the auditor and the works council. Microsoft's Cloud Adoption Framework says companies at an early stage "benefit from a centralized CoE to consolidate expertise and foundational practices". It also advises against a standalone AI team where an existing Cloud Centre of Excellence can absorb AI.
Where it breaks. The CoE becomes the queue. Every idea waits for the same people, who know the models but not the process. Business units work around it, and shadow AI grows faster than the backlog shrinks.
② The federated model
Each business unit builds or hires its own AI team, funds it and owns its use cases end to end. The centre keeps a thin community of practice and minimum standards.
Where it wins. In organisations with strong, AI-literate business units, it puts the people who understand the process next to the people who build. Priorities are set where the value is, and nobody waits.
Where it breaks. Federated without a platform means duplication. Each unit picks its own model provider, vector store and guardrails, and signs its own contracts. Risk becomes invisible. Nobody can say how many AI systems you run or which touch personal data, and building the inventory for the AI Act or ISO/IEC 42001 turns into archaeology.
③ The platform team
A central team builds shared AI capabilities as an internal product: model gateway, approved model catalogue, evaluation harness, guardrails, observability and agent identity. Business units build their use cases on top. In Team Topologies terms, from Matthew Skelton and Manuel Pais's 2019 book, this is a platform team serving stream-aligned teams "as a service".
Where it wins. Once several units are building, every project rebuilds the same plumbing: model authentication, prompt logging, PII redaction and cost attribution. A platform builds it once and makes the safe path the easy one. Governance moves from approval meetings into code. We described the same shift for software delivery in our platform engineering guide. AI is a new tenant on the same idea.
Where it breaks. It breaks when the platform is built before the demand exists. We regularly meet AI platform teams that spent a year on a gateway and a portal and serve two internal customers. A platform is a product, and without customers it is a cost centre with a roadmap.
④ Decision rights: six questions that define your model
Strip away the org chart and six questions remain. The diagram shows the default answers per model.
- Priorities. Who decides which use cases are funded, and in what order?
- Models and vendors. Who decides which models, providers and tools are allowed?
- Risk sign-off. Who accepts the risk of going live, and on what evidence?
- Build. Who designs and builds?
- Run. Who operates the system and gets paged when it fails?
- Budget. Who pays for build, run and inference?
The most common failure we see is a mismatch between these answers: a business unit pays, the CoE approves the model and the platform runs it, yet nobody owns the outcome. Put the six answers in a one-page charter. Wherever two teams claim the same decision, you have found your next escalation.
⑤ Funding flows: who pays shapes who decides
Funding overrides every other decision right. There are three patterns:
- Central cost centre fits the CoE. It is quick to start, but demand is free to the business, so the backlog never shrinks and value is never tested.
- Business-unit P&L fits federated. Ownership is clear, but shared capabilities never get funded, because no single unit pays for what everyone uses.
- Seed plus chargeback fits the platform team. The centre funds year one. After that, usage, mostly inference tokens, is shown back or charged back to each unit. Inference is metered per call, so AI is unusually easy to allocate. The mechanics match cloud costs, as in our chargeback and showback model.
Our recommendation is to fund the platform centrally and the use cases from business budgets. Show inference costs back from the first month. Teams change behaviour when they see their own number.
⑥ The governance spine every model needs
Whatever topology you choose, five things stay central:
- An AI inventory and risk register that records each system's purpose, data, risk class and owner. Our AI system inventory guide has a template.
- AI literacy measures in the form of a documented, role-based programme you can show under Article 4.
- Named human-oversight owners, especially for future Annex III systems.
- Works council involvement through a standing forum, in Germany and other co-determination countries.
- One accountable executive who owns the portfolio, arbitrates budget and holds risk acceptance.
ISO/IEC 42001 clause 5.3 asks top management to assign and communicate responsibilities and authorities for relevant roles, and these are the roles that matter. We map the overlap in our EU AI Act vs ISO 42001 crosswalk.
What the numbers say
The heat grid is our analysis, not survey data. It reflects what we see in engagements and agrees with Microsoft's adoption guidance and Team Topologies. Read it column by column. At stage 1, only the centralised CoE fits well. At stage 3, with several business units building, the platform team and the hybrid take over and the pure CoE drops to "weak". At stage 4 the hybrid scores best, federated and platform team are good, and the pure CoE is poor. Federated improves with maturity but never beats the hybrid, because it leaves shared capabilities unfunded. No single model is right for the whole journey, and the expensive mistake is moving too late.
The second chart uses two self-reported surveys, one named source per bar, so read it as directional. Adoption is near universal; value at enterprise level reaches roughly a third, and only about 6% count as high performers. That distance is the operating-model gap: the technology is there, but the owner for workflow redesign, the funding for shared foundations and a risk process that does not stall everything are not.
In a recent engagement…
Earlier this year we worked with a European industrial group that had built a capable central AI team two years before. It had done what a CoE should: it shipped the first use cases, built a model gateway and wrote the AI policy. By the time we arrived it had become the problem. Every request went through one intake board, and the backlog was measured in quarters. Two divisions had quietly signed their own AI vendor contracts, and nobody centrally knew what data those tools could reach.
The team wanted more headcount. We argued against it and split the CoE in two. The engineers who built the gateway became a platform team with a product owner, a published model catalogue and self-service onboarding. The architects and the governance lead became a small advisory group that owns policy, inventory, literacy and risk acceptance, and no longer approves projects that stay inside the guardrails. The two rogue divisions came onto the platform because it was now cheaper and faster than their workaround, not because the workaround was banned.
The hard part was moving the budget. Platform run costs stayed central for a year, while inference costs were shown back to each division from month one, which changed the tone of prioritisation quickly. The works council joined from the first workshop and co-designed the literacy programme.
The decision table
Find your row, then read across.
| Signal | Centralised CoE | Federated | Platform team | Hybrid (platform + thin CoE + embedded) |
|---|---|---|---|---|
| Use cases in production | 0 to a handful | Many, independent domains | Several, on shared components | Many, across units |
| Business-unit AI literacy | Low | High | Medium | Medium to high |
| Shared components rebuilt per project | Not yet | Accepted cost | The trigger | Solved by the platform |
| Funding | Central | Unit P&L | Seed plus chargeback | Platform central, use cases from units |
| Main failure mode | Queue | Duplication, invisible risk | Platform without demand | Blurred platform/CoE boundary |
| Move on when | Backlog outgrows delivery | Duplicated spend or a risk incident | Units want more autonomy | Rarely; tune the boundaries |
And the decision rights as a charter, for the hybrid most of our clients land on:
| Decision | Accountable | Consulted | Evidence |
|---|---|---|---|
| Use-case priority | Business-unit owner | Platform, AI governance lead | Business case, named owner |
| Allowed models and vendors | Platform product owner | Security, procurement, legal | Evaluations, contract terms, data residency |
| Risk sign-off | Business-unit owner, within central policy | AI governance lead, DPO | Risk class, evaluation report, oversight owner |
| Policy exceptions | Accountable executive | AI governance lead, CISO | Written risk acceptance with expiry |
| Build and run | Unit builds; platform runs the runtime | Platform patterns | Paved road, runbook, alert routing |
| Literacy programme | AI governance lead | HR, works council | Role-based curriculum, completion records |
Governance for European enterprises
Only a few regimes genuinely shape the operating model.
EU AI Act, Article 4. Since Regulation (EU) 2026/1744 took effect on 27 July 2026, providers and deployers must "take measures to support the development of AI literacy" of their staff and others operating AI on their behalf, without guaranteeing any individual's level. The Commission's Q&A adds that no certificate is needed and that national market surveillance authorities supervise from 2 August 2026. A role-based programme with internal records is enough, but someone must own it.
EU AI Act, Articles 26 and 50. Deployers of high-risk systems must assign human oversight to competent, trained people with authority. For Annex III systems that applies from 2 December 2027, so name the roles in your charter now. Article 50 transparency duties have applied since 2 August 2026, with a four-month transition for marking the output of generative systems already on the market. They need an owner in every team that ships user-facing AI. In a federated model, that means every business unit.
German Works Constitution Act (BetrVG). Section 90(1) no. 3 requires timely information about planned work procedures "including the use of artificial intelligence". Under section 80(3), an expert the works council calls in to assess AI is deemed necessary. Section 87(1) no. 6 gives co-determination over technical systems that can monitor behaviour or performance, which covers many AI tools. Treat the works council as a late gate and you will stall. Give it a standing seat and you will not.
ISO/IEC 42001 and DORA. ISO/IEC 42001 is not a law, but clause 5.3 in effect asks you to document who is responsible for what. Under DORA Article 5, a financial entity's management body is responsible for ICT risk. AI is ICT, so in banks and insurers AI accountability cannot sit outside ICT risk governance.
A pragmatic adoption path
- Weeks 1–2: map decisions, not boxes. Answer the six decision-rights questions as things work today. List every conflict and gap. Start the AI inventory at the same time.
- Weeks 3–6: name the spine. Appoint the accountable executive and the AI governance lead. Agree a standing forum with the works council. Launch role-based literacy with records that hold up under Article 4.
- Months 2–3: pick the model for your stage, plus the trigger for the next one. Use the decision table. If you choose a CoE, write down the signals that will move delivery out of it. If you choose a platform, name its first three customers before you build.
- Months 3–6: move funding to match. Fund the platform centrally and use cases from business budgets. Show inference costs back from day one. Make the platform the cheapest route to production, so shadow tools lose on convenience, not prohibition.
- Every six months: review the boundaries. Re-score yourself on the maturity grid. Move decision rights outward as units mature, and inward when an incident exposes a gap.
The verdict
CoE versus federated versus platform team is usually framed as a choice. In practice it is a sequence. Start central, because early on you need concentrated talent and one voice. Build a platform once several units are building, not before. Push delivery and priorities out to the business as fast as its AI literacy allows. Keep a thin, firm governance spine in the centre that never moves.
Most enterprises we meet are one stage behind: the CoE has become a queue, or federated units duplicate spend nobody sees. The fix is rarely more headcount. It is redrawing six decision rights and moving the budget to match. Article 4 gives you a reason to start that conversation at your next leadership meeting.
If you want an outside view on where you stand and which move comes next, our AI advisory team runs this decision-rights assessment as a short, fixed-scope engagement. You leave with the charter filled in for your own organisation.
FAQ
What is an AI operating model?
It is the set of answers to six questions: who sets AI priorities, who chooses models and vendors, who signs off risk, who builds, who runs and who pays. The three common shapes are a centralised Centre of Excellence, a federated model where business units own delivery, and a platform team that offers shared AI capabilities as a service. Most mature enterprises run a hybrid of the last two with a thin central governance function.
Should we start with an AI Centre of Excellence?
Usually yes, if you have few use cases in production and scarce AI talent. A central team concentrates expertise and sets the first standards. Microsoft's Cloud Adoption Framework gives the same advice and suggests folding AI into an existing Cloud Centre of Excellence where one exists. Plan the hand-over from day one, because a CoE that keeps building everything becomes the bottleneck.
When should a CoE evolve into a platform team or a federated model?
Watch for three signals. The CoE backlog grows faster than the team delivers. The same components, such as gateways, evaluation and guardrails, are rebuilt for each project. Business units hire their own AI engineers anyway. When you see these, move shared capabilities into a platform team and let business units own delivery. Shrink the CoE to an advisory group that sets guardrails instead of approving projects.
What does EU AI Act Article 4 on AI literacy require now?
Article 4 has applied since 2 February 2025. Regulation (EU) 2026/1744, in force since 27 July 2026, rewrote it. Providers and deployers must take measures to support the AI literacy of staff and others operating AI on their behalf, considering their knowledge and the context of use. No individual level must be guaranteed and no certificate is needed. The Commission says national supervision applies from 2 August 2026, so keep records.
Who should own AI: the CIO, the CDO or a Chief AI Officer?
The title matters less than the decision rights. One executive must be accountable for the AI portfolio, arbitrate the budget and own risk acceptance. That is often the CIO or CDO, where data and platforms already sit. A separate Chief AI Officer helps only if the role comes with real budget and authority. Otherwise it adds coordination without changing who decides.
Do we need the works council for an AI operating model in Germany?
Very likely. Under the Works Constitution Act, the employer must inform the works council in good time about planned work procedures, including the use of AI (section 90). An expert the works council calls in to assess AI is deemed necessary (section 80(3)). Systems that can monitor behaviour or performance are subject to co-determination (section 87(1) no. 6). Give the works council a standing seat, not a late approval step.
Sources
All sources checked on 15 November 2026. Survey figures are self-reported by respondents. The maturity-fit grid is our own analysis.
Tier A: primary and regulatory
- Regulation (EU) 2026/1744 (Digital Omnibus on AI), amending Article 4 of the AI Act: https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng
- Regulation (EU) 2024/1689 (AI Act), Articles 4, 26, 50 and 113: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
- European Commission, AI literacy: questions and answers: https://digital-strategy.ec.europa.eu/en/faqs/ai-literacy-questions-answers
- Betriebsverfassungsgesetz § 80 (expert for AI): https://www.gesetze-im-internet.de/betrvg/__80.html
- Betriebsverfassungsgesetz § 87 (co-determination): https://www.gesetze-im-internet.de/betrvg/__87.html
- Betriebsverfassungsgesetz § 90 (information on work procedures including AI): https://www.gesetze-im-internet.de/betrvg/__90.html
- Regulation (EU) 2022/2554 (DORA), Article 5: https://eur-lex.europa.eu/eli/reg/2022/2554/oj/eng
- ISO/IEC 42001:2023, AI management systems: https://www.iso.org/standard/42001
Tier B: research and analysis
- McKinsey, "The state of AI in 2025: Agents, innovation, and transformation" (November 2025): https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
- BCG, "Where's the Value in AI?" press release (24 October 2024): https://www.bcg.com/press/24october2024-ai-adoption-in-2024-74-of-companies-struggle-to-achieve-and-scale-value
- Team Topologies, key concepts (team types and interaction modes): https://teamtopologies.com/key-concepts
Tier C: vendor guidance, directional
- Microsoft Cloud Adoption Framework, "Establish an AI Center of Excellence" (updated October 2026): https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ai/center-of-excellence
Topics