A municipality wants to give its staff a writing assistant and its residents a chatbot. The supplier builds both on a large language model that, according to the offer, is a general-purpose AI model with systemic risk, the heaviest category in the AI Act. What does that mean for the municipality? And what should it make of the AI Office report of 21 October 2026 on the requirements for external evaluators who test such models?
The answer starts with Article 55, which gives the provider of such a model four additional obligations. The examples in this piece are fictional.
In short
Article 55 applies only to providers of general-purpose AI models with systemic risk. They evaluate the model, including adversarial testing (paragraph 1, point (a)), assess and mitigate systemic risks (point (b)), report serious incidents to the AI Office (point (c)) and ensure cybersecurity protection (point (d)). The Article has applied since 2 August 2025, the Commission has been able to impose fines since 2 August 2026, and for models placed on the market before 2 August 2025 the deadline is 2 August 2027.
The text of Article 55 does not mention external evaluators. Recital 114 refers to independent external testing "as appropriate"; the commitment to use it is in the code of practice. The report of 21 October 2026 sets out the current views of the AI Office: not guidelines under Article 96, not a decision and, in our reading, not binding.
Other posts cover designation under Article 52, Commission supervision under Articles 90 to 94, the code of practice itself, the scientific panel and contract clauses in procurement. This piece does not repeat them.
When does a model have systemic risk?
A general-purpose AI model has systemic risk if it has high impact capabilities, or if the Commission designates it as such (Article 51(1)). High impact capabilities are presumed when the cumulative amount of computation used for training is greater than 10^25 floating point operations (paragraph 2). The Commission ensures that a list of such models is published and kept up to date (Article 52(6)).
Article 55 paragraph by paragraph
Paragraph 1 applies "in addition to the obligations listed in Articles 53 and 54": the obligations of Article 53, such as technical documentation and information for those who integrate the model into their own system, continue to apply. On top of that come four obligations.
Point (a): model evaluation. The provider performs model evaluation "in accordance with standardised protocols and tools reflecting the state of the art", including conducting and documenting adversarial testing of the model, with a view to identifying and mitigating systemic risks.
Point (b): risk assessment and mitigation. The provider assesses and mitigates possible systemic risks at Union level, including their sources, that may stem from the development, the placing on the market, or the use of the model.
Point (c): serious incidents. The provider keeps track of, documents and reports, without undue delay, relevant information about serious incidents and possible corrective measures to the AI Office and, as appropriate, to national competent authorities.
Point (d): cybersecurity. An adequate level of cybersecurity protection for the model and its physical infrastructure; recital 115 mentions, among other things, accidental model leakage and model theft.
Paragraph 2 deals with evidence. The provider may rely on a code of practice within the meaning of Article 56 until a harmonised standard is published; such a standard grants a presumption of conformity to the extent that it covers the obligations. A provider who does not adhere to an approved code of practice or comply with a harmonised standard demonstrates "alternative adequate means of compliance for assessment by the Commission".
Paragraph 3: information obtained, including trade secrets, is subject to the confidentiality obligations of Article 78.
Since when? Chapter V, which contains Article 55, has applied since 2 August 2025 (Article 113, third paragraph, point (b)). Fines under Article 101, up to 3 percent of annual worldwide turnover or EUR 15 million, whichever is higher, are excepted from that date; the Commission has been able to impose them since 2 August 2026. Providers of models placed on the market before 2 August 2025 must take the necessary steps to comply with the obligations by 2 August 2027 (Article 111(3)). The Digital Omnibus on AI, Regulation (EU) 2026/1744, did not amend Article 55.
Does the law mention external evaluators?
Not in Article 55 itself. Paragraph 1 requires evaluation, testing and documentation, and does not say by whom.
External testing does appear in two other places in the Regulation. Recital 114 says providers should perform the necessary model evaluations, including adversarial testing, "also, as appropriate, through internal or independent external testing". And Annex XI, Section 2, point 2, asks providers of models with systemic risk to include in the technical documentation, "where applicable", a detailed description of the measures put in place for internal and/or external adversarial testing. "As appropriate" means where suitable, not optional. A recital is not itself an obligation, but it informs how point (a) is read; the Annex asks for a description, not for an external test.
The firm commitment comes from the General-Purpose AI Code of Practice, Safety and Security Chapter. Under Measure 3.2, signatories evaluate the model as specified in Appendix 3 of that chapter. Appendix 3.5 says that, in addition to internal evaluations, signatories will ensure that adequately qualified independent external evaluators conduct model evaluations. The exceptions are a model that is similarly safe or safer than a reference model, and the case where the signatory fails to appoint a suitable evaluator despite early search efforts, such as a public call open for 20 business days; the signatory then takes the additional uncertainty into account when determining whether the risk is acceptable, and justifies the exception in the Model Report (Measure 7.4). Measure 3.5 also asks signatories to give an adequate number of independent external evaluators adequate free access to the model after it is placed on the market, unless the model is similarly safe or safer than a reference model.
The report links external evaluators to "Article 55(1)(b)" and to the code of practice. That is defensible: the code places model evaluations within systemic risk analysis, and Commitment 3 cites Article 55(1) and recital 114 as its legal text. In our reading, the wording of recital 114 and Annex XI is closer to point (a), model evaluation with adversarial testing, than to point (b). The report offers an interpretation here, not the text of the law.
What about a provider that does not sign or adhere to the Safety and Security Chapter of the code? It demonstrates alternative adequate means of compliance (paragraph 2). The Commission guidelines on general-purpose AI models (C(2025) 7719 final, 19 November 2025) expect it to report its measures to the AI Office and explain how they ensure compliance, for instance through a gap analysis against a code of practice assessed as adequate, and say it may face more requests for information and for access to conduct model evaluations (point 92). In our reading, this creates no legal obligation to use external evaluators, but a provider without external evaluation must be able to explain why its evaluation still meets the state of the art, measured against the code.
What the AI Office report says
The AI Office held an online workshop with experts from 19 organisations on 15 July 2026 and published the report on 21 October 2026. The report itself says it expresses the current views of the AI Office, which may change. It is not guidelines under Article 96 or a decision and, in our reading, binds no one.
Scoring per risk domain. Under the proposal in the report, the AI Office scores evaluators on qualification and independence. Most criteria are public; the scoring rubrics and point scales are not, to prevent manipulation. Scoring is per systemic risk domain: an evaluator with depth in, say, loss of control need not have it in every domain, and scores do not add up across domains. For highly capable models the AI Office expects highly scoring evaluators; for less capable models slightly lower-scoring evaluators could be acceptable, especially when highly qualified ones are not available.
No fixed cycle. Evaluators are not scored periodically but when needed for supervision, for example when assessing whether a provider has engaged adequately qualified independent evaluators. Where doubts arise, the AI Office may ask the provider for additional evidence.
Qualification. The requirements follow Appendix 3.5 of the code: significant domain expertise for the systemic risk, technical skill and experience in model evaluations, appropriate information security protocols, and agreement to protect commercially confidential information. The report details the first two, including published risk models and evaluation methods, red-teaming and reproducibility. Evaluating open-weight models, as well as closed ones, counts as qualified experience. The criteria apply to the organisation and, where it has no established track record, to the individuals carrying out the evaluation.
Financial independence. The evaluator and its evaluation team members hold no financial interest in the provider and no shares, other than indirectly through a broadly diversified collective investment scheme. Compensation may be at market rates but not performance-based. The evaluator provides no paid advisory or implementation services to the provider related to tax, valuation, human resources or intellectual property.
Management independence. No organisational affiliation with the provider, such as ownership or being a subsidiary, and team members are not employed by the provider or engaged as its personal consultants. The evaluator has a conflict of interest policy, shared with the provider, that also discloses financial dependence.
Operational independence. The report refers to the definition of "independent external" in the glossary of the code and to Measure 3.5: no legal or technical retaliation against evaluators for their testing or publication, as long as they meet five conditions. One is the provider's public procedure for responsible vulnerability disclosure, under which the provider cannot delay or block publication for more than 30 business days from the date it is made aware of the findings, unless a longer timeline is exceptionally necessary, such as where disclosure would materially increase the systemic risk. After the workshop, in bilateral communication with some providers, the AI Office said that the independence of evaluators engaged under Measure 3.2 or 3.5 should be ensured through contractual safeguards. Those safeguards should be state of the art and include at least all the requirements and recommendations of Section 4, parts 3 (Analytic Autonomy) and 4 (Transparent Methods and Results), of AEF-1, a voluntary standard of a private forum to which the report refers. According to the report, the operational independence criteria are compatible with that guidance.
What the report does not cover. It does not concern the requirements for the scientific panel (Article 68(2)) or for the independent experts the Commission itself may appoint to carry out evaluations (Article 92(2)).
What does this mean for a municipality?
The supplier that integrates the model into its chatbot is the provider of the AI system; the municipality is the deployer. If the municipality builds the chatbot itself on a model it accesses through an API and puts it into service under its own name, it is both provider and deployer of that system (Article 3, points (3) and (4)), but still not the provider of the model.
What can it ask in procurement? In our reading, four things. Which model, and is it a model with systemic risk? Does the model provider follow the Safety and Security Chapter of the code (the Commission publishes the signatories)? Have independent external evaluators evaluated the model, and are their reports public? If not, which exception applied? Under the code, evaluators keep control over publication of their findings (Measure 7.4). And what information did the supplier receive from the model provider under Article 53(1)(b)? Record "we do not know" as well; unknown is not the same as fine.
The municipality's own obligations lie elsewhere. It takes measures to support the AI literacy of its staff (Article 4). The chatbot's provider, the supplier or, if it builds the chatbot itself, the municipality, ensures that residents are informed that they are interacting with an AI system, unless this is obvious (Article 50(1)), and if the municipality publishes AI-generated text to inform the public on matters of public interest, it discloses that, unless the text has undergone human review or editorial control and someone holds editorial responsibility (Article 50(4)). Article 50 has applied since 2 August 2026; only for the machine-readable marking of paragraph 2 does a system already on the market before that date have until 2 December 2026 (Article 111(4)). More in disclosing a chatbot as AI.
A chatbot or writing assistant is not in itself a high-risk system. If the municipality uses the system for a task in Annex III, for example to evaluate whether someone is eligible for benefits (point 5(a)), it is in principle high-risk, unless the derogation in Article 6(3) applies. From 2 December 2027 the obligations of Article 26 then apply, including registration in the EU database (Article 26(8) and Article 49(3)), and as a body governed by public law the municipality carries out a fundamental rights impact assessment beforehand (Article 27). For a system already in use before that date, those obligations apply only after a significant change in its design, and by 2 August 2030 at the latest (Article 111(2)).
If the chatbot processes residents' personal data, the GDPR applies, and where the processing is likely to result in a high risk the municipality carries out a data protection impact assessment beforehand (Article 35 GDPR; see when a DPIA is required for AI). Measure 3.5 of the code also addresses user data: external evaluators keep their protection against retaliation only if they do not intentionally access, modify or use sensitive or confidential user data in violation of Union law, and if they do access such data, they collect only what is necessary, refrain from disseminating it and delete it as soon as legally feasible. That is not an obligation of the municipality.
And a bank or insurer?
A bank uses a model with systemic risk to summarise customer files and draft letters; that is not in itself a high-risk use. The bank can ask the same procurement questions as the municipality. If the bank uses the model in a system that evaluates the creditworthiness of natural persons, or an insurer in a system for risk assessment and pricing in life or health insurance, that system is in principle high-risk (Annex III, point 5(b) and (c)), with obligations from 2 December 2027; for a system already in use before that date, only after a significant change in its design (Article 111(2)). The bank or insurer is then a deployer under Article 26 and carries out a fundamental rights impact assessment before putting the system into use (Article 27(1)); if it builds the system itself and puts it into service under its own name, it is both provider and deployer. That the underlying model was tested externally does not, in our reading, make those obligations lighter.
Want to know which AI Act and GDPR obligations come into view for your use case? With the case check you describe it in a few questions.
Frequently asked questions about Article 55 and external evaluators
Sources
Newsletter
Every Tuesday, the AI Act week ahead in 5 minutes
A practical briefing on deadlines, new guidance and enforcement, so you know what matters this week. No spam and you can unsubscribe in one click.
Practical and short · No spam · One-click unsubscribe