General-purpose models and high-risk classification under the EU AI Act
Notes from the AI Act Forum Series
Last Friday I sat through the first meeting of the AI Act Advisory Forum. It ran in two parts, a general plenary in the morning and an afternoon session on the Commission’s draft guidelines for classifying high-risk AI systems, the rules that decide when a system falls into the heavily regulated category.
The Forum is convened by the European Commission’s AI Office, the unit inside DG CNECT (the Commission’s digital department) responsible for implementing the AI Act. The AI Office sets the agenda and runs the meetings, and Roberto Viola, the Director-General of DG CNECT, opened the session.
The morning was slow. It was mostly the AI Office kicking things off.
A few things stood out.
Of the roughly 180 members, many are universities and civil-society organisations, which is what you would expect. On the industry side, the United States has eight organisations on this forum, more than most of the 27 member states. OpenAI, Amazon, Microsoft and Anthropic are all members.
Then there are the co-chairs. The Forum will elect two, one from the commercial side and one from the non-commercial side, and the rule is that they must differ in gender and nationality. I will be honest, I do not think these parity rules help. They tend to get people arguing about gender instead of fixing anything, and the Forum is already well balanced between men and women, so I was not sure what the rule was solving here.
The bigger problem with the co-chairs is the process. We are being asked to nominate and elect them before we have signed off on the rules of procedure that are supposed to govern the Forum. The AI Office was open that this is because they want to move fast. But the election itself is awkward. You put your own name forward, or back someone else, without knowing who else will step up, which fragments votes and leaves you guessing whether anyone will end up representing your side at all.
That ties into another imbalance between commercial and non-commercial stakeholders, where the influence may be asymmetric. No funding is provided to support the work of the representatives, and that lands very differently on the two sides. A company sends someone who is on the clock, with a team and a budget behind them. The same gap shows up in the co-chair election, because the non-commercial side is the one least able to spare the time and budget to put a candidate forward.
The one part of the morning related to Article 50, the transparency rules that require AI-generated content to be labelled. The approaches under discussion include “AI generated” tags and an optional EU icon. For images, audio and video it is clear how that would work. For text it is much weaker, and I think the approach being floated has a privacy problem, because a hidden marker that travels with text starts to look like a tracking cookie. It is enough of its own subject that I will write about it separately, so I will leave it there for now.
The afternoon was on the high-risk guidelines. The argument I want to get into is the one the large model providers have been pressing, in this consultation and in their public positioning.
To follow it you need to know what “high-risk” means. The AI Act sorts systems by what they are used for. Most uses are left alone, a few are banned, and a named set in between are “high-risk”, which means the provider has to ensure compliance with the Act before and after putting the system on the market: a documented risk-management system, data governance, human oversight, testing, and a conformity check.
If you want to know more, you can read these articles either directly in the AI Act here, or in a shorter PDF with just these articles here:
The list of high-risk uses is in Annex III, and the logic running through it is simple once you see it. The AI Act flags as high risk cases where a human being is the target of classification. You point the system at a person and let it judge them: who to hire, who gets credit, who gets a benefit, who the police should look at, what grade a student gets.
General-purpose models, the big foundation models, do sit on a separate track in the Act. But that track is not a free pass, and the framing the providers use blurs this. The GPAI (General-Purpose AI) clauses refer to general-purpose models that do not reach into high-risk uses. They are not for “any model that exists today”, because the models that exist today will happily do the high-risk thing if you ask them.
The industry argument is that, read literally, the guidelines would pull almost every general-purpose model into high-risk, because a general model can be pointed at anything, so merely being able to do a high-risk task would be enough, and that this collapses the whole general-purpose category. The paragraph this argument leans on is Paragraph 12.
But the argument strawmans Paragraph 12. Paragraph 12 does not say that any capable model is high-risk. It says a general system is high-risk if it is presented as do-anything and the provider does not genuinely limit or exclude the high-risk uses, and it spells out the way to stay out: limit those uses. A real exclusion, not a disclaimer in the terms of service while you market the use anyway.
These models are potentially dangerous, and the companies that build them have the budget to ensure they are safe. The argument against the rule is, in effect, “we cannot constrain our models.” That is the wrong argument, because it is not true.
You cannot get bomb-making or bioweapon instructions out of ChatGPT. It has read the whole internet and knows perfectly well how, but the provider trained it to refuse. They have already built the ability to fence off a capability. So when they imply they cannot it's just not true.
In part we must realize that the reason these models are so capable is that they are so general, and generality, usefulness and danger all rise together. The whole point of a large language model is that, at scale and with enough data, a kind of intelligence emerges, and it is that generality that gives the model its edge and that makes it dangerous. I do not mean science-fiction consciousness, just a system smart enough that its raw capability is a risk. The Act has a place for exactly this, a separate tier called systemic risk for the most capable models, with obligations to evaluate and stress-test them, though those obligations feel lighter than the high-risk ones. My worry is that today that tier leans on the providers to report on themselves. The Commission, or perhaps a higher-level body, needs to play a role here.
If the providers wanted to make an honest case, they would argue that a specific use should not count as high-risk, or that the rules hurt how useful their models are. Take grading students. There are real problems with letting a model decide a student’s grade, but there is also ways where, done carefully, it could be fairer than what we have today. That is a debate worth having on the merits, and indeed the AI Act foresees this in Article 7. Instead of having it, the providers are trying to squeeze out of the classification altogether. The honest path is the opposite: classify as high-risk, and work with the Commission so it has what it needs to check these systems are safe.
Beyond this, a question that the AI Act stays quiet on, is whether open-weight models should be treated differently from closed ones, which changes how you would enforce any of this. I will attempt to write about this soon.
So a provider has a few honest choices. Comply for the high-risk parts, treat the whole system as high-risk and comply, or build a version that genuinely fences those uses off. What the guidelines do not offer is selling an unconstrained do-anything system and skipping the obligations, which is what the industry argument, taken to its conclusion, is asking for.
I will send my comments to the AI Office, the short version of them is this: Paragraph 12 is well written and does not collapse the general-purpose category, and we believe it's important to keep both tracks live, high-risk for what the system is used for and systemic risk for how capable the model is, applied in parallel rather than traded off. I will keep going on that here over the coming weeks, starting with the transparency rules and the question of open versus closed weights.
The AI Act Advisory Forum is the body the European Commission’s AI Office has set up to give it and the AI Board independent technical expertise as the Act moves into enforcement.
Its membership is a balanced mix: industry, start-ups and SMEs, civil society, academia.
I was selected to sit on it as an expert on behalf of coop.tech, a Portuguese tech cooperative: technologists working together for mutual benefit, building software for organisations and widening access to technology and tech literacy in the community.