As AI becomes embedded in every corner of our organisation, we must recognise that its decisions now touch customers, employees, and even the continuity of our critical services.
That’s why responsible AI governance rests on five essentials: ensuring our systems treat people equitably and with full awareness of their impact; demanding transparency so every AI decision can be traced, explained, and defended; assigning clear human accountability so we always know who is responsible for the outcomes; enforcing strong data stewardship so no system improperly exposes, infers, or exploits sensitive information; and continuously monitoring performance and security to ensure the technology remains safe, stable, and resilient.
These aren’t theoretical ideals — they’re the safeguards that protect human rights, maintain public trust, and keep us compliant with the strict expectations of modern EU regulation. In short, transparent and disciplined AI governance isn’t a barrier to innovation; it is how we ensure that AI strengthens our organization instead of becoming a silent liability.
Prohibited AI systems (Chapter II, Art. 5)
The following types of AI system are ‘Prohibited’ according to the AI Act.
Source: https://artificialintelligenceact.eu/high-level-summary/
AI systems:
- deploying subliminal, manipulative, or deceptive techniques to distort behaviour and impair informed decision-making, causing significant harm.
exploiting vulnerabilities related to age, disability, or socio-economic circumstances to distort behaviour, causing significant harm.
- biometric categorisation systems inferring sensitive attributes (race, political opinions, trade union membership, religious or philosophical beliefs, sex life, or sexual orientation), except labelling or filtering of lawfully acquired biometric datasets or when law enforcement categorises biometric data.
- social scoring, i.e., evaluating or classifying individuals or groups based on social behaviour or personal traits, causing detrimental or unfavourable treatment of those people.
- assessing the risk of an individual committing criminal offenses solely based on profiling or personality traits, except when used to augment human assessments based on objective, verifiable facts directly linked to criminal activity.
- compiling facial recognition databases by untargeted scraping of facial images from the internet or CCTV footage.
- inferring emotions in workplaces or educational institutions, except for medical or safety reasons.
- ‘real-time’ remote biometric identification (RBI) in publicly accessible spaces for law enforcement, except when:
- searching for missing persons, abduction victims, and people who have been human trafficked or sexually exploited;
- preventing substantial and imminent threat to life, or foreseeable terrorist attack; or
- identifying suspects in serious crimes (e.g., murder, rape, armed robbery, narcotic and illegal weapons trafficking, organised crime, and environmental crime, etc.).
EU-wide Regulatory Action: Clarification and enforcement framework for banned AI practices (2025 onwards)
- On 2 February 2025, the first “prohibited uses” under the AI Act became effective. The European Commission published guidelines clarifying that certain practices are banned: these include social scoring, real-time biometric identification in public spaces (with narrow exceptions), emotion recognition in workplaces/education, and manipulative AI designed to exploit vulnerabilities.
- Legal commentaries and expert analyses have underlined that the prohibition applies broadly — to providers, deployers, distributors, and anyone putting such AI “into service” in the EU.
- Advocacy groups such as the Center for AI and Digital Policy (CAIDP) Europe have submitted formal contributions and comments to national authorities, emphasising that social-scoring systems (and similar prohibited AI) infringe on fundamental rights, dignity, and non-discrimination principles.
There was even a earlier conviction:
Clearview AI – Massive illegal facial-recognition database
- In 2024, the Autoriteit Persoonsgegevens (Dutch DPA) imposed a €30.5 million fine on Clearview AI, declaring that it had built an “illegal database” of billions of facial images — scraped from the Internet (public sources, social media, etc.) — without consent from data subjects.
- The DPA also ordered Clearview to cease its data-collection practices in the EU, and warned that any use of its services by organisations in the Netherlands (or other EU states) is illegal.
- Regulators flagged the core issue: untargeted scraping + mass biometric identification — practices now among those explicitly prohibited under Article 5 of the AI Act.
→ Why it matters: Clearview AI is the most concrete enforcement case showing that biometric-mass scraping + identification run afoul of EU data protection and emerging AI regulation. It directly matches prohibited AI practices such as untargeted biometric databases.
The CEO asked his compliance officer practical examples:
Even if publicised enforcement cases besides Clearview remain rare (so far), the regulatory and normative environment has changed significantly: prohibited AI practices are now explicitly codified, and authorities and civil-society organisations are actively preparing to monitor and crack down on violations.
The experienced compliance officer had studied the EU AI Act and had written down some possible scenario’s adopting AI without risk assessment in alignment with the EU regulation. Allowed in the land of Cowboys does not validate a App for Europe. Insight needed?
When a triage AI unintentionally prioritizes some patients over others
A new AI triage assistant might learn from historical data that younger patients tend to recover faster, and subtly start pushing their cases to the top of the queue. Nurses notice older or vulnerable patients waiting longer than usual.
The lesson: Equity must be monitored, or AI will quietly inherit past biases.
When no one can explain why the AI flagged a “critical” condition
A diagnostics algorithm suddenly labels a routine scan as “suspicious.” The doctor disagrees — but the system can’t explain its reasoning. Radiology pauses work for hours trying to understand what the AI “saw.”
The lesson: In healthcare, unexplained AI decisions delay care and increase liability.
When responsibility becomes fragmented between IT, clinicians, and vendors
An AI falls out of calibration after a software update. IT thinks the vendor should fix it, the vendor claims clinical settings caused the drift, and clinicians assume IT is monitoring performance.
The lesson: Without clear accountable owners, small technical issues can escalate into patient-safety incidents.
When a clerk accidentally uploads sensitive patient images into a public AI tool
A staff member, under pressure, uses a free AI transcription app to summarise a handwritten note — not realising they just exposed protected medical data.
The lesson: Data governance must be simple, strict, and embedded into daily workflow.
When an AI-driven scheduling tool creates hidden operational risks
The system automatically optimises surgeon schedules for efficiency, but during a flu outbreak it keeps pushing staff to the edge of fatigue because it can’t “see” clinical reality. Doctors burn out, error risk increases.
The lesson: Monitoring AI resilience is as critical as monitoring the people who rely on it.
When the clinic’s empathy is quietly eroded by automation
A symptom-checking chatbot begins giving blunt or overly clinical responses to emotionally fragile patients, because it wasn’t tuned for tone, grief, or anxiety.
The lesson: AI must serve care, not replace compassion.
When vulnerable groups are unintentionally misclassified
Patients with disabilities or language challenges may be misunderstood by intake bots, leading to incorrect routing or incomplete records.
The lesson: AI must be designed to respect and protect all patients, especially the most vulnerable.
When an unavailable model leads to silent service degradation
During a network outage, a treatment-planning AI becomes unavailable. Staff revert to manual processes but lose time because nobody anticipated the dependency.
The lesson: AI requires the same business continuity planning as any other critical clinical system.
The CEO looking at these scenarios quickly sees that adopting AI without a structured, EU-aligned risk assessment is not a technical oversight — it’s a business failure waiting to happen.
Each example shows how easily AI can drift outside intended purpose: a triage system that quietly reproduces bias, a diagnostic model issuing alerts no one can explain, responsibility scattered between IT and clinicians, or sensitive patient data leaking through well-intentioned shortcuts. Operational tools can burn out staff, undermine clinical empathy, misroute vulnerable groups, or collapse when a model goes offline — all because business objectives and risks were never clearly defined at the start.
This is why the EU AI Act demands transparent governance, documented risk evaluation, and human-rights-centred safeguards: it forces leaders to set boundaries, assign accountability, and embed resilience before AI touches a patient or a process.
What may be “allowed in the land of cowboys” is absolutely not acceptable in a European clinical context. A CEO must therefore establish clear, measurable objectives for every AI system, backed by rigorous, scenario-based risk assessments that match EU expectations — because in healthcare, trust, safety, and compliance are not constraints on innovation, they are the conditions that make sustainable, competitive, and responsible AI adoption possible.
Closing Conclusion
In the emerging regulatory landscape, the organisations that thrive will be those that treat AI governance not as an obligation, but as a competitive advantage. The enforcement of ISO/IEC 42001, aligned with the strict expectations of the EU AI Act, and balanced against broader EU regulations such as NIS2, GDPR, DORA, MDR, and sector-specific patient-safety rules, forms a single, integrated framework for trustworthy AI. Together, they push enterprises to build systems that are transparent, auditable, secure, explainable, and anchored in human rights. And when executed well, this disciplined compliance model does far more than keep regulators satisfied.
It becomes a differentiator.
Enterprises that adopt legal-balanced AI governance early are better positioned to avoid the costly cycle of fines, emergency recalls, rushed redesigns, or public scrutiny that competitors will be forced to navigate later. They reduce the risk of reputational damage, revenue loss, and shareholder distrust — all of which can be triggered by a single opaque algorithm or a mishandled AI incident. Instead, they gain the confidence of regulators, investors, and customers by demonstrating that innovation is built on responsibility, not risk. In a market where trust is becoming the currency of digital transformation, those who embed ISO 42001 and EU AI Act compliance into their strategy don’t just meet the rules — they set the standard, strengthen resilience, and lead the industry forward.
Strict, transparent, human-rights-aligned AI governance doesn’t slow down progress.
It accelerates it — safely, credibly, and competitively.
