Appendix
Terms and References
Plain-language definitions and authoritative references for the guide. Terms are defined here and at the point of first use in the relevant module.
Glossary
| Term | Definition |
|---|---|
| Acceptable-Use Policy | A document that specifies which AI tools may be used within an organization, by whom, and for what purposes. |
| Automation Drift | The gradual degradation of AI output reliability as the conditions around the task change over time. |
| Authority Confusion | A failure mode in which AI produces confident-sounding output in areas where it has no verified basis or recognized authority. |
| Context Collapse | A failure mode in which an AI answers without accounting for unstated context — your jurisdiction, timing, role, or organizational constraints — producing a generic answer that does not fit your actual situation. |
| Data Exposure | The risk that sensitive information pasted into an AI system may be stored, logged, reviewed, or used in future training. |
| False Precision | A failure mode in which AI states specific numbers, dates, citations, or claims with apparent confidence but without a verified basis. |
| Governance | The policies, controls, oversight procedures, and accountabilities that determine how AI is used. |
| Hallucination | A failure in which an AI generates output that sounds correct but is factually wrong. |
| Human-in-the-Loop | A design requirement in which a human must review and approve AI output before it causes real-world action. |
| Large Language Model | A type of AI system trained on vast amounts of text to generate responses to prompts. |
| Omission | A failure mode in which an AI response leaves out an important constraint, exception, or caveat. |
| Prompt | The text input you provide to an AI system: a question, instruction, or task. |
| Prompt Injection | An attack in which malicious content embedded in input is designed to override instructions or cause unsafe behavior. |
| Prompt Sensitivity | The tendency of AI systems to produce meaningfully different outputs when the same question is phrased slightly differently. |
| Risk Ladder | The five-level framework used in this guide to categorize AI tasks by potential harm and determine the appropriate review level. |
| Rollback Procedure | A documented process for disabling or reverting an automated AI system. |
| Scoped Permissions | Access rights deliberately limited to only what a system needs to perform its task. |
| Security Failure | AI-specific security risks including prompt injection attacks, unsafe access to connected tools, and credential leakage. |
| Training Data | The large collection of text and other content used to teach an AI system to generate responses. |
Reference List
Frameworks & Standards
Provides a structured framework for identifying, assessing, and managing risk in AI systems.
The tiered risk-assessment methodology behind this guide’s Risk Ladder: the severity of the worst-case consequence determines the level of control required.
Documents common and critical security risks in deployed AI systems, including prompt injection and excessive agency.
owasp.org/www-project-top-10-for-large-language-model-applications
Establishes a risk-based regulatory framework for AI systems used in or affecting EU markets.
Primary source: eur-lex.europa.eu (Official EU legislative text)
Peer-Reviewed Research
“A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.” ACM Transactions on Information Systems, 43(2), 2025. Documents hallucination as a persistent problem with no reliable internal detection mechanism.
“How Is ChatGPT’s Behavior Changing over Time?” Harvard Data Science Review (MIT Press), 2024. Documents substantial behavioral differences between successive GPT-3.5 and GPT-4 versions — improvements on some tasks, regressions on others.
“Dissecting racial bias in an algorithm used to manage the health of populations.” Science, 366(6464), 447–453, 2019. A widely-used care-management algorithm systematically underestimated the needs of Black patients relative to equally ill white patients.
Documented Cases & Records
Federal court order sanctioning attorneys who submitted a legal brief containing AI-generated case citations that did not exist. Illustrates hallucination in a professional legal context: the output was polished and looked authoritative; the cases were fabricated.
Case record: CourtListener (free public access) — No. 22-cv-1461 (PKC) (S.D.N.Y. June 22, 2023)
ChatGPT fabricated detailed fraud allegations against a real person with no connection to the case. OpenAI was sued for defamation; the Superior Court of Gwinnett County, Georgia granted OpenAI summary judgment in May 2025. The suit failed — but the fabrication itself occurred, which is the risk it illustrates.
Superior Court of Gwinnett County, Georgia — summary judgment for OpenAI, May 2025.
High Court of England & Wales, judgment 4 November 2025. Getty’s principal claims largely failed, but the court found that earlier versions of the model could generate images bearing Getty/iStock watermarks (a limited trademark finding).
An automated trading system placed millions of unintended orders across roughly 150 stocks in about 45 minutes, causing approximately $460 million in losses; the firm did not survive. No human reviewed the orders before they executed.
SEC Administrative Proceeding File No. 3-15570 (Release 34-70694), Oct 16, 2013
An error in an internal automated tool caused hundreds of homeowners to be wrongly denied loan modifications; many lost their homes. The fault went undetected inside an automated process until regulatory review.
Disclosed in Wells Fargo & Co. Form 10-Q (SEC), August 2018.
Engineers pasted proprietary semiconductor source code and confidential meeting notes into ChatGPT in April 2023. The data left the company’s control and could not be recalled; Samsung restricted generative-AI tools on internal networks shortly after.
Reported April–May 2023; company-confirmed. Widely reported.
An experimental résumé-screening tool learned to penalize applications associated with women, reflecting a decade of historical hiring data. Amazon discontinued the tool.
Amazon confirmed publicly, 2018. Widely reported.
An Associated Press investigation (October 2024) found the speech-to-text model fabricated content never spoken, including invented medications — a concern given its deployment in medical settings. One researcher found hallucinations in roughly 8 of 10 public-meeting transcriptions examined.
Associated Press investigation, Oct 26, 2024 (reported via TechCrunch)
Verify current versions directly with each source. Regulatory and technical standards are updated over time.
Get notified when this guide is updated — sign up here.