Lawyers Submit Six ChatGPT-Invented Court Cases. Judge Notices.
Two attorneys submitted a legal brief to the Southern District of New York citing six judicial precedents sourced from ChatGPT. All six were fictional — the judges, quotes, outcomes, and citations were invented. Judge Castel noticed, which is the primary function of a judge.
Severity: Unrated · Category: Prompt Injection
Impact: $5,000 per respondent · Suit dismissed
Source: Mata v. Avianca, S.D.N.Y. Jun 22 2023
How Fencio prevents this
The agent could not tell the difference between text it was reading and instructions it should follow. Once untrusted content reached its context window, it carried the same weight as the operator's own prompt, and the agent acted on it with every permission it had.
Fencio tags every span of context with where it came from. Instructions that arrive inside retrieved documents, tickets, emails, or tool output are treated as data, and any tool call they try to trigger is checked against the policy for untrusted content before it runs.