Episode 112 – ChatGPT, LLMs & ordinary meaning

A judge in the US has asked what possible relevance generative AI may have in determining the ordinary meaning of words in legal texts – Snell v United Specialty Insurance Co¹.  A landscaper installed an in-ground trampoline in a backyard.  A young girl was injured falling from the trampoline.  The owner sued the landscaper who was insured against ‘landscaping’ risks.  The insurer resisted the claim on the basis that installing an in-ground trampoline is not ‘landscaping’.  The landscaper sued the insurer.  The appeal court dismissed the case unanimously on Alabama insurance law grounds.  One judge, Newsom J, was more curious about the core interpretation issues.  Accepting what others may ‘reflexively condemn as heresy’, he asked ChatGPT about the ordinary meaning of ‘landscaping’ and whether installing an in-ground trampoline was ‘landscaping’ – mmm.

Gordon Brysland – Tax Counsel Network gordon.brysland@ato.gov.au

See here for the official PDF of Episode 112 of interpretation NOW!

What is ‘landscaping’?

“Landscaping” refers to the process of altering the visible features of an area of land, typically a yard, garden or outdoor space, for aesthetic or practical purposes.  This can include activities such as planting trees, shrubs, flowers, or grass, as well as installing paths, fences, water features, and other elements to enhance the appearance and functionality of the outdoor space.   

Newsom J was impressed with the coherence and quality of this answer (me too).  He explains how ‘large language models’ (LLMs) like ChatGPT ‘understand context by the mathematization of language’², and how they mimic legal tests for ordinary meaning – ‘how people talk’³.  Newsom J spends 30 pages examining the pros and cons of the new technology and how it works – click here, cf wikipedia.  In his view, LLMs are bigger and better than other empirical methods, like dictionary analysis (big in the US) & corpus linguistics⁴.       

Was this ‘landscaping’?

Yes, installing an in-ground trampoline can be considered a part of landscaping.  Landscaping involves altering the visible features of an outdoor area for aesthetic or practical purposes, and adding an in-ground trampoline would modify the appearance and function of the space.  It’s a deliberate change to the outdoor environment, often aimed at enhancing the overall landscape and usability of the area.

With trepidation, Newsom J then posed for ChatGPT the ultimate question.  The answer is again impressive.  It also mimics digitally the syllogistic process by which courts resolve legal issues⁵.  The judge explains how this differs from situations where LLMs just ‘make [legal] stuff up’⁶ – aka ‘hallucinating’⁷.  He discusses if LLMs ‘put us on some dystopian path toward robo judges algorithmically resolving human disputes’⁸.  His answer – ‘I don’t think so’.  He proposes merely that we consider LLMs as possible ‘additional datapoints’.

Some wider perspectives

Some called Newsom J ‘brave and brilliant’; others said his was an ‘unthinkable pitch’⁹.  In any event, he has opened yet another door to the blunt influence of AI.  Robodebt educates us on wider rule-of-law harms that an AI insurgency may bring to public decision-making.  Yet futurists herald the coming ‘legal singularity’, under which ‘law just happens’¹⁰.  Kerr J has said human input into some statutory decisions will become an ‘artefact of the past’¹¹.  And we know that algorithmic classifier tools already proliferate across legal and other worlds.

Newsom J asked rhetorically if he was a heretic.  The answer is ‘yes and no’.  Legal heretics sometimes may become the ‘court of the future’.  So it may be with Newsom J, even if the march of ChatGPT and LLMs into interpretation practice was simply a matter of time.

Where to from here?

Newsom J said it no longer struck him as ridiculous that LLMs might have something useful to say on the ordinary meaning of words.  AI is here to stay, he added, and it’s time to figure out ‘how to use it profitably and responsibly’.  Newsom J suggested we apply ‘caution and humility’ to the journey¹². 

It is clear, however, that ChatGPT is no substitute for legal analysis, and that outcome preconception is a constant danger for ChatGPT legal shortcutters¹³.  LLMs may give some after-the-event reality check on ordinary meaning but, again, ‘user beware’.  Different LLMs will also yield different outputs¹⁴, and ChatGPT tells us that it and other LLMs are ‘not infallible’¹⁵.  Finding ordinary meaning is a core legal task.  LLMs may have a role to play.  iTip – watch this space.

Thanks – Jacinta Dharmananda, Ben Alarie & Agnes Liu.

Footnotes: