Four Minnesota Law professors recently drew worldwide attention for their experiment in which an artificial intelligence-powered chatbot averaged a C+ — “a low but passing grade” — on four law school final exams. 

The result, achieved by an artificial intelligence (A.I.) model called ChatGPT, raises questions about the implications of such technology for the future of legal education and the legal profession. While ChatGPT’s arrival has sparked intense discussion among professors, its exam performance also has drawn widespread interest from news outlets across the nation and the globe, including China, India, and Korea, to name a few.

“These tools have a lot of potential to transform our lives,” says Professor Jonathan Choi, who proposed the experiment. “And text-based professions like the law are going to be the most susceptible to change as a result of the introduction of these tools.” 

Choi asked faculty members to take part in the experiment as they exchanged emails about ChatGPT shortly after its release in late November, just before fall semester finals. Choi, whose specialties include applying A.I. or natural language processing to the study of legal issues, wanted to test how it could perform in real exams compared to real students.

Choi used ChatGPT to produce exam answers in four Law School classes: Constitutional Law–Federalism and Separation of Powers; Employee Benefits; Taxation; and Torts. He then had the A.I.-generated responses integrated with actual student exams. Professors Kristin Hickman, Amy Monahan, and Daniel Schwarcz graded ChatGPT’s performance relative to how real students performed but without knowing which exam was the work of ChatGPT.

The professors reported the results in a paper, “ChatGPT Goes to Law School,” posted online in January. Through 95 multiple choice questions and 12 essay questions, ChatGPT “performed on average at the level of a C+ student, achieving a low but passing grade in all four courses,” they reported.

While uneven for now, ChatGPT’S performance “suggests considerable promise and peril,” the paper concludes. Potential problems for law schools range from misuse on exams to equitable concerns if such technology were to become too expensive for some students to access, Choi says. Some practicing attorneys fear A.I. will replace them. A broader concern for society is that chatbots can produce plausible-sounding misinformation at a much lower cost.

Exciting Opportunities

Choi says he is excited about the opportunities, despite the potential for misuses of the technology. 

“Law schools as professional schools are preparing students for real jobs,” Choi says. “If the way that those jobs operate changes, we need to change the way that we teach our students.”

Law schools historically have focused on teaching knowledge, Choi says. While a lawyer in the 1960s could have a good career knowing everything about a specific topic, that vision of the practice has declined significantly as tools like Google have dramatically lowered the cost of knowledge. Choi thinks that cost will continue drop with the introduction of A.I. language models into the profession.

 Choi expects A.I.-assisted legal writing courses to emerge. “Prompt engineering,” knowing how to craft and improve on prompts to get ChatGPT to produce the best results, will be an important job skill and something that most law schools already are discussing how to teach. “It’s something that we’re talking about at Minnesota,” Choi says. “Hopefully, we’ll have a class in the next few years that teaches these skills.”

For practicing lawyers, chatbots may be a boon, Choi says, as they take on menial labor lawyers often do not want to do. That would leave lawyers more time for high value-added work, while the greater productivity afforded by the A.I. model would significantly lower the cost of legal services. That could increase demand for lawyers and expand legal representation to people who cannot afford a lawyer.

“There are a lot of under-lawyered areas of society,” Choi says “If we open up people’s time by using these A.I. tools, it’s possible that being a lawyer will become more fun, and that demand for overall legal services — human legal services — could actually increase. It remains to be seen but I think there’s a lot of opportunity.”

AI as a Starting Point

One of the clearest ways that ChatGPT might change legal education and the practice of law is by serving as a starting point for legal writing, Monahan says. ChatGPT does a good job summarizing the law on broad subjects such as constitutional law and torts as well as on complex subjects like the federal tax code.

“If there's something to write, see what chat GPT produces, and then edit it,” Monahan says. “I see it as a potential aid, not a replacement. That’s one thing that struck me, whether you’re a law student or a lawyer, if you’re trying to get the lay of the land, it probably can be a pretty helpful starting point.  We should as a law school think through how practicing lawyers are likely to use this technology and that should inform how we incorporate it into our curriculum,” Monahan adds.

AI as a Tool to Leverage

In the short term, Hickman expects that some professors will return to traditional, timed exams administered in class where students may have access to books or notes but not the internet. That would reverse the trend toward take-home exams, which has continued in many classes after growing increasingly common during the COVID-19 pandemic.

“I think of ChatGPT as just the latest in a series of tools that have become available to students over time that have required educators to adjust how they teach,” Hickman says. “It undoubtedly can be leveraged to contribute to the educational process and also to potentially improve legal practice. That is worth contemplating as part of a collaborative process with colleagues.”

Hickman urges caution in describing ChatGPT’s performance on those final exams as earning a passing grade. A C+ average would put a real student on academic probation, with one-to-one counseling required to identify how to improve performance. “It’s more indicative that ChatGPT did acceptably but not well,” Hickman says.

Failing grades are not common at top law schools, she notes.

“One of the realities of being at a highly competitive law school is that virtually everyone who goes to law school here, the vast, vast majority of students here, are smart, hardworking, engaged in their legal studies,” Hickman says. “While we have a curve and they do better or worse relative to one another, we don’t have a lot of students who truly would fall into a failing category.”

AI and the Future of Legal Education

When Schwarcz explored ChatGPT’s capabilities, “The power of the A.I. just really blew me away,” he says. He expects it to improve as the A.I. gets more training in specific areas of the law. 

“Because this is such a powerful new tool, in my view, it’s going to be incumbent on law schools to start figuring out how to train people to use this technology effectively and ethically,” Schwarcz says. “Just like we have methods to train our students to use Westlaw and Lexis, I think we’ll have to train them to use these other tools. It’s a real question to figure out the right way to train students, to make clear to them what is appropriate and ethical and what is not on exams and in non-exam situations.”

As a follow-up, Choi hopes to do another experiment to test how the performance of real students on exams would improve with access to ChatGPT. Some students would get training in how to use ChatGPT while others would not.

“We expect that ChatGPT would perform better with a human monitoring it, requesting a better answer when the response is insufficient,” Choi says. “It’s the combination of a computer and human that we think for the immediate future at least will be the best application of this technology.”

 Todd Nelson is a freelance writer in Lake Elmo, Minnesota