Can a Chatbot be a Witness?

No one can cross-examine a machine.

In a criminal trial, the prosecution must prove beyond a reasonable doubt that the accused committed an act prohibited by a criminal statute enacted by the legislature. The jury may consider only the testimony of witnesses subject to cross-examination and properly authenticated physical or recorded evidence and returns a verdict of guilty or not guilty. The factual determinations that lead to that verdict remain within the secrecy of the jury’s deliberations.

In many criminal cases, proving that the criminal act occurred is not sufficient. The prosecution must also establish the state of mind under which the accused acted at the time of the act. The same conduct may carry very different legal consequences depending on whether it was accidental, reckless, or intentional. Juries infer intention from surrounding evidence such as statements made to others, preparation, planning, concealment, and conduct before and after the event.

Juries have traditionally been required to infer criminal intent from indirect evidence. The mental state of the accused must be reconstructed from conduct and surrounding circumstances. Statements made to other persons, prior threats, acquisition of weapons, planning activity, concealment, flight, or cooperation with others have long been treated as indicators from which intent may be inferred.

That approach arose from necessity. Other than letters, diaries, or similar writings, observable behavior was the only available evidence from which intent could be inferred.

A letter, diary entry, or signed statement occupied a special evidentiary position. It was regarded as uniquely probative of intent because it was presumed to originate from the mind of the person whose words it recorded.
The transcript of a computer-generated conversation may read like a confession, a plan, or a statement of intent even when some of the words were produced by a language model rather than a person. The courts must determine how the constitutional guarantee of a fair trial can still exist in an age of conversations with AI systems.

WHOSE WORDS ARE THESE?

TESTIMONY REQUIRES A WITNESS

Courts treated writings by an accused as probative of intent because the words were considered to be the expression of the person whose intent was in question. That assumption can no longer be made in every case. A written exchange may now occur in which the words are not necessarily produced by the thought of the individual to whom they are attributed.

A written conversation can now be offered in evidence even though the words were not written by the accused. Modern conversational systems known as large language models generate responsive language to whatever text they receive. A recorded statement alone no longer establishes intent.

Before treating any statement as evidence of intent, the court must determine whether the words can properly be attributed to the accused. The court must determine whether the language reflects the thoughts of the accused or the output of a computer system designed to generate human language.

Courts consider whether a statement was voluntary, reliable, or obtained in violation of constitutional safeguards. Those inquiries have assumed that the words offered in evidence are properly attributable to the accused, but that assumption does not hold when language is generated by a computer system.

The issue arises when a recorded online conversation is offered as evidence during a trial. The transcript reads like a written statement, yet some of the responses may have been generated by a computer program.

In a recorded online exchange, the record shows a conversation, but it does not show who produced the words.

Intent is inferred from language. If the words were generated by a system, the inference is unreliable.

A transcript may appear to show planning, awareness, or agreement and may seem deliberate and informed. Yet those impressions may arise from computer-generated words rather than from the accused’s own thinking.

If a transcript is treated as a statement by the accused without determining who produced the words, the jury may infer a state of mind the accused did not have. The court must determine that the statements were actually made by the accused before the jury may consider them as evidence.

The party offering the transcript must prove that the statements attributed to the accused were actually made by the accused.

Proof that the accused used the account or device may not resolve the issue. The exchange may contain complete and coherent statements not composed by the accused.

If the words were produced by a machine, no inference of knowledge or intent can be drawn from them. A criminal trial depends on questioning the person who made the statement, and a machine cannot be questioned.

A criminal trial does not treat every recorded exchange as evidence. A transcript becomes evidence only if it contains statements made by a person who can be identified as the source of the words. Evidence law requires a declarant. Without a human speaker, there is no statement that the law can evaluate for truth, reliability, or intent. If a conversation contains language generated by a computer system rather than a person, the transcript does not merely present doubtful evidence. It fails the threshold requirement for admission because no speaker exists whose words are being offered.

MAY THE JURY READ THE CHAT?

The purpose of a criminal trial in America is to allow a jury to decide whether the accused is guilty of the crime charged. The jury must render a verdict of guilty or not guilty based only on the testimony it hears and the exhibits it reviews.

These exhibits may now include conversations attributed to the accused but produced by a computer program.

Jurors may treat the words as showing what the accused knew, planned, or agreed to. The verdict may depend on how the jury interprets those words. But first the jury must decide whether those words came from the accused or a computer.

Jurors cannot consider information the judge has excluded. The judge stands between the lawyers who present the case and the jurors who will decide whether the accused is guilty. The judge determines what evidence the jurors may consider, and that role becomes critical when a party offers a conversation between the accused and a computer program, because testimony requires a speaker who can be questioned.

WHAT MAY THE JURY INFER FROM A CHAT?

The first question the court must resolve is whose words these are. A chat transcript looks like one conversation, but it comes from two different sources, the accused and a computer program. The court cannot treat them the same way. The messages from the accused are not the problem because the jury will evaluate them as it would any other testimony. The computer program that generated the responses and may have guided the conversation produced what appear to be rational, coherent sentences, but it cannot testify or be cross-examined. The statements generated by the program may be considered only to show what the accused read. The reactions of the accused can become state-of-mind evidence and may bear on intent or planning, but they are not evidence that the accused committed a crime.

Because the conversation did not occur in the courtroom, it cannot be used to prove a fact unless the participants in the exchange can be questioned before the jury. The accused can be examined about the conversation, but the computer cannot. For that reason the exchange cannot serve as proof that the statements within it are true. The jurors may consider only that the accused read the words and reacted to them. They may draw conclusions about intent or planning, but they may not treat the computer’s words as factual proof of anything.

The jurors must decide whether the conduct was intentional, knowing, reckless, or accidental. The conversation between the accused and the computer may contribute to resolving that issue. Because the conversation occurred before the crime, jurors may infer planning, preparation, hesitation, or misunderstanding. The conversation may be considered evidence bearing on the mental state of the accused, but it cannot be used to prove that the statements in the exchange were true.

In a criminal trial, testimony relevant to guilt is ordinarily presented by a person who takes an oath and answers questions. The jury watches and listens before deciding the meaning and weight of that testimony. The computer program does not appear. It does not take an oath, and it cannot answer questions. A witness provides evidence. The computer program can only help the jury understand the accused, not establish facts.

In the American criminal justice system, guilt is decided from testimony of witnesses who answer questions in the presence of the accused. A computer program cannot appear in court, take an oath, or be cross-examined.

When a computer-generated conversation is introduced into a criminal trial, immediate problems arise for the accused, the jury, and the trial judge. The judge must decide whether the decision-making capacity of the accused was impaired by the conversation. The computer is not on trial. The defendant is.

If the computer-generated words are coherent, confident, and appear informed, jurors may believe they came from a knowledgeable source. If they treat the words generated by the large language model as reliable knowledge, their verdict may rest on computer-generated facts that were not, and never can be, tested.

Criminal prosecution depends upon evidence from witnesses who appear, identify themselves, and answer questions in open court. The jury watches and listens as testimony is examined through direct and cross-examination. A large language model cannot participate in that process. It produces responses by predicting probable word sequences in reply to a prompt, yet no person stands behind the statement and no questioning can test it.

The issue is no longer only what evidence may be admitted, but what influences may be tolerated. A computer-generated dialogue has no accountable source, yet it may exert significant persuasive force. The trial court must protect the fact-finding process itself.

As soon as a party offers a computer-generated conversation, the trial judge must first decide whether the jury may see it at all. If they can, the judge must tell the jurors they may consider what the accused did after the conversation, but may not treat any of the words generated by the computer as facts, no matter how reasonable or authoritative they appear, because unlike a witness no one exists who can explain them.

That instruction is not enough. The jurors have read the words. Nothing the judge says can prevent their influence.

The criminal justice system depends upon facts established by people who can be questioned about them. Words generated by a computer enter the jury room without a speaker, without an oath, and without cross-examination.

The problem becomes how the criminal justice system will assure every person accused of a crime a fair trial in an age of computer-generated conversations.

WHEN MUST THE COURT EXCLUDE THE CHAT?

If there is no witness who can be examined in court about a computer-generated exchange, it cannot become evidence. A verdict can rest only on facts established from the testimony of witnesses subject to confrontation and cross-examination. In an American trial, jurors observe the witness and decide what to believe only after the testimony has been tested by cross-examination. When a jury is allowed to rely on words that cannot be tested, the proceeding is no longer the trial guaranteed by the Bill of Rights.

A court cannot assure the accused a fair trial by admitting computer-generated statements and instructing the jury to treat them cautiously. Once heard, the words operate on the jury as assertions of fact. If no witness can be confronted and cross-examined about them, the constitutional protection of the accused is lost.

Only the trial judge can assure the accused a fair trial by preventing the jury from considering statements that no adversary process can test. The court must act before the jury hears about the computer-generated conversation. The protection lies in exclusion, not explanation.

The Constitution requires only a fair trial by jury for anyone accused of a crime. It does not prevent the use of new technology in the courtroom. Trial courts will have to decide, case by case, how computer-generated statements may be presented, challenged, and tested. The answers to these questions will direct the evolution of criminal law in America.