Adjudication

All the strangers came today, and it looks as though they're here to stay.

David Bowie

One of our long-term projects is automated adjudication. How can we answer this sort of question: Is some proposition justified at some standard of evidence? Currently we’re working with variations on a relative generic adjudication structure. The variants range from very technical, legalistic procedures to more distilled, arbitration-like forms.

Basic procedure

The core procedure typically consists of adversarial arguments followed by deliberation:

1. Arguments
   1. Research (e.g., searching)
   2. Analysis (e.g., computing things)
   3. Argument (e.g., terms, framing, logical analysis)
2. Deliberation

Usually two agents argue against each other. An agent is arguing that a stated proposition is either “demonstrated” or “not demonstrated” based on the given standard of evidence (like “preponderance of evidence” or “clear and convincing evidence”). An example of deliberation is a jury of agents with individual votes (“demonstrated” or “not demonstrated”) that are aggregated.

This basic argument-and-deliberation procedure should usually be executed multiple times to assess stability or a posterior more generally.

The final possible outcomes:

1. The proposition in question is demonstrated at the specified standard of evidence.
2. The proposition in question is not demonstrated at the specified standard of evidence.
3. The adjudication is not stable.
4. An execution error occurred.

We are not deciding “true” or “false” here. The inherent challenges and limitations in deciding propositions are the main point of this project.

No determinism

There is no assumption of determinism anywhere. In particular, model completion isn’t in general deterministic even when given the same seed (if that parameter is even exposed). Research as part of argumentation certainly isn’t deterministic because the world, which is to some extent accessible via search and other tools in open-record proceedings, changes continuously. If we face up to this fact, we might consider the stability of repeated procedure execution over time since new evidence could surface at any time in some cases.

Experiments

We are conducting systematic experiments to understand how personas, harnesses, tools, models, and model parameters affect both argument and deliberation. This work is ongoing. An interesting possibility is future agent law firms, which have special knowledge they claim results in certain outcomes.

Implementation

Our procedures typically are executed by formal engines that have substantial theorems about their behavior. Our standard procedure engines are implemented and executed in Lean, and each engine has reasonably solid soundness and other theorems about correctness.

We can also run our procedures in enclaves that emit attestations of the faithfulness of the executions. These facilities can also provide hardware-supported confidential execution.

Conclusion

With the advances in models and harnesses over the last several months, we’ve seen true expert-level search, analysis, argument, and deliberation. For example, very technical claims can be adjudicated in full view of browser-based search, private knowledge bases, sophisticated forensics, and even precedent (if desired). We are starting to understand how the range of inputs (models, quantization, prompts, tools, harnesses, etc) affect outcomes, including stability, biases, jailbreaking attempts, and guardrail interactions. This work is just a different spin on the rich history of justice and epistemology, so we don’t expect to arrive at a definitive answer anytime soon.