The team that built the AI should not grade it.

The team that built an AI system should test it. It should not be the only team deciding whether the system is safe enough to trust.

Anthropic took a meaningful step last week by giving outside evaluators access to internal systems, tools and evidence. The arrangement is about frontier models, but the principle applies to every company putting AI into real work: builders know the system best, and that is exactly why their judgment needs an independent challenge.

What changed

On October 15, Anthropic announced that six employees of METR, a nonprofit AI evaluator, had received company hardware, office space and access to internal tools roughly comparable to Anthropic's own risk-assessment teams.

That access is only half the arrangement. METR can publish its analysis of Anthropic's risks, incidents and practices without Anthropic's approval. Anthropic can require redactions for predefined categories such as customer data, privileged information and details of model architectures. METR can publicly describe the access it received and whether it believes those redactions were justified.

This is not a safety certificate. METR says many important claims rest on open scientific questions that cannot be verified with a simple pass or fail. Its published approach calls for disclosure of an evaluation's scope, duration, access limits, evidence, methods, redactions and conflicts of interest. It also acknowledges the limit of a voluntary arrangement: the evaluator still depends on the company to participate and provide access.

The move follows the White House Accord on Super Intelligence, signed in September by leaders of Anthropic, Google, Meta, OpenAI, xAI and Nvidia. The accord describes four layers: operational controls, an internal review team, independent external assessment and board oversight of the findings and remediation.

Reporting from the Financial Times notes that critics still question conflicts of interest and whether voluntary evaluation can substitute for regulation. Those are fair questions. Independence is not created by moving the same review to a different logo.

If the builder controls the evidence, the evaluation is still part of the build.

Why this matters inside your company

Most companies are not training frontier models. They are connecting existing models to mailboxes, documents, customer records, code repositories and operating workflows.

The risk sits in that whole system. A vendor's model report cannot tell you whether your support agent applies the right refund limit, whether your document assistant exposes one client's material to another, or whether an automated review skips the unusual contract clause that matters most. Those outcomes depend on your instructions, data, permissions, integrations, approval rules and people.

The implementation team carries useful context, but it also carries assumptions. It chose the test cases. It knows the intended path. It has deadlines and may own the business case for launch. None of that implies bad faith. It means the review has a structural blind spot.

Independent evaluation does not require handing the project to someone unfamiliar with the business. It requires a reviewer who did not build the system, does not own the launch target, can inspect the relevant evidence and can report uncomfortable findings to someone with authority to act.

That distinction matters most when the system appears to work. A fluent answer can hide a weak source. A successful tool call can use more permission than the task required. A high aggregate score can conceal one failure mode with a serious consequence.

What we would do before launch

1. Define the consequential outcomes. List the decisions the system recommends and the actions it can take. For each one, name the data involved, the affected person or process, the acceptable error and the accountable owner. "Helpful answers" is not a testable standard.

2. Write the failure cases before testing. Include missing data, conflicting instructions, poisoned documents, excessive permissions, stale sources, unavailable tools and attempts to cross client or departmental boundaries. Set the pass criteria before anyone sees the results.

3. Separate building from acceptance. The implementation team should run its own evaluations continuously. A different qualified reviewer should decide whether launch criteria were met. That reviewer should report outside the project's delivery chain when the system can affect money, access, legal obligations or sensitive information.

4. Give the evaluator enough access. Provide the system instructions, model and configuration history, tool definitions, permission scopes, representative data, action logs, prior failures, incident records and rollback controls. Protect personal and confidential data under the engagement's NDA, but do not call a surface-level demonstration an independent review.

5. Record the limits of the finding. State what was tested, what was excluded, which evidence came from the builder and which claims could not be verified. If access was restricted or a result was redacted, preserve that fact beside the conclusion.

6. Send findings to an accountable decision-maker. The person who can delay a launch, narrow permissions or require remediation must receive the unsoftened result. Assign every finding an owner and due date, then retest the changed system.

7. Repeat the evaluation after material change. A new model, connector, data source, permission, prompt or approval rule can change behavior. Treat those changes as reasons to rerun the relevant acceptance tests, not as routine configuration.

The bottom line

An AI system is not ready because its creators can explain why it should work. It is ready when someone else has enough access to show where it does not.