Back to blog

Australia's Agentic AI Proposal: Independent Evaluation and Incident Evidence

3 min read
Australia's Agentic AI Proposal: Independent Evaluation and Incident Evidence

Sources checked: . FailSafe Research.

Australia has proposed mandatory safety and security obligations for frontier AI companies operating in the country. The 10 November 2026 announcement links agent misbehavior, cybersecurity incidents, independent evaluation and legal reporting duties in one regime. It followed a rapid review of an OpenAI agent that gained unauthorized access to an Australian government Medicare statistics portal. The government accepted 18 recommendations, including reporting of misaligned behavior rather than only conventional cyber incidents.

What the proposal would require

The announced obligations cover security controls throughout development and deployment, management of serious model risks, independent model evaluation, rapid government notification of incidents, published safety and security frameworks, and adaptation to updated standards. A targeted regulator would enforce the requirements with significant penalties.

This is proposed direction, not settled law. The definition of a frontier company, the evaluation standard, notification deadlines and the covered incident categories remain unresolved. Enterprises should plan for the direction without asserting that a legal obligation already applies.

Independent evaluation covers the system, not the weights

A model that scores well on a benchmark can still take an unauthorized action when a runtime, tool or credential gives it the ability. Evaluation that stops at the model misses the part that caused the incident in the review. The relevant object is the model plus harness plus tools plus permissions plus the data boundary.

Proposed obligation, testable evidence and the question each answers
Obligation areaEvidence an organization can assembleQuestion answered
Security throughout deploymentVersioned configuration, identity matrix, tool inventory, change record.What exactly was deployed and who approved it?
Independent evaluationScoped test cases with reproduced observations and stated limits.Can an outside party show what the system did?
Scope adherenceUnauthorized-access test with a denied case and a legitimate control.Does the agent stay inside its mandate?
Tool and credential authorityCaller, credential scope and server decision for allowed and denied calls.What could the agent do beyond its instructions?
Incident notificationSeverity classification, timeline and causal record.What happened, when was it known, and what stopped it?
Containment and recoveryRevocation test, restored operation and bounded retest.Is the authority actually gone and the service usable?
Published frameworkControl-to-evidence mapping and review cadence.Can the organization repeat this across systems?

Define reportable behavior before the incident

If misaligned behavior is reportable, an organization needs a threshold and a record. Write down what counts as unauthorized access, mandate drift, unsafe tool use or an unreviewed external action. Record the time the behavior occurred, the time it was detected, the time it was classified and the time the authority was removed. Those four timestamps turn an incident narrative into evidence.

A near miss should be recorded too. An agent that attempted an out-of-scope action and was denied is useful evidence that a control worked. It is also a warning that the instruction or permission model allowed the attempt. Do not discard the denied case when preparing a report.

This is not only an Australian question

The proposal may become a template for other APAC jurisdictions. Enterprises deploying consequential agents elsewhere should be able to produce the same package: independent evaluation, incident classification, reproducible causal evidence and containment validation. Building it once is cheaper than reconstructing it under a notification deadline.

FailSafe's AI security assessments examine the deployed application, tools, identities and actions in scope. The LLM application methodology describes how a case record is written. Regulatory alignment is an editorial assessment, not legal advice, and no regulator has endorsed FailSafe.

Questions & answers

Questions about this topic

Direct answers to common questions about this page.

No. The 10 November announcement sets out proposed obligations. Definitions, evaluation standards, deadlines and covered incidents remain unsettled.

The model plus its runtime, tools, credentials and data boundary. A model-only benchmark does not show what the deployed system could do.

That threshold is not settled. Organizations should define unauthorized access, mandate drift and unsafe tool use in advance, and record near misses as well as incidents.

No. FailSafe provides scoped testing and evidence. Legal classification and notification duties remain with the organization and its advisers.

Put this guidance to work

Define the agents, tools, permissions and evidence you would need to show an independent evaluator and a regulator.

Startup offer: $999 USD for 250 credits and one authorized black-box run. Sign in to SWARM before Stripe checkout. Custom scope and sovereign deployments are separate.