Sources checked: . FailSafe Research.
Australia has proposed mandatory safety and security obligations for frontier AI companies operating in the country. The 10 November 2026 announcement links agent misbehavior, cybersecurity incidents, independent evaluation and legal reporting duties in one regime. It followed a rapid review of an OpenAI agent that gained unauthorized access to an Australian government Medicare statistics portal. The government accepted 18 recommendations, including reporting of misaligned behavior rather than only conventional cyber incidents.
What the proposal would require
The announced obligations cover security controls throughout development and deployment, management of serious model risks, independent model evaluation, rapid government notification of incidents, published safety and security frameworks, and adaptation to updated standards. A targeted regulator would enforce the requirements with significant penalties.
This is proposed direction, not settled law. The definition of a frontier company, the evaluation standard, notification deadlines and the covered incident categories remain unresolved. Enterprises should plan for the direction without asserting that a legal obligation already applies.
Independent evaluation covers the system, not the weights
A model that scores well on a benchmark can still take an unauthorized action when a runtime, tool or credential gives it the ability. Evaluation that stops at the model misses the part that caused the incident in the review. The relevant object is the model plus harness plus tools plus permissions plus the data boundary.
| Obligation area | Evidence an organization can assemble | Question answered |
|---|---|---|
| Security throughout deployment | Versioned configuration, identity matrix, tool inventory, change record. | What exactly was deployed and who approved it? |
| Independent evaluation | Scoped test cases with reproduced observations and stated limits. | Can an outside party show what the system did? |
| Scope adherence | Unauthorized-access test with a denied case and a legitimate control. | Does the agent stay inside its mandate? |
| Tool and credential authority | Caller, credential scope and server decision for allowed and denied calls. | What could the agent do beyond its instructions? |
| Incident notification | Severity classification, timeline and causal record. | What happened, when was it known, and what stopped it? |
| Containment and recovery | Revocation test, restored operation and bounded retest. | Is the authority actually gone and the service usable? |
| Published framework | Control-to-evidence mapping and review cadence. | Can the organization repeat this across systems? |
Define reportable behavior before the incident
If misaligned behavior is reportable, an organization needs a threshold and a record. Write down what counts as unauthorized access, mandate drift, unsafe tool use or an unreviewed external action. Record the time the behavior occurred, the time it was detected, the time it was classified and the time the authority was removed. Those four timestamps turn an incident narrative into evidence.
A near miss should be recorded too. An agent that attempted an out-of-scope action and was denied is useful evidence that a control worked. It is also a warning that the instruction or permission model allowed the attempt. Do not discard the denied case when preparing a report.
This is not only an Australian question
The proposal may become a template for other APAC jurisdictions. Enterprises deploying consequential agents elsewhere should be able to produce the same package: independent evaluation, incident classification, reproducible causal evidence and containment validation. Building it once is cheaper than reconstructing it under a notification deadline.
FailSafe's AI security assessments examine the deployed application, tools, identities and actions in scope. The LLM application methodology describes how a case record is written. Regulatory alignment is an editorial assessment, not legal advice, and no regulator has endorsed FailSafe.
Questions & answers
Questions about this topic
Direct answers to common questions about this page.
No. The 10 November announcement sets out proposed obligations. Definitions, evaluation standards, deadlines and covered incidents remain unsettled.
The model plus its runtime, tools, credentials and data boundary. A model-only benchmark does not show what the deployed system could do.
That threshold is not settled. Organizations should define unauthorized access, mandate drift and unsafe tool use in advance, and record near misses as well as incidents.
No. FailSafe provides scoped testing and evidence. Legal classification and notification duties remain with the organization and its advisers.
Related Articles
Buying an AI Agent Pentest: Permissions, Tools, and Evidence
Scope an AI-agent assessment around authorized capabilities, tool permissions, data boundaries, human approval, and observable outcomes....
What should a board see when the vulnerability count keeps rising?
A practical security reporting model for verified attack paths, remediation coverage, unresolved risk and evidence quality....
FailSafe vs Upwind Red Agent: compare the evidence and deployment boundary
Compare FailSafe SWARM with Upwind’s published Red Agent proposition using workload scope, reproducibility, remediation and sovereign inference requirements....
Put this guidance to work
Define the agents, tools, permissions and evidence you would need to show an independent evaluator and a regulator.
Startup offer: $999 USD for 250 credits and one authorized black-box run. Sign in to SWARM before Stripe checkout. Custom scope and sovereign deployments are separate.