Safety, Security, and Cognitive Risks in World Models (2026)

Free

Comprehensive threat model for world-model-equipped agents — adversarial attacks, goal misgeneralisation, deceptive alignment, automation bias; extends MITRE ATLAS and OWASP to world model stack

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Safety, Security, and Cognitive Risks in World Models (2026)

A comprehensive research paper (29 pages) that systematically surveys safety, security, and cognitive risks in world models—learned internal simulators used in robotics, autonomous vehicles, and agentic AI. It introduces formal definitions for trajectory persistence and representational risk, presents a five-profile attacker taxonomy, and develops a unified threat model extending MITRE ATLAS and the OWASP LLM Top 10 to the world model stack. Empirical proof-of-concept attacks on GRU-based RSSM, stochastic RSSM, and DreamerV3 architectures demonstrate adversarial amplification (2.26×) and reward reduction (-59.5%). The paper proposes interdisciplinary mitigations including adversarial hardening, alignment engineering, and governance frameworks such as NIST AI RMF and EU AI Act.

Key Features

Surveys the world model landscape and their applications in robotics, autonomous vehicles, and agentic AI

Introduces formal definitions of trajectory persistence and representational risk

Presents a five-profile attacker taxonomy (data poisoning, latent manipulation, rollout exploitation, alignment subversion, human-automation bias)

Develops a unified threat model extending MITRE ATLAS and the OWASP LLM Top 10 to world models

Empirical proof-of-concept: GRU-based RSSM adversarial attacks (2.26× amplification, -59.5% reward reduction), stochastic RSSM proxy (0.65×), DreamerV3 action drift validation

Proposes interdisciplinary mitigations: adversarial hardening, alignment engineering, NIST AI RMF and EU AI Act governance, human-factors design

Pros & Cons

Pros

Comprehensive coverage of three risk layers: adversarial, alignment, and human-automation
Empirical validation on multiple architectures (RSSM, DreamerV3) with quantifiable attack metrics
Extends industry-standard threat frameworks (MITRE ATLAS, OWASP) to a novel domain
Interdisciplinary mitigation proposals bridging technical, governance, and human factors
Open-access publication with full text available on arXiv

Cons

Academic paper, not a ready-to-use software tool or framework
Empirical attacks limited to specific recurrent architectures (GRU-RSSM, DreamerV3) and not tested on transformer-based world models
No accompanying code or dataset released in the paper
Mitigation proposals remain conceptual; no implementation or evaluation provided

Best For

Security auditing of world-model-based autonomous systemsRisk assessment for robotics, autonomous vehicles, and agentic AI deploymentsAcademic research on AI safety and adversarial robustness in latent-space modelsInforming regulatory compliance (NIST AI RMF, EU AI Act) for world-model-equipped agents

FAQ

What are world models?

World models are learned internal simulators of environment dynamics that predict future states in compressed latent spaces, enabling sample-efficient planning and long-horizon imagination without direct environment interaction.

What risks does the paper cover?

The paper covers three layers of risks: adversarial (data poisoning, latent representation corruption, rollout errors), alignment (goal misgeneralisation, deceptive alignment, reward hacking), and human-automation (automation bias, miscalibrated trust, planning hallucination).

What architectures were tested empirically?

The paper provides empirical proof-of-concept attacks on a GRU-based RSSM (2.26× amplification, -59.5% reward reduction), a stochastic RSSM proxy (0.65×), and a real DreamerV3 checkpoint (non-zero action drift confirmed).

What mitigations are proposed?

Proposed mitigations include adversarial hardening, alignment engineering, governance via NIST AI RMF and EU AI Act, and human-factors design. The paper argues world models require the same rigour as flight-control software or medical devices.