At 02:17, the ownership diagram failed

The mobile manipulator did not collide with the technician. It stopped eleven centimetres short, holding a crate at shoulder height while its drive controller reported a valid safe-stop state. Nothing was broken. Nobody was injured. Yet the morning review had no simple answer to the most important question: why had the robot entered an occupied aisle after the night supervisor restricted that zone?

The fleet service had issued the task. The site integrator had changed the map. A third-party perception model had been updated two days earlier. The owner had approved unattended operation, but only within a defined area. The maintenance contractor had replaced a lidar and completed its own calibration procedure. Every participant had performed a recognisable part of the work. No participant could reconstruct the whole decision.

This is the governance problem in embodied AI. A physical action can be produced by a chain of design assumptions, integration parameters, learned behaviour, live data, supervisory software and operating choices distributed across several organisations. Calling the robot “autonomous” does not concentrate responsibility in the machine. It conceals how widely the authority to shape its behaviour has been distributed.

The engineering answer is not to nominate one person who is responsible for everything. That creates a ceremonial owner with neither information nor control. The answer is to connect each material decision to a named authority, a bounded operating condition, a verified system configuration and surviving evidence. Responsibility remains human and organisational, but it becomes technically executable.

Responsibility follows control, knowledge and benefit

A useful allocation begins with three tests. Who can change the risk? Who possesses the information needed to recognise it? Who chooses to obtain the benefit from operation? The answers often point to different parties.

The robot developer controls intrinsic capabilities, default constraints, interfaces and disclosed limitations. The application integrator controls how the robot, tooling, environment and safety functions become one application. The operator controls live task initiation and local intervention. The owner or deployer decides where the system is used, which performance trade-offs are accepted and whether continued operation remains justified. Maintainers alter physical condition and calibration. Software, cloud and model providers can change behaviour after commissioning without touching the robot. Facilities teams control doors, traffic rules, radio coverage and floor conditions that the autonomy stack may silently assume.

These are not interchangeable roles. ISO 10218-1:2025 addresses industrial robots as partly completed machinery, while ISO 10218-2:2025 addresses complete industrial robot applications, including integration, commissioning, operation, maintenance and decommissioning. The split illustrates a broader point: responsibility changes when components become an application and changes again when the application enters a specific workplace. ISO 10218-2 does not cover public-access service robots or several other categories, so its scope must not be stretched into a universal legal answer. Its lifecycle separation remains instructive for governance design (ISO 10218-1:2025; ISO 10218-2:2025).

Responsibility therefore resembles a changing control surface, not a static handover. During design, the developer has the greatest leverage. At commissioning, the integrator does. During a live shift, the operator and site authority dominate. After a remote model release, influence moves back toward the provider and whoever authorises deployment. Figure 1 makes those transfers visible: knowledge and decision authority move at different rates, leaving gaps whenever one role can act without the evidence held by another. Good governance records the transitions instead of relying on contractual language written before the actual system existed.

Governance is not another safety function

Functional safety, cybersecurity, privacy, legal compliance and governance overlap, but none substitutes for the others. Functional safety asks whether faults in electrical or electronic functions can create unacceptable physical risk and how that risk is reduced. Cybersecurity asks whether unauthorised action can compromise confidentiality, integrity, availability or control. Privacy asks whether personal data is processed legitimately and proportionately. Compliance asks which binding obligations apply in a particular jurisdiction and role.

Governance decides who sets the objectives, operating limits and acceptance criteria; who may approve exceptions; who monitors performance; who can suspend operation; and who must act when evidence contradicts assumptions. A safe motion controller cannot decide whether a new use case was ethically or commercially authorised. A secure update channel cannot determine whether the new model was validated for the installed gripper. A privacy assessment cannot establish whether a degraded brake still supports the mission. Governance coordinates these disciplines without pretending to replace their specialist methods.

That distinction matters because a robot can be functionally correct and still be improperly governed. It may execute a permitted trajectory in an area where deployment was never approved. It may remain inside force limits while using a model whose licence, training provenance or application validation is unresolved. Conversely, a governance committee cannot vote away a violated stopping-distance requirement. Authority must operate inside technical and legal constraints.

ISO/IEC 42001:2023 frames an artificial-intelligence management system as interrelated organisational elements that establish policies, objectives and processes for responsible development, provision or use. NIST’s voluntary AI Risk Management Framework similarly spans design, development, use and evaluation and organises work around Govern, Map, Measure and Manage. Both place AI risk inside an ongoing management system rather than treating approval as a single gate (ISO/IEC 42001:2023; NIST AI RMF).

Authority must be executable

A responsibility matrix is useful only when the system can enforce it. “Operations approves deployment” is incomplete. Approval of what, identified how, for which site, under which conditions, until when, and with what revocation path?

The central artefact should be a deployment authority record. It binds a defined mission to a configuration identity, operating envelope, location or class of environment, supervision mode, authorised users, validity period and explicit constraints. The record is signed or otherwise attributable to the approving role. The robot or fleet controller checks it before accepting a mission and whenever a relevant state changes.

This turns governance into a chain of machine-checkable decisions. A task request expresses desired work. An action contract bounds what the autonomy layer may ask the physical system to do. Local control enforces dynamic limits. Safety functions remain independent where required. The deployment authority states whether that combination is permitted in the present context. Figure 2 shows the resulting authority stack: information moves downward, while permission narrows before it reaches the actuator. No single layer receives unlimited authority.

Escalation must be designed with the same precision. Conditions can include repeated perception uncertainty, loss of required connectivity, entry into a restricted zone, an unrecognised payload, a configuration mismatch, excessive human overrides or evidence that performance is drifting. Each condition needs a response class: continue with a bounded adaptation, request operator confirmation, enter a degraded mode, reach a minimum-risk condition, quarantine the unit or suspend the fleet.

The difficult decision is not whether an emergency stop exists. It is who may resume operation afterward. Recovery authority should normally be narrower than stop authority. Many people and automated monitors may be allowed to stop a robot; only a qualified role, supported by diagnostic evidence and a known configuration, should release it. Otherwise the strongest governance control becomes the button everyone learns to reset.

The configuration is part of the responsible system

A robot is not identified adequately by model and serial number. Its behaviour depends on firmware, autonomy software, machine-learning models, maps, calibration, safety parameters, tooling, payload assumptions, cloud services and site rules. Two physically identical units can therefore possess different operational capabilities and different risks.

Configuration identity should cover at least hardware revisions; firmware and software versions; model and parameter hashes; safety and motion limits; sensor calibration; maps and semantic zones; end effector; approved payload; dependency versions; active feature flags; and the deployment authority record. Cryptographic identity and signed artefacts improve attribution, but signatures alone prove origin and integrity, not fitness for use.

Every material change needs classification. A patch that changes only diagnostic formatting may follow a lighter path than a perception update that shifts detection confidence, a new grasp model that changes contact behaviour or a map update that removes an exclusion zone. The classification determines required review, regression tests, site validation, rollout size, monitoring period and rollback readiness. NIST’s Secure Software Development Framework provides a common set of secure-development practices and a vocabulary that purchasers and suppliers can use in acquisition and management. It does not validate a robot application, but it supports the discipline needed to produce and communicate trustworthy releases (NIST SP 800-218).

Remote update capability makes this governance unavoidable. The party technically able to deploy a model may not have the authority to change the approved application. A sound release path separates creation, verification, approval and deployment. It also retains the exact release artefact, validation evidence, approver identity, target population, deployment time and rollback decision. A fleet dashboard showing “version current” is operationally convenient and evidentially weak.

Human oversight needs time, information and power

Human oversight often degenerates into a comforting icon: a person beside a machine. Effective oversight is a control function with prerequisites. The person needs a comprehensible representation of system state, enough time to intervene, competence to interpret the evidence, authority to stop or override, and a safe mechanism through which the intervention takes effect.

The EU Artificial Intelligence Act makes this concrete for AI systems that fall within its relevant high-risk provisions. It requires oversight measures proportionate to risk, autonomy and context, and describes capabilities to understand limitations, monitor anomalies, avoid automation bias, interpret outputs, disregard or reverse them, and interrupt operation into a safe state. It also assigns deployers duties concerning competent and authorised overseers and, where under their control, retention of automatically generated logs. Whether a particular robot or function is covered depends on the Act’s classification rules and the surrounding product and use context; the provisions should not be presented as a universal robotics code (Regulation (EU) 2024/1689).

For a fast-moving robot, intervention latency is an architectural property. A remote supervisor watching twelve video streams over a variable network cannot reliably catch a 200-millisecond instability. The physical controller must handle immediate limits and local safety reactions. Human authority is more effective at slower boundaries: approving missions, resolving ambiguity, controlling exceptional states, suspending deployment, accepting updates and authorising restart.

The interface should reveal decisions, not merely telemetry. It should answer: what is the robot trying to achieve; which authority permits it; which constraints are active; what changed; what confidence is material; what will happen if nobody responds; and which interventions remain available? When the system cannot present those facts, the assigned human may carry nominal responsibility without practical control.

An incident begins before impact

The near miss at 02:17 became governable only when investigators correlated four clocks. The fleet service had dispatched the job under a map revision that the robot had not received. The robot reported its local map hash, but the orchestration log recorded only a friendly version name. The replaced lidar passed calibration, yet the maintenance system had no link to the deployed configuration. The supervisor’s zone restriction existed as a message in a shift channel, not as a machine-enforced constraint.

No component had lied. Each had told a locally consistent truth. The organisations had failed to preserve the causal chain connecting them.

Incident readiness begins with an evidence contract agreed before deployment. It defines the events, states and identities that must be captured; their time semantics; retention and access rules; integrity protection; privacy limits; and the conditions that trigger preservation. Useful records include task issuance, authority checks, model outputs relevant to action, constraint activations, operator interventions, safety events, command acceptance, actuator response, configuration changes and communication degradation. Raw sensor recording may be selective because bandwidth, privacy and storage impose real limits. Causal provenance matters more than indiscriminate accumulation.

The record also needs acquisition timestamps or a documented relationship between local monotonic counters and synchronized time. “Received at 02:17:04” is not equivalent to “observed at 02:17:04.” Without that distinction, investigators can reverse cause and effect. Figure 3 aligns the records behind one action and exposes the decisive discontinuity: a command derived from an old map meets a newer operational constraint only after both have already influenced different parts of the system.

Response authority must be assigned before the incident: who secures the area, who preserves evidence, who may access personal or commercially sensitive data, who informs affected parties, who coordinates suppliers, who decides whether one unit or the fleet is quarantined, and who authorises recovery. NIST SP 800-61 Rev. 3 treats incident response as part of cybersecurity risk management, connecting preparation, detection, response and recovery rather than isolating them in an emergency team. A robot incident may also involve safety, privacy, labour and product obligations, but the same organisational lesson holds: response capacity is built before the alarm (NIST SP 800-61 Rev. 3).

Retirement is an operational state

Robots rarely move cleanly from service to disposal. They are stored, resold, cannibalised, reassigned to less demanding work or kept alive after cloud support ends. Each path changes responsibility.

Retirement planning should define how authority is revoked, credentials and certificates are removed, personal and operational data are retained or erased, remote services are disconnected, hazardous energy is controlled, batteries are handled, evidence holds survive and reusable components lose stale trust relationships. A retired controller that still possesses fleet credentials is not waste; it is an unmanaged endpoint. A second-hand robot carrying the previous site’s maps and incident logs is not merely a privacy problem; it can reveal physical security and operational patterns.

The final configuration record should state why the unit left service and which components were transferred. Where a safety-relevant part is reused, its history and remaining assumptions should travel with it. Where provenance cannot be maintained, governance should treat the part as unqualified rather than allowing its physical compatibility to imply approval.

Lifecycle standards reinforce the point. ISO 10218-2:2025 explicitly includes decommissioning and disposal within the industrial robot application lifecycle. Governance that ends at commissioning abandons the system precisely when support, ownership and evidence become most fragmented.

Nine questions that make responsibility real

1. What work is approved? Define the intended mission, environment, payload, people exposed and foreseeable boundary conditions.
2. Which exact system is approved? Bind permission to hardware, software, models, calibration, tooling, maps and safety parameters.
3. Who may change each element? Separate technical access from organisational authority and require attributable approval.
4. Who can stop, degrade and restart? Make stop authority broad enough for protection and restart authority strict enough for evidence-based recovery.
5. What must remain local? Keep fast physical constraints and required safety reactions independent of remote services and human reaction time.
6. What can the overseer actually know? Present intention, active authority, constraints, confidence, anomalies and consequences in operational time.
7. Which changes invalidate approval? Classify modifications by behavioural and application impact, not by file size or supplier label.
8. What evidence will survive? Preserve causal events, configuration identity, timestamps, interventions and outcomes with proportionate privacy controls.
9. How does authority end? Define expiry, suspension, transfer, decommissioning, credential revocation, data disposition and residual support.

Together, the questions form a compact governance review, not a claim of legal compliance. Every answer must survive contact with the system. If an owner says a robot is approved only for warehouse aisle A, the control architecture should prevent acceptance of a task in aisle B. If a model provider says a release was not deployed, the installed model hash and signed release record should settle the matter. If oversight belongs to a shift supervisor, the supervisor must possess the information, interface and authority to exercise it.

The accountable robot is a shared construction

The investigation into the 02:17 near miss did not discover a rogue machine. It discovered an ungoverned gap between four valid local views. The fleet service believed the zone was open. The supervisor believed it was closed. The robot possessed an older map. The maintenance record described a sensor replacement but did not alter deployment status. Each system behaved consistently with its own evidence.

The repair was not a larger warning label. Zone restrictions became signed, time-bounded operational constraints. Fleet dispatch checked robot configuration before task assignment. Maintenance affecting perception forced a restricted state until application validation. The evidence contract connected task, map, model, calibration and operator intervention through stable identities. Restart required a named site authority.

No architecture can make responsibility effortless. Suppliers will still disagree about causes, operators will still face incomplete evidence, and laws will continue to vary by jurisdiction and application. Yet ambiguity can be reduced before harm occurs. The decisive move is to treat responsibility as a system property supported by authority boundaries, identities, state machines, logs and controlled updates.

A robot never becomes morally or legally responsible merely because its behaviour is complex. It acts through authority that people and organisations design, grant, supervise and revoke. The machine’s autonomy is therefore not the disappearance of responsibility. It is the test of whether responsibility was engineered well enough to survive the moment when the robot acts.