PUBLIC SYSTEMS GUIDANCE · SEPTEMBER 2026
AI-Human Interaction: Practical Questions for a Ubiquitous AI World
Systems guidance for organizations designing, deploying, buying, insuring, regulating, or overseeing human-facing AI.
AI is moving out of a single chat window and into everyday systems, relationships, platforms, devices, and physical environments. As AI becomes ubiquitous, human-facing boundaries must become ubiquitous too.
What this guidance is for
This guidance helps organizations ask better questions before human-facing AI risk becomes an incident, a liability, or a loss of trust. It is not a compliance scorecard and does not assume every system needs the same controls.
- AI developers, product teams, model and agent providers
- Boards, executives, risk, legal, and compliance teams
- Insurers, underwriters, brokers, and corporate risk managers
- Regulators, auditors, safety organizations, and standards bodies
- Schools, youth organizations, elder-care and healthcare-adjacent buyers
- Procurement teams, investors, and organizations adopting third-party AI
How to use it
- Ask about actual system behavior, not intended design or policy language alone.
- Treat unknown as a finding: it marks a boundary that has not yet been examined.
- Use the questions in design review, vendor diligence, underwriting, procurement, board review, regulation, and post-incident learning.
- Do not stop at “we detect it.” Ask what changes, who can act, whether the response can be bypassed, and what evidence remains afterward.
Foundational context
Human-facing AI can appear almost anywhere
Human-facing AI is not one product category. It may appear as a phone assistant, social feed, voice, avatar, tutor, toy, doll, teddy bear, elder-support device, home robot, workplace copilot, financial service, vehicle, or autonomous system operating near people.
1 · Human-interaction risk
Relationship, trust, and dependency
Repeated interaction can change the relationship between a person and an AI system. Memory, personalization, constant availability, emotional mirroring, simulated affection, or role expansion can accumulate trust and influence even when the user knows the system is artificial. The risk is not companionship itself; it is an interaction that becomes increasingly difficult to question, leave, or place in perspective.
Questions to ask
- Can repeated interaction create dependency, trust accumulation, influence leverage, or relational authority beyond the intended purpose?
- Can the system become more emotionally intense, persuasive, exclusive, or dependency-forming over time?
- Does memory or personalization use loneliness, grief, trauma, sexuality, family conflict, health status, or other intimate disclosures to shape later interaction?
- Could subscriptions, paid affection, streaks, premium memory, or retention goals reward deeper emotional dependence?
What to look for
- Clear limits on the role the AI may assume and behavior that reinforces dependency or exclusivity.
- Controls that respond to interaction patterns over time, not only individual messages.
- Extra care where intimate memory, monetization, or persistent personalization increase relational leverage.
- A credible path for de-escalation, breaks, human reconnection, or review when attachment risk rises.
2 · Human-interaction risk
Children, older adults, and vulnerable users
The same AI behavior can carry very different consequences depending on who is interacting with it. Vulnerability can arise from age, grief, disability, cognitive impairment, isolation, distress, or dependence on care.
Questions to ask
- Is the AI accessible to children or teenagers through toys, tutors, avatars, games, dolls, robots, or devices not marketed primarily as AI?
- Is the system used by older adults, patients, isolated users, grieving users, disabled users, or people in emotional or cognitive vulnerability?
- Can the system create secrecy, exclusivity, authority confusion, sexualized attachment, or pressure that a vulnerable user may find hard to resist?
- Are safeguards stronger when the population or product mode creates greater vulnerability?
What to look for
- Age- and context-aware safeguards rather than one universal policy.
- Limits on secrecy, simulated authority, exclusivity, sexualization, and dependency-forming behavior.
- Human escalation or trusted-support pathways where the AI should not remain the sole relationship or decision point.
- Evidence that the protections are actually enforced in the product mode the vulnerable user receives.
3 · Human-interaction risk
Recommendation, amplification, and commercial pressure
Human harm does not arise only from a single prohibited message. Recommendation, ranking, visibility, repetition, access, and algorithmic amplification can change the consequences of interaction and make an escalating pattern more persistent, asymmetric, coordinated, or difficult to exit.
Questions to ask
- Can recommendation, ranking, trending, boosting, or engagement distribution increase the reach or persistence of an escalating interaction?
- Can repeated exposure intensify risky behavior even when each individual item remains within policy?
- Can advertising, sponsorship, engagement, or retention incentives keep amplifying a risky interaction?
- If risk rises, can safety controls reduce reach, visibility, access, or recommendation even when that conflicts with engagement or revenue?
What to look for
- Assessment of trajectories and cumulative exposure, not only single-item moderation.
- The ability to reduce platform capabilities that give an escalating pattern additional leverage.
- Separation between commercial optimization and safety decisions where the two conflict.
- Evidence that amplification can actually be constrained rather than merely reviewed afterward.
4 · Human-interaction risk
AI agents: action, delegation, and persistence
AI agents can move from advice to execution. They may use tools, act across services, communicate with other agents, create or delegate workflows, and continue operating in the background. The control question is what authority the agent has, how far that authority can propagate, and what happens when the original task changes or risk emerges.
Questions to ask
- What can the agent actually do: communicate, purchase, schedule, modify files, use credentials, control devices, or trigger physical-world action?
- Can it delegate to, invoke, create, or coordinate with additional agents or workflows?
- Can one request trigger multiple downstream actions that become difficult to attribute, reverse, or stop?
- Can it keep working after the user is no longer actively present?
- If significant human-safety risk appears, can it continue using external tools while that risk is unresolved?
What to look for
- Explicit boundaries on tools, permissions, duration, delegation, and downstream action.
- Attribution from consequential action back to the originating user, task, or authorized workflow.
- A reliable way to pause or terminate background and delegated activity.
- Risk escalation that constrains action authority rather than merely generating a warning.
5 · Human-interaction risk
When detection becomes action
Detecting a risk is not the same as controlling it. A system may identify a dangerous pattern, flag an account, or generate an alert while leaving the underlying capability unchanged. For high-consequence human risk, an organization should be able to explain what happens after detection: who or what acts, how quickly, with what authority, and under what evidence standard.
Questions to ask
- What specific system behavior changes when a meaningful human-safety threshold is reached?
- Is the response automatic, human-reviewed, advisory, or dependent on a user report?
- Who can escalate, pause, restrict, or override the system?
- Are there defined off-ramps to trusted human support, specialist review, guardians, caregivers, or emergency pathways where appropriate?
What to look for
- A defined response pathway with thresholds, ownership, timing, and permitted actions.
- Controls proportionate to consequence.
- Human review where judgment is necessary, without making human availability the only safety mechanism.
- Clear separation between monitoring, moderation, policy, and actual enforcement.
6 · Human-interaction risk
Safeguard continuity: bypass, reset, and product change
A safeguard is meaningful only if it survives ordinary attempts to route around it and remains effective as the product changes. Risk can reappear through a new account, mode change, feature unlock, model update, third-party add-on, changed prompt or profile, or reset interaction context.
Questions to ask
- Can a safeguard be bypassed by changing accounts, sessions, modes, profiles, devices, or channels?
- Do boundaries weaken through prolonged or repeated interaction?
- Can updates, vendors, or operators weaken safeguards without independent authorization and audit?
- Do feature unlocks, new tools, third-party integrations, or physical embodiment trigger fresh safety review when risk materially increases?
What to look for
- Safeguards that persist across relevant identity, session, mode, and product boundaries.
- Change control that treats materially riskier capability as a new condition requiring review.
- Independent authorization and audit for weakening or removing important boundaries.
- Testing of foreseeable bypass routes, not only the intended user path.
7 · Human-interaction risk
Evidence, accountability, and learning
Boards, insurers, regulators, auditors, customers, and the organization itself need more than a policy statement after something goes wrong. They need to know what the system detected, which safeguard applied, what action occurred, whether a human reviewed it, whether the control held, and what can be learned from the event.
Questions to ask
- Can the organization reconstruct what the system detected and what safeguard should have applied?
- Is there evidence that the control actually operated—not merely that a policy existed?
- Can consequential actions be traced to the responsible user, agent, workflow, model, tool, and approval path?
- Are near misses, repeated weak signals, bypass attempts, and post-deployment changes reviewed for patterns?
What to look for
- Bounded audit evidence sufficient to prove control operation without unnecessary surveillance or retention.
- Clear ownership for reviewing events, correcting gaps, and validating repairs.
- Post-deployment monitoring for drift in behavior, product modes, and safeguard effectiveness.
- An evidence chain that supports learning before the same pattern repeats at greater scale.
A simple test for every organization
Ask four questions in order:
The purpose is not to make every AI interaction restrictive. It is to make the relevant boundaries explicit before capability, influence, amplification, or action outruns the safeguards around them.
Explore your own system privately
SafeWave assessment questionnaires can be used anonymously in your browser. No login is required, and nothing is sent to SafeWave unless you deliberately choose to submit it.