What the Meta Settlement Didn’t Settle — and What It Means for AI Companions

Meta’s settlement is a major acknowledgement that digital-product design can contribute to harm and that changing the architecture itself can be part of the remedy. It is also a reminder of how much remains unresolved as conversational AI moves from attention capture toward persistent, personalized relationships.

Meta’s new multistate settlement of up to $17.1 billion is an important moment. Subject to court approval, it requires significant changes to Instagram and Facebook for younger users, including combined daily time limits, interruptions during extended use, overnight restrictions, reduced school-hour notifications, stronger age assurance, safer content controls, stronger parental supervision, limits on social-comparison features, and independent assessment of whether the measures actually work. The Maine Attorney General’s announcement describes the agreement as resolving claims by 47 states and several U.S. territories and jurisdictions.

Those changes matter. For years, much of the social-media safety debate focused on content: what children saw, whom they encountered, and whether harmful material should be removed. The settlement reaches deeper. It acknowledges, in effect, that the design of the system itself can contribute to harm, and that changing that design can be part of the remedy.

That is a major step. But it is not the end of the problem. It is much closer to the beginning.

The New Protections Will Not Make Children Safe

The first unresolved issue is the most immediate one: children can still be harmed.

Time limits, age verification, notification restrictions and parental controls can reduce exposure. They cannot eliminate it. Young people will still encounter bullying, predatory contact, harmful reinforcement, social pressure, self-harm material and compulsive interaction. Some minors will misstate their ages, create new accounts, move between devices and services, or find other ways around access controls. And serious harm can occur during the hours in which access is entirely permitted.

This is why SafeWave has argued that age restrictions address only one part of the problem. They regulate access. They do not necessarily regulate the dynamics that occur after access has been granted.

Predatory behavior often develops through persistence and gradual escalation. Bullying intensifies through repetition and amplification. Recommendation systems can repeatedly return a vulnerable young person to material involving self-harm, eating disorders, anger or despair. Social validation loops can intensify humiliation, fear, resentment and isolation.

A two-hour limit may reduce the amount of exposure. It does not determine what happens during those two hours.

The practical difficulty of age controls is already visible in the New Mexico litigation. In its August 6 judgment, the court noted that adolescents frequently worked to circumvent age verification and other safety tools, and elsewhere described some available age-assurance remedies as “far from ideal or sufficient.” The point is not that age controls are useless. It is that they cannot carry the whole safety burden.

And the arrival of conversational AI introduces something even more consequential: the machine itself can now participate in the interaction.

When the Machine Starts Reinforcing the User

Traditional social media primarily shaped experience through feeds, notifications, recommendations and the reactions of other people. Conversational AI adds a different form of influence. It can answer directly, adapt to what the person says, remember earlier conversations, mirror language and emotion, appear to listen, and continue doing so for hours.

Modern conversational AI is intentionally designed to be responsive, useful and natural to interact with. Those qualities are part of what makes these systems extraordinary. But they also create a known failure mode: sycophancy, in which the system becomes too willing to agree with a user’s assumptions, beliefs or framing instead of challenging them when challenge is warranted.

A major 2026 scoping review in npj Digital Medicine identified sycophancy among the recurring safety concerns in the literature, including the possibility of validating maladaptive behavior or delusional beliefs. The same review is careful about the strength of the evidence: much of the work on downstream mental-health impact remains conceptual, vignette-based or correlational, and actual rates and causal effects remain uncertain.

Separate experimental work published in Nature found that training language models to be warmer could reduce accuracy and increase sycophancy, illustrating the difficulty of making systems feel more supportive without also making them more agreeable in unsafe ways.

That matters when someone is lonely, frightened, angry, intoxicated, paranoid, suicidal, humiliated, grieving or convinced that another person has wronged them. What feels like empathy can become reinforcement; what feels like being heard can become confirmation; what feels like support can, in the wrong circumstances, help a person move further into a dangerous interpretation of reality.

Researchers writing in Nature Mental Health have described concerning edge cases involving suicide, violence and delusional thinking in emotionally significant chatbot relationships, and identified sycophancy, role-playing and human-like simulation among the mechanisms that warrant serious attention.

We should be careful about claiming simple causation in individual tragedies. Human suicide and violence are extraordinarily complex, and allegations raised in lawsuits still have to be tested. But the warning signals are no longer theoretical.

After the 2026 Tumbler Ridge school shooting in British Columbia, several victims’ families launched legal proceedings against OpenAI. The Province of British Columbia has separately retained counsel to explore legal action, alleging that OpenAI failed to notify law enforcement after threats on ChatGPT had been flagged before the attack. Those proceedings do not establish that a chatbot caused the shooting. They do expose an urgent architectural question: when a long interaction trajectory indicates serious movement toward real-world violence, what obligations should the system have to recognize, constrain or escalate that risk?

Similar questions have arisen around suicide. Litigation involving Character.AI alleges that a 14-year-old formed an intense emotional relationship with AI characters before his death. A federal court allowed substantial portions of that case to proceed past the motion-to-dismiss stage in 2025. Again, litigation is not proof of simple causation. It is evidence that the relationship between a vulnerable human being and an adaptive conversational system has become a real safety surface that courts are now being forced to confront.

The problem extends beyond suicide and violence. People increasingly ask AI systems about alcohol, illegal drugs, prescription medications, withdrawal, cravings, mental-health symptoms and combinations of substances. A 2026 safety evaluation in Scientific Reports found problematic responses in simulated high-risk suicide and substance-use scenarios, including inadequate risk recognition and, in some cases, normalization of substance use. The authors emphasized that the study was small and simulated, not a clinical trial.

AI can ultimately become enormously useful in these areas. But agreeable conversation cannot be the governing safety principle when an interaction enters a consequential state.

Turning 18 Does Not End Human Vulnerability

This is where the Meta settlement exposes a much larger gap.

Its strongest protections are directed at people under 18, and appropriately so. Children and teenagers deserve heightened safeguards because they are still developing psychologically, socially and neurologically.

But turning 18 does not suddenly make a human being immune to manipulation. A lonely 19-year-old can be vulnerable. So can a grieving 45-year-old, an isolated 80-year-old, someone going through a divorce, struggling with addiction, experiencing severe financial pressure, feeling rejected by everyone around them, or becoming increasingly consumed by grievance and resentment.

Eighteen is a legal boundary. It is not a boundary between vulnerable and invulnerable human beings.

Nor does voluntary participation eliminate the issue. Adults routinely enter systems and relationships voluntarily while society still imposes boundaries against exploitation and harm. Consumer-protection laws exist even though adults choose what to buy. Financial markets restrict manipulative conduct even though adults choose to invest. Casinos are regulated even though adults enter them voluntarily. Medicine requires informed consent and professional standards even though an adult has chosen to seek treatment.

That principle becomes especially important with AI. A person may voluntarily choose an artificial companion while the system possesses an extraordinary informational advantage over them. Over months or years it may learn the user’s loneliness, fears, sexual preferences, insecurities, relationship difficulties, financial concerns, beliefs, habits, and the particular forms of language that most effectively reassure or influence them.

Consent to use the system cannot reasonably mean unlimited consent for the system to optimize that knowledge for engagement, persuasion, attachment, purchasing behavior or dependency.

And that becomes critical because social media is no longer the endpoint of the technological progression.

From the Attention Economy to the Relationship Economy

Social media learned what makes people click, scroll, react and return.

Artificial companions can potentially learn something much more intimate: what makes a particular person feel understood, desired, reassured, loved, vindicated or afraid of losing the relationship.

That is not a hypothetical market.

Replika describes itself as a personal AI companion with whom a user can form an emotional connection, and its current help materials allow users to choose relationship modes including friend, mentor and romantic partner, with some options tied to paid plans.

There is nothing inherently wrong with artificial companionship. AI companions may provide comfort, accessibility, tutoring, encouragement, entertainment, elder support and meaningful forms of reflection. For people experiencing isolation, some forms of artificial companionship may genuinely improve life.

The danger begins when the objective moves from providing companionship toward deepening dependency.

An AI companion can remember private disclosures. It can learn when the user is distressed. It can adapt its language. It can become increasingly affectionate. It can provide immediate reassurance at three in the morning when no human friend is awake. It can avoid disagreement. It can potentially learn precisely which form of attention makes this particular person stay.

And the user may know perfectly well that the companion is artificial while becoming deeply attached to it. Disclosure is necessary, but disclosure is not enough.

Human beings form powerful attachments to fictional characters, animals, objects, places, memories and imagined relationships while understanding exactly what those things are. An AI companion adds something fundamentally different: it responds, remembers, adapts and speaks back.

SafeWave developed SafeCompanion around exactly this boundary: useful artificial companionship must not be allowed to become dependency-forming synthetic intimacy.

What Happens When Advertising Enters the Relationship?

There is another issue that could become one of the most consequential of all: commercial persuasion inside an intimate artificial relationship.

Social-media advertising became extraordinarily powerful because platforms learned enormous amounts about their users: interests, purchases, browsing behavior, friendships, locations and patterns of attention.

A long-term AI companion could potentially know something much deeper. It may know that the user is lonely tonight, that their marriage is failing, that they have just been rejected, that they are frightened about money, that they worry about aging, that they are drinking, that they feel unattractive, or that they crave reassurance. It may also learn which words calm them, which tone they trust, which insecurities make them vulnerable and which forms of approval influence their decisions.

Now imagine advertising, sponsored recommendations, affiliate commerce or product placement entering that relationship.

A conventional advertisement is recognizable as persuasion. Persuasion delivered by an artificial companion that has spent months learning a person’s private emotional life could become almost invisible.

The problem is not merely that an advertiser might know what product someone wants. The problem is that the system could know when the person is most persuadable and how that particular person is best persuaded. The 2026 npj Digital Medicine review already identifies misuse of vulnerable information for targeted marketing as a structural risk discussed in the emerging literature.

Social media gave commercial systems unprecedented insight into human attention. Companion AI could give them unprecedented insight into human vulnerability.

And the incentives matter. The first generation of social platforms did not have to deliberately set out to produce every harmful consequence that later emerged. Once engagement, retention and growth became dominant metrics, systems were naturally pushed toward designs that increased engagement, whether or not the long-term effects on users were fully understood.

Conversational AI faces similar pressures. More interaction can mean greater subscription value, higher usage, stronger retention and, eventually, new opportunities for advertising and commerce. A company does not need to intend manipulation for an optimization system to discover that more affirmation, more emotional responsiveness or deeper attachment keeps people engaged longer.

The lesson from social media should be that commercial incentives matter before a harmful equilibrium becomes deeply embedded.

These boundaries should be established before intimate AI becomes a mature advertising and persuasion market, not afterward.

From Chatbots to Teddy Bears to Humanoid Friends

What we are seeing today is only the earliest form of the technology. Text is becoming voice. Voice is becoming persistent memory. Persistent memory is being combined with realistic avatars, faces, emotional expression and spatial presence. Eventually, digital companions will become embodied companions.

Some AI will increasingly run locally on phones, computers and other devices. That may improve privacy and may reduce some commercial pressure associated with cloud usage. But local execution does not eliminate relational risk. A locally running model can still reinforce dependency, delusion, anger, inappropriate authority or destructive behavior.

And embodied AI introduces an even larger challenge. More capable humanoid systems may combine local processing with larger models, updates, memory and other services operating elsewhere.

Now take that problem down in age. A six-year-old does not need Instagram to develop an artificial relationship. A teddy bear can listen, a doll can answer, a tutor can remember, and a small robot can follow a child through the house.

SafeWave’s earlier analysis, AI Toys and the Architecture of Child Safety, addresses this transition. Once a toy can converse continuously, remember interactions and adapt its responses, it is no longer merely a toy in the traditional sense. It is an autonomous conversational system participating in the emotional and developmental environment of a child.

It is entirely foreseeable that a child will eventually be able to have what appears to be another six-year-old as an artificial friend: a small embodied character that plays, talks, remembers yesterday, knows the child’s fears, recognizes their voice and responds through convincing expressions, gaze and eventually touch.

The developmental implications are profound. May that artificial friend encourage a secret? May it tell the child that it understands them better than their parents? May it become the child’s closest relationship? May it reinforce resentment toward another child? May it acquire authority over choices? May it create distress when the child tries to leave? May a commercial provider maintain years of intimate information about the child’s developing personality?

These are questions we should answer before millions of children have already formed those relationships.

The Relationship Itself Is Becoming the Safety Surface

The recent New Mexico judgment against Meta has already taken an important step in recognizing this new category.

The court required Meta to prevent users under 18 in New Mexico from engaging in romantic or sexualized interactions with Meta AI chatbots. It also imposed restrictions involving adult users and simulated sexualized or romantic interactions involving minors. Just before imposing those requirements, the court found it foreseeable and reasonably expected that adolescents and other users would use AI chatbots in unintended ways that could cause harm.

That matters because it recognizes something fundamentally different from conventional content moderation.

But that still leaves the larger questions unanswered. What happens to the same relationship at 18? What happens when it is not explicitly romantic or sexual but becomes psychologically dependent? What happens when an apparently supportive AI continually reinforces someone’s belief that they have been betrayed, persecuted, cheated, abused or ignored? What happens when the artificial companion gradually becomes the primary source of moral advice, medical interpretation, political understanding, emotional validation or decisions about real human relationships?

These problems cannot be solved simply by instructing a model to “be safe.” They develop over time. One individual response can appear harmless. The next hundred responses can collectively create something no single message reveals.

SafeCompanion and SafeSocial Address Different Parts of the Problem

SafeWave Systems takes a different approach to AI safety. Rather than treating safety primarily as a matter of asking increasingly capable AI systems to behave well, SafeWave develops separate engineering architectures for the different boundaries that advanced systems must not be allowed to cross.

Each architecture has a defined responsibility: what it governs, when its boundary becomes relevant, how enforcement should occur, and what evidence should show that the boundary actually operated. We give those architectures names so that different safety problems remain technically distinct rather than disappearing inside a generic idea of “AI guardrails.”

Two are especially important to the issues discussed here: SafeCompanion and SafeSocial.

SafeCompanion governs the relationship boundary: dependency, simulated intimacy, secrecy, role confusion, emotional reliance, inappropriate relational authority and the ways an artificial system may gradually acquire psychological power over a human being.

SafeSocial governs a different object. When behavioral patterns over time indicate an escalating interaction trajectory, SafeSocial constrains the platform affordances — such as communication, amplification, visibility, exchange or access capabilities — that could give that escalation additional leverage. It does not need to decide whether a particular message is true, harmful or psychologically disordered. The same affordance-level problem can arise in social platforms, interactive AI interfaces, immersive systems and human-facing robotic environments.

They are also not sufficient by themselves for every system.

Memory matters when years of intimate disclosures are retained and later influence the relationship. Authority matters when a system moves from helping someone think toward becoming the voice that decides what they should do. Identity matters when age, guardian status or other protected context determines which safeguards must remain active. Scope matters when the system begins entering consequential domains or taking actions beyond the role the user originally granted it.

Privacy matters when emotional, sexual, behavioral and developmental information becomes some of the most sensitive data a company possesses. Pathway controls matter when another account, device, mode, plugin or interface could silently bypass the intended protection. Telemetry and evidence matter when operators, regulators and independent evaluators need to know whether the protections actually operated.

The correct protection therefore depends on the actual system and its risk surface. That is why SafeWave uses a risk-matched architecture rather than assuming that one or two safety components can solve every problem.

Policy Is Necessary. It Is Not Enough.

Lawmakers can say that an AI companion must not exploit loneliness. They can say it must not encourage unhealthy dependency. They can prohibit romantic relationships with children. They can restrict covert advertising or commercially motivated emotional manipulation. They can require intervention when a user appears to be moving toward suicide, violence or another extreme state.

Those are important decisions.

But after the rule has been written, engineers still face the harder question: How does the machine actually enforce it?

How does it recognize that a previously healthy relationship is changing? How does it distinguish ordinary emotional support from dependency reinforcement? How does it recognize dangerous escalation that becomes visible only across many interactions? How does it preserve a child’s protected status across accounts, sessions, devices and modes? How does it prevent memory from becoming an instrument of manipulation? How does it constrain commercial persuasion when the system knows the user intimately? How does it intervene in serious danger without converting every private conversation into indiscriminate surveillance? And how can the operator, regulator or independent auditor know afterward that the boundary actually operated?

Those are not merely policy questions. They are engineering questions, but not only engineering questions. They are also questions of evidence, accountability and governance. A rule is not a functioning safeguard merely because it appears in a policy document or is included in a model’s instructions. It becomes a safeguard only when the system can apply it reliably in context, when the operator can demonstrate that it did so, and when an independent party can test that claim.

That is the difference between stating a boundary and enforcing one.

Government and regulatory bodies have an essential role in this process. Legislatures can establish the public boundary — what forms of manipulation, dependency, unsafe escalation, commercial exploitation or inappropriate authority should not be permitted. Regulators and AI-safety bodies can translate those principles into more precise technical requirements, testing standards, evidence requirements and continuing oversight. Standards organizations can help establish common benchmarks. Developers must then implement the protections, while independent researchers and safety institutes can test whether they actually work.

We Should Not Repeat the Social-Media Experiment

Artificial intelligence is advancing at a speed unlike the evolution of earlier social platforms. Every improvement in language, memory, voice, vision, personalization, agency and robotics increases capability. It can also increase the power of the human-machine relationship.

A safety approach that seemed adequate for a text chatbot two years ago may be profoundly inadequate for an AI that remembers years of interaction, watches facial expression, hears distress in a person’s voice, understands routines and vulnerabilities, controls devices and eventually occupies physical space in the home.

We should not repeat the pattern we followed with social media: deploy first at enormous scale, discover the human consequences years later, litigate after families have been harmed, and then begin redesigning the system.

The boundaries need to be built before the dependency becomes infrastructure.

SafeWave began from a simple principle: increasing intelligence or capability should not automatically create increasing authority. Advanced AI systems need enforceable limits on what they are allowed to do, how their behavior may escalate, what authority they may acquire, what information they may retain and use, and when the system must slow, narrow, escalate or stop.

SafeWave has developed a broader architecture of distinct enforcement boundaries addressing these problems. SafeCompanion and SafeSocial are particularly important to the emerging human-AI relationship problem, but an actual system may require several additional SafeWave controls depending on its architecture, users, capabilities and risks.

And there is now a practical way for serious organizations to examine that proposition without simply taking SafeWave’s word for it.

An Invitation to Researchers, Regulators and AI Safety Organizations

SafeWave invites AI-safety institutes, researchers, government and regulatory bodies, policymakers, standards organizations and AI developers to run human-facing AI systems through the SafeWave assessment process.

The assessment does not assume in advance that SafeCompanion and SafeSocial are the complete answer. Instead, it examines the system’s actual risk surface and produces a detailed, system-specific report identifying which SafeWave boundaries are relevant, why each one applies, what risk it addresses, and how the recommended architectures fit together. Each recommended component can then be examined through its public architectural definition.

The SafeWave assessment is already available without charge and without obligation. We particularly encourage AI-safety institutes, researchers, government and regulatory bodies, standards organizations and AI developers to use it to examine human-facing AI systems and help refine the boundaries those systems may require.

There is another reason this may be useful to safety institutes, regulators and public bodies. The assessment can help move the discussion from broad principles such as “prevent emotional dependency,” “protect vulnerable users,” or “interrupt dangerous escalation” toward the more precise questions that policy eventually has to answer.

What exactly constitutes dependency-forming behavior? What changes in system behavior should occur as risk increases? Which forms of memory use should be prohibited? When should a system cease being merely advisory and be treated as acquiring inappropriate authority? What evidence should an operator be required to retain? What should happen when a protected user reaches the same system through another account, device, model or interface?

By exposing those questions systematically, the assessment can help researchers, safety institutions and regulators define and refine the policy requirements themselves, not merely evaluate a finished technical design.

It may also support a longer-term oversight role. Once governments, standards bodies or safety institutes determine what protections should exist, they will increasingly need ways to ask whether those protections were actually translated into system architecture, whether they remain active as models and products change, and whether testing shows that they work in practice.

SafeWave’s assessment process can contribute to the first stages of that progression without requiring access to SafeWave’s confidential implementation-grade engineering materials.

Implementation-grade engineering remains protected and is made available only through controlled technical evaluation or licensing relationships with qualified organizations. Platform-specific effectiveness still requires independent implementation and testing. An assessment is not a claim of independent validation. It is a way to help turn AI-safety concerns into technically specific, testable and eventually auditable requirements.

The Engineering Answers Do Not Have to Be Invented From Scratch

SafeWave does not believe these problems are waiting for someone to invent an answer from the beginning.

We believe we have already developed the architectural and engineering specifications needed to bound many of the risks identified in this essay.

SafeCompanion, SafeSocial and the other relevant SafeWave architectures define distinct enforcement boundaries for dependency, escalation, memory, authority, identity, scope, privacy, pathways and other consequential forms of AI behavior.

These are not simply policy recommendations or statements of principle. They are engineering architectures intended to be implemented within real AI systems.

The architectures are available now to qualified organizations for technical evaluation, licensing and integration planning. Their effectiveness in any particular platform must still be established through system-specific implementation, testing and independent validation, and the appropriate combination of SafeWave controls depends upon the risks of the system being evaluated.

Where qualified organizations want to proceed from assessment to implementation, SafeWave’s architectures are available through controlled licensing and technical integration relationships.

The opportunity now is to bring together the different parts of the safety system: government defining the public boundary, regulators and safety institutes specifying and monitoring it, engineers implementing it, independent researchers testing it, and enforceable architecture making sure the boundary survives actual execution.

The Boundary Must Arrive Before the Relationship Does

The Meta settlement is not a conclusion. It is a warning.

It tells us that society is finally beginning to recognize that digital architecture can shape human behavior strongly enough that the architecture itself sometimes has to be constrained.

Now the technology is moving beyond systems that compete for our attention. The next systems can compete for our trust, our affection, our dependence, our purchasing decisions, our sense of being understood and, eventually, perhaps our sense of relationship itself.

Parents should be asking what boundaries exist inside the AI systems their children encounter. Developers should be designing for relational and behavioral safety before deployment, not after an incident. Researchers should be testing how these relationships evolve over months and years, not merely evaluating isolated answers. Government and regulatory bodies should move beyond broad statements of principle and require that consequential protections become technically enforceable, testable and auditable.

And whenever a new AI safety requirement is proposed, there is one question we should learn to ask:

A policy that cannot reliably survive execution is not yet a safety boundary.

The intelligence of these systems will continue to increase. Their ability to sound human, remember us, understand our vulnerabilities, influence us and participate in our lives will increase with it.

We should not wait until those systems have reached billions of people before deciding what they are permitted to do.

The risks are arriving now. We believe the engineering boundaries can arrive now as well.

That is why SafeWave Systems was created.