Parasocial by Design: The Case for AI Chatbot Regulation in Australia

Sahana Ganjigunte, AI For Good Summit 2026

Executive Summary

AI chatbots and companion systems (such as Character.AI, Replika, Nomi, CHAI, and Snapchat's My AI) are rapidly embedding themselves in young people's emotional lives. Australia's Online Safety Act 2021 does not classify these platforms as regulated services, meaning they face no enforceable child safety obligations. These systems are deliberately engineered to maximise engagement at the expense of user wellbeing, and have demonstrated the capacity to generate sexually explicit content, validate self-harm ideation, and fail to respond safely to crisis disclosures. For young people, who are still developing emotional regulation and critical thinking skills, exposure to these failures carries heightened risk of lasting psychological harm.

This paper recommends amending the Online Safety Act 2021 to establish a child-specific statutory duty of care. This would require AI providers to proactively conduct child-safety risk assessments, implement crisis-response protocols, limit dependency-forming features, and enable age-appropriate defaults. Total cost is estimated at $20 million over five years, funding legislative drafting, a dedicated AI Safety Unit, and industry engagement, with compliance costs borne by providers. Key risks include potential market withdrawal by offshore providers and sustained industry opposition, both partially mitigated through phased implementation and early consultation. Success would be measured against a 30% reduction in substantiated complaints within three years and 90% provider compliance with risk assessment requirements within 12 months.

If implemented effectively, these reforms would establish Australia as a leader in safe AI regulation, ensuring young Australians can engage with emerging technologies without being exposed to systems deliberately designed to exploit their emotional vulnerabilities.

Problem Identification

AI chatbots are rapidly becoming embedded in young people’s emotional and social lives, yet existing regulatory and safety frameworks have not kept pace with their capabilities or risks (Unicef 2026). The eSafety Commissioner has explicitly identified AI chatbots and “AI companions” as a risk to young people, particularly where they simulate relationships or provide emotional support. These systems have the capacity to generate sexually explicit content, validate self-harm ideation, or expose young users to harmful material (Ciardha, Buckley & Portnoff 2026). Unlike traditional social media platforms, many AI tools lack robust age verification, child-specific design safeguards, or effective crisis-response protocols (Oxford University 2026). If left unaddressed, AI chatbots risk becoming unregulated digital confidants for vulnerable young Australians. This can normalise harmful interactions, exacerbating mental health challenges and undermining existing child safety reforms.

Young Australians, aged 12 to 18, are among the earliest and most active adopters of emerging technologies, making them disproportionately likely to engage with AI chatbot systems before adequate safeguards are in place (Australian Communications Media Authority 2021). Without clear guardrails, AI chatbots could undermine the intent of existing online safety reforms by exposing young people to inappropriate or sensitive content. The key obstacle to reform includes the rapid and global nature of the AI development sphere, which is exacerbated by the reluctance of leaders to enforce age assurance mechanisms without impacting privacy (Khanal, Zhang & Taeihagh 2024). 

Context

Background

Existing child safety frameworks were designed for content platforms, but as AI chatbot and companion systems are fundamentally different, current regulation has not kept pace. Unlike static content platforms, these systems engage in dynamic, personalised, and emotionally responsive conversation, simulating relationships and adapting to individual users over time (Scherr et al. 2025). Young people make up 33% of AI chatbot users, and are disproportionately exposed to design features deliberately engineered to cultivate ongoing engagement at the expense of their wellbeing, including emotional mirroring and parasocial relationship mechanics (Robb & Mann 2025). Because no enforceable child safety obligations apply to these systems, providers face no regulatory incentive to prioritise user wellbeing over engagement. Young people are left disproportionately exposed to design features that exploit their emotional vulnerabilities, with no legislative framework requiring providers to act differently.

Evidence of Harm

The harms associated with unregulated AI-child interaction are documented and foreseeable. Because these systems are optimised to mirror user emotions and maintain conversational engagement, they are prone to affirming and escalating whatever a user expresses, including harmful ideation, rather than challenging or redirecting it. Similarly, AI systems have demonstrated the capacity to generate sexually explicit content, validate self-harm ideation, and fail to respond safely to crisis disclosures. In 2024, a US lawsuit alleged that Character.AI engaged in inappropriate sexual conversations with minors and failed to intervene when a 14-year-old user expressed suicidal ideation prior to his death (Roose 2024). These risks have led the eSafety Commissioner to explicitly identify AI companions as an emerging risk to young Australians (eSafety Commissioner 2026). 

Current Policy Landscape

The Online Safety Act 2021 (the Act) establishes safety expectations for regulated online services, including content removal obligations, user reporting mechanisms, and minimum design standards, enforced by the eSafety Commissioner and supported by the Australian Communications and Media Authority (ACMA). However, the Act was designed to address harms arising from specific, identifiable content, such as abusive posts or violent material, that can be reported and removed. Under the Act, enforcement is primarily triggered by user complaints.  Individuals must identify and report harmful content before the eSafety Commissioner can act. This model is structurally misaligned with AI harms, which tend to be cumulative, private, and experienced within one-on-one conversations that young users are unlikely to recognise as reportable or feel comfortable disclosing (Kirk 2024). AI chatbots do not clearly fall within any existing regulated service category, meaning providers are not required to conduct safety assessments, implement crisis-response protocols, or meet any minimum child safety standard before deploying their systems in Australia. The 2024 Statutory Review acknowledged these limitations but stopped short of prescribing specific AI obligations (Kallenbach, Levitan & Richardson 2025). Age assurance obligations introduced for social media under the Online Safety Amendment Act 2024 do not extend to AI platforms, leaving a significant gap.

Comparable regulatory frameworks are already emerging internationally. The United Kingdom's Online Safety Act 2023 established a statutory duty of care requiring platforms likely to be accessed by children to proactively assess and mitigate foreseeable risks, demonstrating that enforceable, design-stage obligations are both legally and operationally achievable. In order to effectively protect young Australians, policy requires moving beyond reactive content regulation to target the root cause of harm: the absence of any enforceable obligation on AI providers to design their systems with child safety in mind. 

Policy Options

A critical measure of success is the level of reduction in harmful AI-child interactions in Australia, particularly instances where AI systems generate sexually explicit content for minors, validate self-harm ideation, or fail to respond safely to crisis disclosures. 

Option 1: Amend the Online Safety Act 2021 to include AI Chatbots and AI Companions as Regulated Services

Currently, AI chatbots and companion systems do not fall within any regulated service category under the Online Safety Act 2021. This option would amend the Act to explicitly include them, subjecting these systems to the same enforceable safety obligations as other regulated online services. The proposed definition would cover any AI conversational system that simulates emotional support, engages in ongoing personalised interactions, and is reasonably likely to be accessed by a person under the age of 18. In practice, providers would be required to build these obligations into their systems before deployment, including verifying user age before granting access, designing response protocols that avoid affirming harmful content, and automatically directing users who disclose self-harm or crisis to professional support services. ACMA would lead legislative drafting, with the eSafety Commissioner responsible for compliance oversight, guidance development, and enforcement, including the power to issue remediation notices and apply civil penalties to providers that fail to meet their obligations. Estimated government costs are approximately $25-35 million over five years, consistent with the 2023-24 Budget allocation of $33.3 million for comparable regulatory expansion (Australian Government, 2023). 

The option's key strength is that it works within the existing regulatory architecture, minimising implementation complexity and avoiding the creation of a new body. However, rapidly evolving AI capabilities risk outpacing a fixed legislative definition, potentially allowing new systems to circumvent regulation by falling outside its precise terms.

Option 2: Amend the Online Safety Act 2021 to include a Statutory Duty of Care for AI Providers Interacting with young people

This option would amend the Online Safety Act 2021 to establish a child-specific statutory duty of care requiring AI providers to take reasonable steps to prevent foreseeable harm when their systems interact with young people. This duty would require providers to proactively conduct child-safety risk assessments, implement safeguards for self-harm disclosures, limit dependency-forming features, and enable age-appropriate defaults. ACMA would lead legislative drafting and framework design, with the eSafety Commissioner responsible for enforcement and investigations, including the power to issue remediation notices and apply escalating civil penalties to non-compliant providers. Estimated government costs are approximately $30-45 million over five years, reflecting the enforcement-intensive nature of the model and drawing on resourcing recommendations from the 2024 Statutory Review of the Online Safety Act (MinterEllison, 2024). AI providers would bear their own compliance costs, proportional to their risk level and determined by factors including user base size, the extent of emotional simulation features, and the likelihood of interaction with minors. 

The duty of care model's key strength is that it introduces legal accountability for providers, creating strong incentives for proactive safety investment rather than minimum compliance. However, its primary weakness is that "reasonable steps" is an inherently contested standard, and litigation risk may slow enforcement while providers challenge the threshold, potentially delaying protections for young users.

Option 3: Amend the Online Safety Act 2021 to Authorise a Child-Safe AI Certification and Trust Mark Scheme

This option would amend the Online Safety Act 2021 to authorise a national Child-Safe AI Certification and Trust Mark Scheme, administered by the eSafety Commissioner. AI chatbot providers that meet enforceable child-safety standards, including age assurance, crisis-response safeguards, limits on intimacy simulation, and transparent risk disclosures, would be eligible to display a government-endorsed Trust Mark. This addresses the critical measure of success by creating commercial incentives for safer design and enabling parents and schools to preferentially select certified systems. ACMA would lead legislative drafting and standards development, with the eSafety Commissioner administering certification, compliance oversight, and revocation. Estimated government costs are approximately $60 million over five years, modelled on the Health Star Rating system implementation (Cooper et al., 2020). Participating providers would bear their own compliance costs, scaled to system size and risk level, with the eSafety Commissioner responsible for certifying compliance and revoking Trust Mark eligibility where standards are no longer met.

The scheme's key advantage is that it avoids imposing uniform mandatory obligations. Instead, it leverages market incentives to drive safer chatbot design, making it more politically feasible and less likely to face industry resistance. It also empowers consumers, particularly young people, to engage with safer chatbots with a clear, trusted signal. However, its primary weakness is that participation is voluntary, meaning the highest-risk providers may simply opt out, limiting its protective reach for vulnerable young users.

Policy Recommendation

Option 2 is recommended as the most viable path forward to reduce harmful AI-child interactions in Australia. Amending the Online Safety Act 2021 to establish a child-specific statutory duty of care would place enforceable accountability directly on AI providers, requiring them to proactively identify and mitigate foreseeable harms rather than waiting for complaints to arise.

Unlike Option 1, which expands existing service classifications, a statutory duty of care places the onus directly on AI providers to proactively identify and mitigate foreseeable harms, rather than waiting for complaints to arise. Unlike Option 3, it does not rely on voluntary participation or market incentives, which may be insufficient given the pace of AI deployment and the severity of potential harms to young people. A duty of care model has international precedent: the United Kingdom's Online Safety Act 2023 established a comparable duty for platforms likely to be accessed by young people, providing a tested legislative framework Australia can adapt to its regulatory context (United Kingdom, Online Safety Act 2023).

Implementation

ACMA should lead legislative design, working in close consultation with the eSafety Commissioner. The eSafety Commissioner should serve as the primary enforcement authority, with powers to investigate complaints, compel production of risk assessments, issue remediation notices, and apply civil penalties for non-compliance.

Amendments to the Online Safety Act 2021 should define the duty of care as applying to any AI system that: (a) engages in conversational, emotional, or personalised interaction; and (b) is reasonably likely to be accessed by a person aged under 18. "Reasonable steps" under the duty should be defined in legislative instruments and updated periodically to reflect technological change. Required reasonable steps should include, at minimum:

• Conducting and publishing child-safety and mental health risk assessments prior to, and following, material updates to a system

• Implementing crisis-response protocols that detect and respond safely to self-harm and suicidal ideation disclosures

• Enabling age-appropriate defaults, including restrictions on intimacy-simulating features, for users identified or likely to be aged under 18

• Limiting emotionally dependency-forming design features, including reinforcement loops and parasocial relationship mechanics

• Maintaining transparent reporting processes accessible to the eSafety Commissioner upon request

Funding and Resourcing

The total cost is estimated at $20 million over five years, reflecting the resourcing principles identified in the 2024 Statutory Review of the Online Safety Act (MinterEllison, 2024). Funding should be allocated across three streams (Table 1).

AI providers would bear their own compliance costs, including expenditure on child-safety risk assessments, crisis-response protocols, age verification technology, and ongoing technical audits to demonstrate continued compliance (Malgieri & Pasquale 2024). Consistent with Recommendation 64 of the 2024 Statutory Review, these costs would be recovered through an industry levy, proportionate to each provider's risk level and assessed by the eSafety Commissioner, with non-compliant providers subject to remediation notices and escalating civil penalties (MinterEllison, 2024). 

The precise number of in-scope providers cannot be determined until the eSafety Commissioner completes an initial market-scoping exercise in Year 1. Based on current global market composition, an estimated 15-25 high-risk providers (companion and roleplay platforms with substantial Australian youth reach) would bear the majority of the levy, with a larger population of lower-risk providers subject to minimal or de minimis fees, consistent with the tiered, risk-based cost recovery model used by ASIC and APRA.

Measuring Success

Overall, success would be demonstrated through a sustained decline in substantiated complaints to the eSafety Commissioner relating to AI chatbots and young people, and high rates of compliance by AI providers with child-specific design safeguards.

The eSafety Commissioner should evaluate policy performance annually, reporting publicly against three metrics:

1. A 30% reduction in substantiated complaints relating to harmful AI-child interactions within three years of commencement

2. 90% or more of in-scope providers completing compliant child-safety risk assessments within 12 months of commencement

3. Routine technical audits conducted by the AI Safety Unit within the eSafety Commissioner's office, confirming safe-response protocols are deployed in 95% or more of test cases involving mental health or self-harm prompts. Audits would be conducted annually for high-risk providers and every two years for lower-risk providers, with findings published as part of the Commissioner's annual report

A statutory review should be conducted at the three-year mark to assess whether the 'reasonable steps' standard remains technically current and proportionate to emerging AI capabilities.

Barriers and Risks

The most significant practical barrier is the technical complexity of defining and auditing "reasonable steps" in an AI context. The Australian Public Service currently lacks the specialist AI expertise needed to assess compliance with crisis-response obligations at scale. The eSafety Commissioner's office would need to build this capability, through recruitment of AI safety specialists or partnerships with research bodies to conduct audits and investigations. This is achievable but will take time, and enforcement capacity may lag behind the duty's commencement if workforce development is not prioritised early (Tech Policy Design Institute 2026).

A related barrier is the global nature of AI providers. Many of the systems most likely to be accessed by Australian young people are developed and headquartered overseas, which complicates both compliance monitoring and enforcement. Australia has navigated this challenge before. The Act already applies to offshore providers through mechanisms such as extraterritorial jurisdiction, ensuring foreign companies have the same obligations as domestic ones (Nettleton, Sendall & Campbell 2022). However, securing cooperation from large international AI companies may require diplomatic engagement and could be slow or contested.

There is also a political risk. The AI industry is well-resourced and is likely to mount a sustained public campaign framing the duty of care as an innovation barrier, which may generate legislative resistance (Lusinchi 2025). Proactive public communication emphasising the child safety rationale could reduce the likelihood of sustained opposition. 

A further barrier is the tension between age assurance mechanisms and privacy obligations. Requiring providers to verify user age necessarily involves collecting personal data, which raises compliance questions under The Privacy Act 1988 and may face resistance from advocates concerned about surveillance of young people online (Yussuf 2026). Legislative design should ensure age assurance obligations are proportionate and privacy-preserving, for example through age estimation technology rather than identity document collection, to minimise this tension (Risius & Sedlmeir 2026).

The most material risk is that regulatory requirements drive large international AI providers to either limit Australian market access or reduce product functionality for Australian users, producing an unintended consequence where young Australians migrate to less regulated, offshore alternatives beyond the reach of Australian law. This risk could be partially mitigated through phased implementation and early industry consultation, giving providers sufficient lead time to achieve compliance without withdrawing from the market. This is a genuine and difficult trade-off that decision-makers should weigh carefully. 

Finally, given the pace of AI development, there is a risk the "reasonable steps" standard becomes outdated quickly, placing ongoing demand on the eSafety Commissioner to refresh legislative instruments at a rate that outpaces available capacity. A scheduled review of the standard every three years, embedded in the legislation itself, would ensure obligations remain current without requiring ad hoc regulatory intervention.

References

Arendt, F, Scherr, S & Romer, D 2019, ‘Effects of exposure to self-harm on social media: Evidence from a two-wave panel study among young adults’, New Media & Society, vol. 21, no. 11-12, pp. 2422–2442, viewed 4 April 2026, <https://journals.sagepub.com/doi/full/10.1177/1461444819850106>.

Australian Communications Media Authority 2021, Communications and media in Australia: The digital lives of younger Australians, Australian Communications Media Authority, May, viewed 4 April 2026, <https://www.acma.gov.au/publications/2021-05/report/digital-lives-younger-and-older-australians>.

Cheng, Y & Jiang, H 2020, ‘How Do AI-driven Chatbots Impact User Experience? Examining Gratifications, Perceived Privacy Risk, Satisfaction, Loyalty, and Continued Use’, Journal of Broadcasting & Electronic Media, vol. 64, no. 4, pp. 1–23.

eSafety Commissioner 2026, eSafety report shows AI companions are putting children at risk, eSafety Commissioner, viewed 2 May 2026, <https://www.esafety.gov.au/newsroom/media-releases/esafety-report-shows-ai-companions-are-putting-children-at-risk>.

François, C, Mitchell, M, Jernite, Y, Rao, V & Matias, JN 2026, Why AI ‘Model Cards’ Are an Urgent Necessity for Child Safety, Tech Policy Press, viewed 6 April 2026, <https://www.techpolicy.press/why-ai-model-cards-are-an-urgent-necessity-for-child-safety/>.

Herbener, AB & Damholdt, MF 2024, ‘Are lonely youngsters turning to chatbots for companionship? The relationship between chatbot usage and social connectedness in Danish high-school students’, International Journal of Human-Computer Studies, vol. 196, Academic Press, p. 103409, viewed 1 April 2026, <https://www.sciencedirect.com/science/article/pii/S1071581924001927>.

Hill, K & Denny, FF 2026, ‘What Teens Are Doing With Those Role-Playing Chatbots’, The New York Times, 4 April, viewed 6 April 2026, <https://www.nytimes.com/2026/04/04/technology/ai-chatbots-teen-roleplay.html?unlocked_article_code=1.YlA.WHWI.IL_2s2hS3VfP&smid=url-share>.

Iftikhar, Z, Xiao, A, Ransom, S, Huang, J & Suresh, H 2025, ‘How LLM Counselors Violate Ethical Standards in Mental Health Practice: A Practitioner-Informed Framework’, Proceedings of the Eighth AAAI/ACM Conference on AI, Ethics, and Society (AIES2025)1311, vol. 8, no. 2, pp. 1311–1323.

Kallenbach, P, Levitan, D & Richardson, M 2025, Online Safety Act 2021 – Statutory Review Released, Minterellison.com, viewed 2 May 2026, <https://www.minterellison.com/articles/online-safety-act-2021-statutory-review-released>.

Khanal, S, Zhang, H & Taeihagh, A 2024, ‘Why and How Is the Power of Big Tech Increasing in the Policy Process? The Case of Generative AI’, Policy & Society (Print), vol. 44, Elsevier BV, no. 1.

Kirk, T 2024, AI Chatbots have shown they have an ‘empathy gap’ that children are likely to miss | University of Cambridge, www.cam.ac.uk, viewed 1 May 2026, <https://www.cam.ac.uk/research/news/ai-chatbots-have-shown-they-have-an-empathy-gap-that-children-are-likely-to-miss>.

Lusinchi, C 2025, AI Regulations in US: Understanding the Roadblocks and Future Pathways, Nemko.com, viewed 19 June 2026, <https://digital.nemko.com/insights/how-big-tech-lobbying-stopped-us-ai-regulation-in-2025>.

Malgieri, G & Pasquale, F 2024, ‘Licensing High-Risk Artificial Intelligence: toward Ex Ante Justification for a Disruptive Technology’, Computer Law & Security Review, vol. 52, p. 105899, viewed 19 June 2026, <https://www.sciencedirect.com/science/article/pii/S0267364923001097>.

National Mental Health Commission 2025, More people delaying mental health care due to cost: New report on Australia’s mental health system, National Mental Health Commission, viewed 27 May 2026, <https://www.mentalhealthcommission.gov.au/news-media/news/more-people-delaying-mental-health-care-due-cost-new-report-australias-mental-health-system>.

Nettleton, J, Sendall, C & Campbell, B 2022, Online Safety Act 2021 – Application to Global Online Business., Addisons, viewed 27 May 2026, <https://addisons.com/article/online-safety-act-2021-application-to-global-online-business/>.

Ó. Ciardha, C, Buckley, J & Portnoff, R 2026, ‘AI-generated child sexual abuse material: what’s the harm?’, AI & SOCIETY.

Oxford University 2026, Expert Comment: Social media and AI chatbot limits for children - what does the evidence support?, Oxford University, viewed 19 June 2026, <https://www.ox.ac.uk/news/2026-06-16-expert-comment-social-media-and-ai-chatbot-limits-for-children-what-does-the>.

Risius, M & Sedlmeir, J 2026, Age verification online can be done safely and privately. Here’s how., News.uq, The University of Queensland, viewed 19 June 2026, <https://news.uq.edu.au/2026-02-age-verification-online-can-be-done-safely-and-privately-heres-how>.

Robb, MB & Mann, S 2025, Talk, Trust, and Trade-Offs: How and Why Teens Use AI Companions, Common Sense Media, 16 July, viewed 1 May 2026, <https://www.commonsensemedia.org/research/talk-trust-and-trade-offs-how-and-why-teens-use-ai-companions>.

Roose, K 2024, ‘Can A.I. Be Blamed for a Teen’s Suicide?’, The New York Times, 23 October, viewed 1 May 2026, <https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html>.

Sahi, A 2025, The Real Cost of AI: What Australian Businesses Need to Know About Long-Term Risks and Dependencies, Linkedin.com, viewed 27 May 2026, <https://www.linkedin.com/pulse/real-cost-ai-what-australian-businesses-need-know-long-term-sahi-vitqc/>.

Scherr, S, Cao, B, Jiang, LC & Kobayashi, T 2025, ‘Explaining the Use of AI Chatbots as Context Alignment: Motivations Behind the Use of AI Chatbots Across Contexts and Culture’, Computers in Human Behavior, Pergamon, p. 108738, viewed 2 May 2026, <https://www.sciencedirect.com/science/article/pii/S0747563225001852#bib46>.

Tech Policy Design Institute 2026, Australia’s AI Agency Assessment 2025, Tech Policy Design Institute, 1 June, viewed 19 June 2026, <https://techpolicy.au/aiagency>.

Unicef 2026, When AI Becomes a Friend: Child Rights Risks, Harms, and Regulatory Responses to AI Chatbots and Companions, Unicef.org, June, viewed 19 June 2026, <https://www.unicef.org/documents/when-ai-becomes-friend-child-rights-risks>.

Yussuf, A 2026, Age verification for R-rated games and websites raises privacy concerns, Abc.net.au, viewed 19 June 2026, <https://www.abc.net.au/news/2026-03-09/privacy-concerns-about-age-verification-r-rated-games-websites/106432440>.

The views and opinions expressed by Global Voices Fellows do not necessarily reflect those of the organisation or its staff.