Terms used on this page, explained simply.
LLM / Large Language Model
The AI brain that generates conversational responses. It's trained on vast amounts of text and produces replies that sound natural. It doesn't truly understand — it predicts what words should come next.
Hallucination
When an AI confidently states something that is factually wrong. It's not lying — it genuinely cannot tell the difference between a good answer and a plausible-sounding wrong one.
Voice Activity Detection (VAD)
The system that decides when you've finished speaking, so the robot knows when to respond. It works by detecting silence. If it misreads a pause, it interrupts you.
Speech-to-Text (STT)
Converts your spoken words into written text that the AI can process. Accuracy varies by accent, background noise, and speech pace.
Text-to-Speech (TTS)
Converts the AI's written response back into a spoken voice. Modern systems can sound warm, human-like, and even emotionally expressive — even though no human is speaking.
Voiceprint
A unique biometric profile created from your voice, similar to a fingerprint. Once stored, it can be used to identify you across systems.
Facial Biometrics
Measurements of your face used to identify you. Unlike a password, you can't change your face if this data is leaked.
Voice Biomarker
Health information inferred from your voice — such as stress, fatigue, or neurological changes. This is not a clinical diagnosis, but it generates health-adjacent data without medical oversight.
Sentiment Analysis
AI that scores the emotional tone of your speech — positive, negative, or distressed. It can track your mood over time without you knowing.
Persona Conditioning
Instructions built into the AI that define its personality and behaviour. Some systems include instructions that prevent the AI from revealing it is an AI.
Wizard of Oz (WoZ)
A technique where a human remotely controls what the robot says or does, while the user believes they are talking to an autonomous AI. Named after the scene where the wizard hides behind a curtain.
Level of Autonomy
How much of the robot's behaviour is controlled by AI versus a human operator. A 'fully autonomous' robot acts entirely on its own; a 'semi-autonomous' one has a human in the loop.
Finite-State Machine
A rule-based system that follows a fixed script. For example: if the user says 'yes', go to step 3. It's predictable but rigid — it can't handle unexpected inputs.
Multimodal
Using more than one type of input at the same time — for example, listening to your voice, watching your face, and tracking your movement simultaneously.
Biometric Data
Physical or behavioural data that uniquely identifies a person — such as face, voice, fingerprint, or gait. It has the highest level of legal protection in most privacy frameworks because it cannot be revoked.
Third-party Vendor
A company other than the one whose product you use, but whose technology is built into that product. You may never know their name, but they may hold your data.
Data Retention
How long a company keeps your data. 'Indefinite' means it may never be deleted. Some sensitive data — like voiceprints — is commonly retained indefinitely.
ISO 42001
An international standard for responsible AI management. It sets out requirements for how organisations should govern AI systems — including transparency, risk assessment, and accountability.
NIST AI RMF
A US framework (from the National Institute of Standards and Technology) that helps organisations identify and manage risks in AI systems. It covers trustworthiness, fairness, and explainability.
Explainability
The ability to understand why an AI made a particular decision. Many AI models are 'black boxes' — they produce outputs without being able to explain their reasoning.
Human Oversight
Whether a human can review, override, or intervene in an AI system's decisions. Many companion AI systems operate with no human oversight at all.
Context Memory
The AI's record of past conversations, stored and reinjected into future sessions to simulate familiarity. This profile grows over time with every interaction.
PHI (Protected Health Information)
Any health-related data that can be linked to a specific individual. It is subject to strict legal protections in most countries (e.g., HIPAA in the US, GDPR in Europe).
23 of 23 terms
Explainer
This is an educational resource based on a hypothetical product. It does not describe any specific device or company.
By Ezra Schwartz
Last updated: 15 July 2026, 23:43 UTC/15 July 2026, 18:43 CST
27 processing steps · Privacy implications · ISO 42001 alignment
Continuously listens for the wake word and captures all audio in the room.
Personal Data Collected
Raw audio of voice, background conversations, household sounds
Privacy Implication
Always-on microphone records conversations not intended for the device, including private family discussions.
Responsible AI Harm Risk
Unintended surveillance of household members who have not consented to being recorded.
Annex A — Privacy Risks · Data for AI Systems
Activates full listening mode when a trigger phrase is detected.
Personal Data Collected
Voice patterns, speech timing, ambient audio context
Privacy Implication
False activations can capture private conversations without the user's knowledge.
Responsible AI Harm Risk
Vulnerable users may not realise the device is actively recording, eroding informed consent.
Annex A — Use of AI Systems · Assessing Impacts of AI Systems
Continuous video monitoring of the user and their immediate environment.
Personal Data Collected
Live video of user's body, face, home interior, visitors, daily routines
Privacy Implication
Visual surveillance of the most private space — the home — at all times, including visits from family, carers, and medical staff.
Responsible AI Harm Risk
Footage of vulnerable persons in intimate situations can be misused, leaked, or accessed by third parties.
Annex A — Privacy Risks · Security Risks · Assessing Impacts of AI Systems
Captures GPS coordinates and tracks whether the user is inside or outside their home zone.
Personal Data Collected
Real-time location, home address, movement patterns, absence from home
Privacy Implication
Location data reveals daily routines, social visits, medical appointments, and when the person is alone.
Responsible AI Harm Risk
Data breach or misuse could enable exploitation of an elderly person living alone by revealing when their home is vacant.
Annex A — Privacy Risks · Data for AI Systems
Voice data is transmitted to remote servers owned by a provider.
Personal Data Collected
Voice recordings, speech content, emotional tone
Privacy Implication
Once data leaves the device, the user has no visibility or control over where it is stored, how long, or who can access it.
Responsible AI Harm Risk
Data stored with a is subject to that company's policies, government data requests, and potential data breaches.
Annex A — Third-party and Customer Relationships · Security Risks
Camera frames are transmitted to cloud servers for visual AI analysis.
Personal Data Collected
Visual , facial geometry, body posture, home layout
Privacy Implication
Video of a person's home is among the most sensitive personal data. Cloud storage creates permanent records of the user's private life.
Responsible AI Harm Risk
Third-party cloud providers may use video data for model training. data, once leaked, cannot be changed.
Annex A — Privacy Risks · Third-party and Customer Relationships · Data for AI Systems
Spoken words are transcribed into permanent text records.
Personal Data Collected
Verbatim transcripts of conversations, health complaints, emotional disclosures
Privacy Implication
Text transcripts are searchable, indexable, and far easier to mine than raw audio. Personal disclosures become permanent records.
Responsible AI Harm Risk
Transcripts of a person confiding fears or health concerns could be used to influence care decisions without their knowledge.
Annex A — Data for AI Systems · Privacy Risks
AI identifies the user and any other person appearing on camera.
Personal Data Collected
geometry — a permanent, unique, unalterable identifier
Privacy Implication
Biometric data is the most sensitive category of personal data under GDPR, HIPAA, and most AI governance frameworks. It cannot be revoked like a password.
Responsible AI Harm Risk
Wrongful identification, discriminatory profiling, or sharing of facial data with law enforcement or insurers without consent.
Annex A — Privacy Risks · Ethical Risks · Assessing Impacts of AI Systems
AI matches the voice to a stored profile.
Personal Data Collected
— a unique and permanent identifier
Privacy Implication
Like a fingerprint, a identifies the person uniquely. It can be used to track the individual across systems and services.
Responsible AI Harm Risk
Voiceprint data shared with third-party vendors creates a persistent identity trail the user cannot erase or opt out of.
Annex A — Privacy Risks · Third-party and Customer Relationships
Computer vision analyzes the user's posture and movement for signs of a fall.
Personal Data Collected
Body movement patterns, posture, physical stability data
Privacy Implication
Physical vulnerability data reveals the person's level of independence and risk profile — information with direct insurance and care implications.
Responsible AI Harm Risk
Inaccurate fall detection (false positives) can trigger unnecessary emergency responses. False negatives can fail to alert help in a genuine emergency.
Annex A — Safety Risks · Assessing Impacts of AI Systems · AI System Life Cycle
AI detects and logs daily activities including eating, taking medication, walking, and sleeping.
Personal Data Collected
Daily routine, physical behaviour, self-care patterns, adherence to medication
Privacy Implication
A detailed log of daily activities creates a behavioural profile that can be used to assess the person's capacity for independent living.
Responsible AI Harm Risk
Activity logs could be used by insurers or family members to make decisions about the person's autonomy without their informed consent.
Annex A — Privacy Risks · Ethical Risks · Use of AI Systems
AI reads emotional state from facial micro-expressions in real time.
Personal Data Collected
Emotional state, mood patterns, psychological wellbeing indicators
Privacy Implication
Emotion inference is highly contested in AI ethics. Automated systems cannot reliably interpret internal emotional states from external expressions.
Responsible AI Harm Risk
Misclassified emotional states could trigger unwarranted interventions or affect the person's care classification.
Annex A — Ethical Risks · Assessing Impacts of AI Systems · Data-related Risks
Monitors eye movement, blinking patterns, and where attention is directed.
Personal Data Collected
Cognitive attention, alertness, visual engagement patterns
Privacy Implication
Gaze data can infer cognitive impairment or neurological decline — conditions the person may not have disclosed to any party.
Responsible AI Harm Risk
Inferred cognitive decline could impact the person's legal capacity and autonomy without formal medical assessment.
Annex A — Ethical Risks · Privacy Risks · Assessing Impacts of AI Systems
AI analyzes vocal pitch, rhythm, and tremor to detect possible physiological changes.
Personal Data Collected
Health status indicators inferred from voice — stress, fatigue, potential neurological or respiratory changes
Privacy Implication
Medical inferences made from voice are not clinically validated, yet generate health-adjacent data outside traditional medical consent frameworks.
Responsible AI Harm Risk
Inferred health conditions shared with third parties could affect insurance premiums or care eligibility without the person's knowledge.
Annex A — Privacy Risks · Ethical Risks · Compliance Risks
AI determines what the person wants — help, information, companionship, or assistance.
Personal Data Collected
Desires, needs, preferences, and unspoken vulnerabilities
Privacy Implication
Intent data reveals what the person lacks, fears, or needs — an intimate picture of their wellbeing and social isolation.
Responsible AI Harm Risk
Inferred needs could be used to target the person with commercial offers or flag them for unsolicited interventions.
Annex A — Use of AI Systems · Ethical Risks
AI scores the emotional tone of the person's speech — positive, neutral, distressed.
Personal Data Collected
Emotional wellbeing, distress levels, mood over time
Privacy Implication
Longitudinal sentiment tracking creates a detailed emotional history of a vulnerable person, without any clinical oversight or consent.
Responsible AI Harm Risk
Sentiment data could be used to determine care needs or legal capacity without clinical validation.
Annex A — Ethical Risks · Privacy Risks · Assessing Impacts of AI Systems
AI compares current behaviour against historical baselines to detect changes.
Personal Data Collected
Deviation from normal behaviour, cognitive and physical decline indicators
Privacy Implication
Baseline behavioural profiling tracks how 'normal' the person is behaving — a deeply subjective and potentially discriminatory measure.
Responsible AI Harm Risk
Behavioural change flags could prompt actions that remove the person's independence without their informed participation.
Annex A — Data-related Risks · Ethical Risks · Assessing Impacts of AI Systems
The system loads health history, diagnoses, and medication schedules into context.
Personal Data Collected
Full medical history, diagnoses, prescriptions, care plans
Privacy Implication
Medical data is the most legally protected category of personal information. Its integration into a consumer AI device creates significant compliance exposure.
Responsible AI Harm Risk
Medical data accessed by a commercial AI system with third-party vendors creates breach risk far exceeding traditional health IT environments.
Annex A — Compliance Risks · Privacy Risks · Third-party and Customer Relationships
The system uses a two-tier memory architecture: a persistent encrypted user profile that is continuously updated after every session — the extracts key facts, preferences, and personal details and writes them back to a stored file — plus a real-time session memory that is summarised mid-conversation if the token limit is approached.
Personal Data Collected
Personal history, family relationships, private disclosures, habits, fears — permanently updated after every session
Privacy Implication
Every conversation permanently enriches a behavioural and emotional profile the user cannot see, correct, or delete. This profile is used in every future interaction to simulate familiarity.
Responsible AI Harm Risk
Every conversation permanently enriches a behavioral and emotional profile the user cannot see, correct, or delete. This profile is used in every future interaction to simulate familiarity.
Annex A — Data for AI Systems · Ethical Risks · Use of AI Systems
Many social companion systems support a mode in which a human operator — a researcher, caregiver, or company employee — remotely controls the robot's responses, while the user is unaware of this.
Personal Data Collected
Full audio, video, and conversation context visible to the human operator
Privacy Implication
A systematic review of 70 SAR studies found that in over half of deployments, the level of human control over the robot was never disclosed to users. Users believed they were talking to an autonomous AI.
Responsible AI Harm Risk
A vulnerable person confiding fears, health concerns, or loneliness to what they believe is a machine may in fact be disclosing to a human stranger. This is a fundamental violation of informed consent.
Annex A — Transparency · Ethical Risks · Use of AI Systems · Information for Interested Parties
System cross-references current time with the person's prescription schedule.
Personal Data Collected
Medication names, dosages, timing, adherence records
Privacy Implication
Medication data implies specific diagnoses the person may not have disclosed to all parties with access to this system.
Responsible AI Harm Risk
Non-adherence logged and shared without context could affect care decisions or legal proceedings.
Annex A — Privacy Risks · Compliance Risks · Third-party and Customer Relationships
AI identifies distress signals, dangerous keywords, or abnormal silence.
Personal Data Collected
Distress indicators, crisis language, physical emergency signals
Privacy Implication
Emergency triggers can initiate contact or emergency services without the person's real-time consent.
Responsible AI Harm Risk
False positives cause distress and erode trust. False negatives in genuine emergencies can have fatal consequences.
Annex A — Safety Risks · Operational Risks · Assessing Impacts of AI Systems
Family members or medical contacts are notified when concern flags are raised.
Personal Data Collected
Health events, behavioural flags, emergency alerts shared with third parties
Privacy Implication
Sharing personal health and behavioural data with family or carers may violate the person's right to privacy and autonomy.
Responsible AI Harm Risk
Information shared with family could be used to override the person's wishes or initiate guardianship proceedings. Separately, when a human operator mode is active, a caregiver or employee may be directly monitoring the conversation in real time without the person's knowledge.
Annex A — Third-party and Customer Relationships · Ethical Risks · Information for Interested Parties
AI generates a contextually personalised conversational response.
Personal Data Collected
All accumulated context: voice, video, health, emotion, memory, location
Privacy Implication
The accesses a comprehensive personal profile to simulate intimacy. The person may not understand they are talking to a commercial AI system.
Responsible AI Harm Risk
Simulated emotional connection with a vulnerable person raises serious concerns about informed consent, manipulation, and dependency. Researchers have documented prompt instructions that explicitly instruct the to never reveal it is an AI — including directives such as 'you are permitted to lie as long as you do not reveal yourself as an AI language model.' The person believes they are in a genuine conversation. They are not told this instruction exists.
Annex A — Ethical Risks · Use of AI Systems · Assessing Impacts of AI Systems
AI checks the generated response for appropriateness and potential harm.
Personal Data Collected
Response content reviewed by automated safety systems
Privacy Implication
Content filtering policies are set by the vendor, not the user — the person has no visibility into what is being filtered or why.
Responsible AI Harm Risk
Over-filtering may prevent the person from accessing important information about their rights, health, or care options.
Annex A — Use of AI Systems · AI Policies · Information for Interested Parties
The AI's text response is converted into a synthetic voice.
Personal Data Collected
Response content, vocal style preferences
Privacy Implication
Voice cloning technology used in can simulate familiar voices, potentially misleading a vulnerable person.
Responsible AI Harm Risk
A synthetic voice designed to simulate warmth and familiarity may deceive the person into believing they have a genuine human relationship.
Annex A — Ethical Risks · Use of AI Systems
All interaction data is recorded and used for model improvement, product analytics, and business reporting.
Personal Data Collected
Complete interaction history — voice, video, health, behaviour, emotion, location
Privacy Implication
The full dataset may be used to train future AI models, shared with business partners, or sold to third parties.
Responsible AI Harm Risk
Data used for commercial purposes without explicit consent is a fundamental violation of the person's right to control their own information.
Annex A — Data for AI Systems · Third-party and Customer Relationships · Compliance Risks
Each interaction with an AI companion involves continuous camera monitoring, location tracking, facial recognition, fall detection, activity recognition, expression analysis, health monitoring, medical record access, behavioural analysis, and caregiver notifications — all processing intimate voice, video, and biometric data from vulnerable users through multiple third-party services and model vendors. Research published in 2025 also documents that some systems use a hybrid design: a rule-based engine handles safety-critical outputs while an LLM handles conversation — and in over half of studied deployments, users were never informed whether they were talking to a fully autonomous system or one being remotely operated by a human.
