Garante per la protezione dei dati personali (Italy) - 10130115

Garante per la protezione dei dati personali - 10130115
Authority:	Garante per la protezione dei dati personali (Italy)
Jurisdiction:	Italy
Relevant Law:	Article 3(2)(a) GDPR Article 5(1)(a) GDPR Article 5(1)(c) GDPR Article 6 GDPR Article 12 GDPR Article 13 GDPR Article 24 GDPR Article 25(1) GDPR
Type:	Investigation
Outcome:	Violation Found
Started:
Decided:	14.05.2025
Published:
Fine:	5,000,000 EUR
Parties:	Luka Inc.
National Case Number/Name:	10130115
European Case Law Identifier:	n/a
Appeal:	Unknown
Original Language(s):	Italian
Original Source:	GPDP (in IT)
Initial Contributor:	cci

The DPA fined the provider of the “virtual companion” Replika AI €5,000,000 for unlawfully processing personal data, for transparency violations, and for failing to implement effective age verification mechanisms.

English Summary

Facts

US company Luka Inc. (the controller) made available Replika, a chatbot based on generative AI. Replika was meant to be a virtual companion that could help users track their mood, cope with stress, and work out their emotional and psychological problems. Replika could be configured to fulfil various roles, including therapist and romantic partner.

Replika gained attention from international media after it allegedly encouraged minors to engage in self-harm. The news prompted an ex officio investigation from the DPA.

In early 2023 the Italian DPA ordered the controller to halt the processing of personal data of all users in Italy, as a precautionary measure. In this early phase of the procedure, the DPA found evidence of possible GDPR violations and reserved the right to investigate further.

Months later the DPA lifted the ban on the condition that the controller took steps to ensure Replika’s compliance with data protection law, including implementing an effective age verification system to prevent minors from accessing the service. The controller made Replika available again after the ban was lifted.

In May 2025 the DPA issued a final decision and closed the procedure. The DPA reserved the right to further investigate certain aspects of the case in a different procedure.

Holding

The DPA found violations of Articles 5(1)(a), 5(1)(c), 6, 12, 13, 24, and 25(1) GDPR.

The DPA fined the controller €5,000,000. This is a notably high fine amounting to 2% of the controller’s global turnover - half the statutory maximum under the GDPR.

On the scope of the decision

In its decision, the DPA found violations of the principles of lawfulness and transparency based on the version of Replika’s privacy notice in force on February 2023 (the date of the DPA’s order to halt the processing).

The controller later made changes to its notice. The DPA reserved the right to open a second investigation on lawfulness, based on the controller’s up-to-date notice.

The DPA also found that the controller failed to implement age verification measures to prevent minors from using Replika. In this regard, the DPA assessed the measures in place both before and after the suspension order and concluded that both were insufficient.

On the competence of the DPA and the scope of the GDPR

The DPA held that the controller’s processing of personal data was within the scope of the GDPR, and that the Italian DPA was competent to enforce the GDPR with regards to the processing of personal data from users in Italy.

With regards to the applicability of the GDPR, the controller implicitly admitted that its service targeted users in the European Union (specifically, by stating that it stopped providing Replika to users in Italy, in response to the DPA’s early order to halt the processing of personal data). Therefore, the DPA held that the data processing fell within the scope of the GDPR.

With regards to the DPA’s competence, the controller claimed that it had a European establishment in the Netherlands. On this basis, the controller contested the Italian DPA’s competence and claimed that the Dutch DPA was the lead supervisory authority for the case.

However, the controller provided no evidence whatsoever of this alleged Dutch establishment. Additionally, the controller never mentioned an European establishment on Replika’s website or privacy notice.

For these reasons, the DPA rejected the controller’s arguments. The DPA held that the controller had no European establishment and that, therefore, the cooperation mechanism of the GDPR did not apply. On these grounds, the DPA considered itself competent in the case.

On lawfulness

The DPA examined the controller’s (now outdated) privacy notice. The notice did not clearly mention the legal bases for processing users’ personal data. Instead, it only contained generic and implicit hints at the legal bases.

Furthermore, the notice did not connect legal grounds to the purposes of each data processing operation. This violated the principle of granularity and made it impossible to assess whether each legal basis was invoked validly.

Finally, the privacy notice did not explicitly mention the training of the AI model as a purpose for the processing, and did not mention the legal basis for that processing of personal data for that purpose.

On these grounds, the DPA held that the processing of personal data was both unlawful and opaque, in violation of Articles 5(1)(a) and 6 GDPR.

The DPA noted that the controller's new privacy notice was more precise on legal bases. The DPA reserved the right to assess the lawfulness of the processing in a new investigation.

On transparency

The DPA found other violations of the principle of transparency in addition to unclear legal bases. The privacy notice in force in 2023:

• was only available in English;
• did not distinguish between the different purposes for processing personal data (especially with regards to “chatbot interaction” and “AI training”);
• did not clarify that Replika was meant for adult users only;
• did not mention for how long personal data would be stored;
• incorrectly stated that personal data might have been transferred to the US;
• incorrectly implied that users could be subject to automated decision-making covered by Article 22 GDPR (i.e.: automated decision making which "produces legal effects" concerning the data subject or "similarly significantly affects him or her").

For these reasons, the DPA held that the controller violated Article 5(1)(a) GDPR, 2, and 13 GDPR.

On age verification

Throughout the procedure the controller maintained that Replika was meant for adult use only. However, the DPA The DPA noted that the controller implemented no age verification system for Replika at the time of the temporary halt to the processing.

Interestingly, the DPA also found a violation of the data minimisation principle (Article 5(1)(c) GDPR). The lack of age verification resulted in the collection and processing of personal data from minors, even though the controller did intend to provide Replika to minors. So, the processing of personal data exceeded its stated purpose (i.e.: providing Replika to adult users).

The controller later implemented age verification mechanisms for Replika. In practice, users were asked to state their age and were refused access if under 18. However, the DPA found these mechanisms lacking. In particular:

• users were able to change their birth date in their profile after sign up. This would not prompt age verification from the controller’s website;
• the controller implemented a 24 hour “cooling off” period during which users who claimed to be minors, could not connect again. This prevented them from connecting Replika again and providing a different birth date. However, users could circumvent this “cooling off” period via incognito browsing.

On these grounds, the DPA held that the controller violated Articles 24 and 25(1) GDPR, and that the violation continued until the final decision.

Interestingly, the DPA also found a violation of the data minimization principle (Article 5(1)(c) GDPR). The lack of age verification resulted in the collection and processing of personal data from minors, even though the controller did intend to provide Replika to minors in the first place. So, in the DPA's view, the processing of personal data exceeded its stated purpose (i.e.: providing Replika to adult users).

Comment

The decision is available in both Italian and English on the DPA's website.

Further Resources

Share blogs or news articles here!

English Machine Translation of the Decision

The decision below is a machine translation of the Italian original. Please refer to the Italian original for more details.

SEE ALSO

Press release of May 19, 2025
Measure of June 22, 2023
Press release of February 3, 2023
Measure of February 2, 2023


- English version

[web doc. no. 10127930]

Measure of April 10, 2025

Register of measures
no. 232 of April 10



THE GUARANTOR FOR THE PROTECTION OF PERSONAL DATA

IN today's meeting, attended by Prof. Pasquale Stanzione, President, Prof. Ginevra Cerrina Feroni, Vice President, Dr. Agostino Ghiglia and Attorney Guido Scorza, members, and Dr. Claudio Filippi - Acting Secretary General;

HAVING SEEN Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation, hereinafter “Regulation”);

HAVING SEEN the Personal Data Protection Code (Legislative Decree 30 June 2003, no. 196), as amended by Legislative Decree 10 August 2018, no. 101, containing provisions for the adaptation of national law to the aforementioned Regulation (hereinafter “Code”);

HAVING SEEN Regulation no. 1/2019 concerning internal procedures with external relevance, aimed at carrying out the tasks and exercising the powers delegated to the Guarantor for the protection of personal data, approved with resolution no. 98 of 4 April 2019, published in the Official Journal no. 106 of 8 May 2019 and in www.gpdp.it, web doc. no. 9107633 (hereinafter “Regulation of the Guarantor no. 1/2019”);

SEEN the documentation in the files;

SEEN the observations formulated by the Secretary General pursuant to art. 15 of the Regulation of the Guarantor no. 1/2000;

REPORTER the lawyer Guido Scorza; 1.    INTRODUCTION 

The proceeding originated from an investigation initiated by the Authority following the publication of press releases and preliminary investigations conducted on the Replika service (https://replika.com/), a chatbot with a written and vocal interface, developed and managed by the US company Luka Inc. (hereinafter “Luka” or the “Company”) and based on a generative artificial intelligence system.

Replika is presented as a chatbot capable of improving the user’s mood and emotional well-being, helping them understand their thoughts and feelings, track their mood, learn coping skills (i.e. stress management), calm anxiety and work towards goals such as positive thinking, stress management, socialization and the search for love. Replika generates a “virtual companion” that the user can decide to configure as a friend, therapist, romantic partner or mentor. 

Replika uses a LLM (Large Language Model) system that is constantly fed and perfected through interaction with users.

For the purposes of this provision, “generative artificial intelligence” means the field of artificial intelligence that focuses on the creation of new and original content with respect to input data in response to user requests (prompts), through the use of predominantly neural algorithms. “Neural network” means a standard computational model applicable in the most diverse contexts that allows the recognition of objects, shapes or patterns within a given data or set of data (for example, a human face in a photograph). Generative artificial intelligence algorithms are used in a wide range of applications, including the recognition and generation of images, vocal or musical tracks, texts and videos.

An example of generative artificial intelligence are large-scale linguistic models (Large Language Models). For the purposes of this provision, “Large Language Model” means a probabilistic model of a natural language, such as English or Italian, which is based on the assumption that all natural languages are highly redundant and correlated; hence the ability of the LLM to identify the word or symbol that, probabilistically, immediately follows a given piece of data.

In light of the aforementioned elements, the Guarantor has initiated an ex officio investigation, noting that the processing of personal data by Luka in the context of the Replika service could give rise to a violation of the legislation on personal data with particular reference to: the privacy policy and the obligations provided for in terms of transparency; the absence in the privacy policy of a specific indication of the legal bases of the processing in relation to the various processing operations carried out; the legal basis of the processing of personal data of minors, having to exclude that, in this case, it could be identified in the execution of a contract; the absence of any filter to verify the age of users, both when accessing the service (by registering the account) and during interaction with the chatbot; the proposition, through the chatbot, of content in conflict with the protections that should be ensured to minors and, more generally, to all the most vulnerable subjects.

In this context, on 2 February 2023, having detected that the processing of personal data by Luka within the Replika service could give rise to the violation of articles 5, 6, 8, 9 and 25 of the Regulation and presented concrete risks for minors, also due to the proposition of responses in conflict with the strengthened protections to be ensured to minors and vulnerable subjects, the President of the Authority adopted against Luka, pursuant to art. 5, paragraph 8, of the regulation of the Guarantor n. 1/2000, an emergency measure (no. 39/2023, prot. no. 18321/23) for the temporary limitation of the processing of personal data of data subjects established in the Italian territory, pursuant to art. 58, par. 2, letter f), of the Regulation.

Subsequently, with measure no. 280 of 22 June 2023 (prot. no. 104960/23), the Authority decided to suspend measure no. 39/2023 for the temporary limitation on condition that the controller, pursuant to art. 58, par. 2, letter d) of the Regulation, adopted suitable measures to ensure that the processing of personal data within the Replika service took place in compliance with the legislation on the protection of personal data. In particular, the Authority ordered the owner to:

1.    present, to all users in Italy, before registration and before accessing the Replika service, an updated privacy policy;

2.    implement an age gate mechanism on all registration pages for the services;

3.    implement a “cooling-off period” aimed at preventing minors from entering a different date of birth when they are denied access to the services;

4.    provide, for users in Italy, the possibility of exercising their rights in terms of personal data protection in a simple and effective way, including the right to object to the processing of personal data and to request access, rectification and deletion of data;

5.    submit to the Guarantor, fifteen days before the scheduled date for the opening of the service to Italian users, a plan for the development of a process aimed at preventing access to the service to persons under the age of 18, possibly accompanied by a language analysis mechanism with subsequent interdictory effect;

6.    submit to the Guarantor, fifteen days before the reopening to Italian users, a plan for the implementation of functions that allow users to report inappropriate content to prevent the Replika chatbot from re-proposing it, such as, for example, the possibility of marking specific responses as inappropriate and providing feedback on the user experience during the session.

The Guarantor has indicated different terms for the implementation of the above-mentioned provisions, establishing that those referred to in points 1 to 4 were to be fully fulfilled no later than 28 July 2023, and that those referred to in points 5 and 6 were to be implemented within fifteen days of the date of reopening of the service to Italian users.

2. LUKA'S RESPONSES TO PROVISIONS NOS. 39/2023 AND 280/2023

The Company, with a note dated 3 March 2023 (prot. no. 38795/23), communicated that it had promptly taken action to follow up on the Authority's requests, in particular to comply with the request for temporary limitation of processing for users established in Italian territory, promptly inhibiting access to the Replika service from Italy, both through the app and through its website.

Luka also reported that it has launched a series of initiatives aimed at concretely implementing the requests of the Guarantor, also through the involvement of external consultants and industry experts; in particular, the Company declared that it has launched a series of assessments, actions and processes aimed at:

-    implementing more robust user age verification mechanisms, in order to strengthen the guarantee that minors in Italy do not use the “Replika” service, a service reserved for adults; in addition to the age gate tools already in use, the Company has undertaken to introduce automated measures aimed at recognizing underage users based on the analysis of indicators contained in conversations with the chatbot;

-    implementing adequate algorithms and processes for the moderation of inappropriate content, according to the best state of the art;

-    ensure compliance with the Regulation by, inter alia, updating the register of processing activities, reviewing and updating data protection impact assessments (DPIA), as well as updating the privacy notice relating to the service, in order to increase the level of transparency for users.

The Company, with a note dated March 31, 2023 (prot. no. 55533/23), requested the revocation of the corrective measure of the temporary limitation ordered by emergency provision no. 39/2023, specifying:

-    that it has designed the Replika service in such a way as to limit the extent of personal data processing, in accordance with the principles set out in art. 5 of the Regulation, including i) minimizing the collection of user registration data (name, email address, date of birth - to verify age - and any third-party log-in data); ii) adopting data retention and deletion procedures that achieve a balance between the need to provide the user with a seamless experience and that of minimizing the personal data that remain accessible; iii) designing proprietary artificial intelligence (AI) models to respond to users; iv) not sharing user conversations with third parties other than the Company’s essential service providers, who are bound by confidentiality obligations; v) implementing strict controls designed to limit access to personal data by its staff; vi) not using user conversations for advertising or marketing purposes; 

-    not offering the service to minors and basing the processing of users’ personal data on the legal basis of contractual performance; 

-    having implemented, following the order of the Guarantor, numerous measures aimed at preventing minors from accessing the service in violation of the Company’s terms;

-    having listed its mobile application in the Apple App Store with an age classification of 17 or older, which is the highest age classification allowed by Apple;

-    not to collect special categories of personal data, given that the sharing of special categories of personal data by users during the interaction with the chatbot occurs spontaneously and must therefore be qualified as covered by an explicit consent to the processing, in accordance with art. 9 of the Regulation;

-    to take seriously its responsibilities in terms of data protection and to have integrated data protection into the design of the service, in accordance with art. 25 of the Regulation and to continue to “develop and improve its policies and procedures to provide users with a consistent, safe and rewarding experience”. 

With specific reference to the provision of the Guarantor, the Company declared:

-    to have promptly blocked access to the Replika service to natural persons who are in Italy;

-    to have strengthened the measures aimed at preventing access to the service by minors under 18, in particular by:  i) introducing an age gate on all service registration pages that requires the indication of a date of birth greater than or equal to 18 years to access the service; ii) providing for a “cooling-off period”, in line with the indications of the data protection authorities and with best practices, to prevent minors from entering a different date of birth when the system denies access to the service; iii) starting activities aimed at improving the automated processes for controlling content (reporting individuals presumably under 18 and preventing use of the service until their age is verified through more robust means);

-    to have updated its privacy policy to resolve the transparency issues identified by the Guarantor;

-    to continue to develop and improve its content moderation practices to avoid harm to users, in particular by creating a trust and safety program to prevent the chatbot from engaging in offensive or harmful conversations;

-    to have limited access to conversations of a sexual nature or relating to other adult content to users active on 1 February 2023, excluding the availability of such types of conversations to new users;

-    to continue to make efforts to ensure compliance with the Regulation with the support of an external consultant in matters of personal data protection. The commitments undertaken by the Company include: i) updating and maintaining the register of the Company’s processing activities; ii) reviewing and updating the data protection impact assessments (DPIA), which include the documentation of the processes for data protection by design and by default; iii) refining and verifying the Company’s security policies and procedures; iv) the review of the Company's governance in terms of data protection (including the possibility of appointing a DPO following the expansion of the Company's activities in the European Union).

The Company, with a note dated 26 April 2023 (ref. 68896/23), submitted a second request for revocation of the corrective measure of the temporary limitation ordered by emergency provision no. 39/2023, reiterating the measures adopted, as already illustrated in the previous note.

The Company, with a note dated 14 June 2023 (ref. no. 93675/23), following up on what was discussed at the hearing held on 31 May 2023, reiterated that it had given timely feedback to provision no. 39/2023, immediately blocking access to Replika in Italy and that it had implemented adequate measures in response to the issues raised by the Authority in the aforementioned provision. The Company also represented its commitment to interrupt the possibility for users located in Italy to engage in sexual conversations, providing, once reactivated, two versions of the Replika service: a free version and a paid version containing romantic, but not sexual, content. The introduction of a paid “romantic” version, according to the Company, involves an additional age verification based on the insertion of the user’s payment card data, in line with the most recent market standards for age verification mechanisms.

The Company, with a note dated 14 July 2023 (prot. no. 109176/23), communicated that it had fulfilled the requests referred to in points 1-6 of provision no. 280/23 and, in particular, declared:

1.    in relation to the information referred to in point 1 of provision no. 280/23, to have implemented an updated privacy policy in the registration process and before accessing the service and that this information would be shown to Italian users when the service was reactivated;

2.    with reference to the age gate mechanism referred to in point 2 of provision no. 280/23, to have implemented an age verification system on all registration pages and that this system would be applied when the service was reactivated;

3.    with reference to the cooling-off period referred to in point 3 of provision no. 280/23, to have implemented a cooling-off period to prevent minors from attempting to access the service again by entering a different date of birth. This period - lasting 24 hours - is expected to be managed by recognizing the credentials of the account of a minor user and consequently inhibiting the entry of a different date of birth and ii) by installing a cookie suitable for preventing minors from entering a different date of birth again from the same browser. The Company stated that such cooling-off period would be applied when the service is reactivated;

4.    as regards the exercise of the rights referred to in point 4 of provision no. 280/23, to provide users with a simple and effective method to exercise their data protection rights, including the right to object to the processing of their personal data and the rights to request access, rectification and erasure of their data and that such mechanism would be applied when the service is reactivated;

5.    as regards the request to prepare a plan for the development of an age verification mechanism during registration referred to in point 5 of provision no. 280/23, to have implemented processes to prevent access to minors under 18, including a linguistic analysis mechanism that requires users to reconfirm their age through the age gate process when users identify themselves as minors under 18. in the absence of a birth date that satisfies the age gate, the user cannot access the service. The Company stated that such processes would be applied in Italy upon reactivation of the service;

6.    with regard to the request to prepare a plan for the development of an age verification mechanism during the use of the service referred to in point 6 of provision no. 280/23, to have implemented functions that allow users to report inappropriate content to prevent the Replika chatbot from re-proposing it, such as, for example, the ability to mark specific responses as inappropriate and to provide feedback on the user's experience during the session. The Company stated that such functions would be applied in Italy upon reactivation of the service.

Luka produced, together with the note of 14 July 2023, a copy of the privacy policy updated to 12 June 2023.

3. INVESTIGATIVE ACTIVITY

In parallel with the adoption of the precautionary measure, the Authority proceeded to acquire the elements deemed necessary for carrying out the investigation through a request for information, pursuant to Articles 58, paragraph 1, letter e), of the Regulation and 157 of the Code.

With a note dated 6 April 2023 (prot. no. 58925/23), the Data Protection Authority sent a request for information to Luka asking for clarifications regarding the functioning of Replika (categories of personal data processed and source from which they are collected; methodology applied for collection; methods of processing of the data collected; place of data storage; security measures adopted; processing of user data for system training purposes or for other purposes pursued by Luka), the processing of users' personal data (legal basis; retention period; minimum age to access the service provided by Replika; DPIA; appointment of a representative pursuant to art. 27 of the Regulation; procedures for managing rights pursuant to art. 12–22 of the Regulation; legal basis and guarantees of adequacy pursuant to Chapter V of the Regulation, where applicable; clarifications regarding automated processing pursuant to art. 22 of the Regulation), and the age verification measures for access to the service on the date of notification of the emergency measure no. 39/23.

With respect to this request, with a note dated 8 May 2023 (ref. no. 74173/23), the Company, after having preliminarily claimed to have a sole establishment in the European Union in the Netherlands, represented:

-    to use the messages and contents that the user sends to the chatbot to enable that user's conversations (the “Chatbot Interaction”). With reference to the Chatbot Interaction, the content of the database may include basic profile information, conversation topics, questions that the user can ask and selected preferences or interests. When a user sends a message, the model analyzes the text to allow the chatbot to generate a response based on the latest messages in the conversation. The Company also specified that it uses a database that contains all the information sent through the chat to create de-identified data and refine the LLM model that forms the basis of the chatbot (“Model Development”). The part of the database used as a source to create de-identified data is limited to: 1) user “Reactions” (“like”, “dislike”, “love”, “funny” “meaningless” or “offensive”), if the user chooses to make such a selection; 2) user “Feedback” of the satisfaction levels of the conversation (“happy”, “neutral” or “sad”); 3) “Snippets”, which are small parts of the user conversations that provide context for the interpretation of the Reactions and Feedback. The information used by the Company for the Development of the Model does not identify specific individuals and cannot be associated with specific individuals (“De-identified Data”) as any personal identifiers (such as names, addresses, emails, phone numbers and identification numbers) that may be contained in the conversation fragments are removed and the fragments are “shuffled” in a randomized manner;

-    to collect all the personal data described above from the interaction of users with the service;

-    to employ a system for collecting (of “Reactions”, “Feedbacks” and “Fragments”) and processing in real time the interactions of users with the chatbot using webhooks, i.e. automated tools that capture such information and send it to the Company’s servers;

-    to follow, in the processing of the “De-identified Data” for the Development of the Model, the following phases: 1) data collection, as illustrated above; 2) pre-processing consisting in the cleaning, structuring and elimination of any personally identifiable data from the data itself, in order to safeguard privacy (through aggregation and randomization techniques); 3) labeling of the pre-processed data; 4) analysis and development to evaluate the performance of the LLM model, identify patterns and develop filters that prevent the model from producing outputs with inappropriate content; 5) testing and validation (regular testing and validation against predefined criteria);

-    to store personal data on encrypted databases hosted by Amazon Web Services, Inc. in the United States;

-    not to use personal data provided by users for Model Development;

-    to employ technical and organizational measures to protect the security of personal data and “De-identified Data” from unauthorized access, use and disclosure. Such measures include encryption, access controls, vulnerability management, pre-processing and anonymization of “Fragments”, “Reactions” and “Feedback”, training and possible disciplinary measures in case of non-compliance with the measures by Company personnel;

-    to rely on the contractual legal basis for “Chatbot Interaction” as the processing of user data is necessary for the provision of the service, in accordance with the Terms of Service. Such processing includes creating and maintaining user account profiles, facilitating payments and transactions and processing data entered by users to generate the chatbot response;

-    to rely on the legal basis of legitimate interest for the “Model Development”;

-    to retain the data for “the time it deems reasonably necessary to offer users a safe, enjoyable and effective experience on the platform”, in compliance with the principle of minimization;

-    to retain the data of the “Chatbot Interaction” for “a period sufficient to facilitate the recall of information to ensure users seamless conversations with the chatbot, in line with user expectations”;

-    to retain [without further specification, editor’s note] the user data to create “De-identified Data” for the “Model Development”;

-    that the minimum age required to use the Replika service is 18 years;

-    that there is no contradiction between the preceding point and the provision in the Company’s privacy policy, which states: “we do not knowingly collect Personal Data from children under the age of 13. If you are under the age of 13, please do not submit any Personal Data through the Services”, as this statement was included because it is required under the US federal law (COPPA);

-    Replika’s mobile application included an age gate that prevented minors under the age of 18 from accessing the service even before the provision of the Guarantor of February 2, 2023. The Company also placed its application in the Apple App Store with an age rating of 17 or above, which is the highest age rating allowed by Apple;

-    all adult content has been placed behind a paywall, out of reach of minors;

-    following the provision of 2 February 2023, the Company has voluntarily improved the measures aimed at preventing subjects under 18 years of age from accessing the service;

-    not to have appointed a representative pursuant to art. 27 of the Regulation as the Company has an establishment in the European Union;

-    as regards the exercise of the rights of the interested parties, the relevant information is provided through a privacy policy published on the Company's website and in the App. Access, rectification and deletion can be requested by users, who can also object and limit the processing of any personal data not necessary for the provision of the service. Requests are assessed individually;

-    not to carry out any profiling activity of the interested parties or to take automated decisions that have legal effects or similar relevance;

-    to directly collect personal data from users and not to transfer them from Italy or the European Union pursuant to Chapter V of the Regulation and to have entered into data processing agreements with data processors, which include standard contractual clauses, where required;

-    for content control purposes, to have trained its models to avoid the emergence and escalation of inappropriate content or inappropriate responses. As part of this process, the Company uses open-source data sets specifically designed and made available to the AI research community to improve the safety and robustness of machine learning models. The Company has also developed, and continues to refine and improve, filters that recognize keywords, phrases and patterns associated with harmful behavior, such as self-harm, insult or murder. The filters trigger the LLM model to respond appropriately to such content, for example by changing the topic of the conversation or providing users with self-help resources. The Company also uses human reviewers both in evaluating the AI model and in developing filters;

-    to use, to control inappropriate content or content contrary to the application's terms of service, also other methods including: 1) placing so-called romantic content behind a paywall and disabling sexually explicit content for new users; 2) allowing users to flag certain content or conversations as offensive in real time and using those flags to improve the models and prevent them from developing similar content in the future; 3) prohibiting users, in the terms of service, from uploading illegal, harmful and threatening content.

Together with the response of May 8, the Company has produced a copy of the privacy policy in force on February 2, 2023, the updated version of the same dated March 22, 2023, as well as a copy of the impact assessment (without date and signature).

With a note dated 27 February 2024 (ref. no. 23744/24) the Authority notified the Company of the communication of the initiation of the procedure for the adoption of corrective and sanctioning measures pursuant to art. 166, paragraph 5, of the Code and art. 12 of the Internal Regulation of the Guarantor no. 1/2019, contesting Luka for the alleged violation of arts. 5; 6; 7; 8; 12; 13; 24; 25, paragraph 1, of the Regulation in relation to the processing of personal data carried out by the Company, through the Replika service on 2 February 2023.

The Company did not provide feedback to the communication of the initiation of the procedure nor did it request to be heard pursuant to art. 166, paragraph 6, of the Code and art. 13 of the Regulation of the Guarantor no. 1/2019. 

In the act initiating the proceeding, which is hereby deemed to be fully and expressly referred to, the Authority contested the Company for three violations on the basis of the critical issues identified in the emergency provision no. 39/2023. The analysis carried out by the Authority focused on the state of the facts, the processing and the obligations implemented by Luka as of 2 February 2023.

With reference to the failure to identify the condition of lawfulness of the processing, the Authority noted that in the text of the privacy policy published on the date of adoption of the emergency provision of the Guarantor, updated to 5 July 2022, the legal basis underlying the various processing operations carried out by the Company within the Replika service had not been identified in a granular manner. The reference to the legal bases of the execution of a contract (art. 6, par. 1, letter b), of the Regulation) and of the consent of the interested parties (art. 6, par. 1, letter a), of the Regulation) as well as the reference to a generic authorization (“authorization”, not obligation) by law, were not in fact referred, nor referable, to specific processing operations (so-called granularity), with the consequent impossibility of identifying and evaluating the suitability of the legal bases themselves. Furthermore, the privacy policy dated 5 July 2022, in force as of 2 February 2023, did not reveal any reference to the legal basis underlying the processing of personal data for the purpose of developing the LLM model that powers the chatbot, nor did the documentation subsequently produced, in particular the privacy policy, also in the version updated to 22 March 2023, and the DPIA, provide elements from which to derive evidence that the Company had identified a legal basis for this purpose at a time prior to 2 February 2023.

In light of the above, the Authority contested Luka for the possible violation of art. 5, par. 1, letter a) and art. 6 of the Regulation for having failed to identify, as of 2 February 2023, the legal bases of the various processing operations carried out through the Replika service.

With reference to transparency obligations, the Authority's assessment concerned the privacy policy in force as of 2 February 2023, i.e. the version updated as of 5 July 2022. From a formal point of view, the Authority, in the act of initiating the proceeding, noted that as of 2 February 2023 the privacy policy was only available in English (including for minors) and was not easy to find. From a content point of view, it was noted that as of 2 February 2023 the privacy policy:

-    did not report any indication of the relevant legal basis in correspondence with the processing activities carried out and the type of data processed;

-    did not indicate the purposes of the processing with reference, in particular to the two distinct types of processing, i.e. the processing aimed at "Chatbot Interaction" and that aimed at "Model Development";

-    in the sections “People mentioned in the chat” and “Integration with your Instagram account” two categories of personal data processed in order to allow users to have conversations were indicated; 

-    it did not clarify that the service was offered exclusively to adults given that, as reported above, the privacy policy only included a reference to minors under 13 years of age in compliance with the requirements imposed by COPPA (Children's Online Privacy Protection Act);

-    it did not provide any specific indication regarding the period of retention of personal data or the criteria used to determine this period;

-    it did not clarify whether there was a transfer of personal data outside the EEA and, if so, what the legal basis and the guarantees of adequacy were as per Chapter V of the Regulation. In particular, the text of the privacy policy (see, in particular, the wording “By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen” is in open contradiction with what was stated by the Company itself in the note of 8 May 2023 (protocol no. 74173/23), where it is stated that, since the criterion of establishment in the European Union does not apply, no transfer of personal data from the European Union (specifically, from Italy) to the United States of America would be possible, pursuant to Chapter V of the Regulation;

-    in section 6 entitled “Your data protection rights”, the privacy policy provided specific information regarding the right under Article 22 of the Regulation, even though the provision was not expressly referred to. This reference (no longer present in the version dated 22 March 2023), was able to make the user believe that his/her personal data were the subject of an automated decision-making process in violation of the principles of transparency and fairness. This circumstance was denied by the owner himself in the response note (prot.74173/23), where he argued that "although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling that have legal effects or similar relevance pursuant to Article 22 of the Regulation".

In light of the above, the Authority contested Luka for the possible violation of art. 5, par. 1, letter a), 6, 12 and 13 of the Regulation given that, as of 2 February 2023, the privacy policy relating to the Replika service was not compliant with the obligations and general principles regarding transparency and was provided in ways and times that did not allow users to promptly view it.

Finally, with reference to the absence of mechanisms for verifying the age of minors, the Authority contested the absence of measures aimed at ensuring specific protection for minors in relation to access and use of the Replika service as of 2 February 2023. In particular, the absence of:

-    a procedure for verifying the user's age (the system only required name, email and gender) with the consequent risk of proposing to minors responses that were unsuitable for their level of development and self-awareness, including sexually explicit content;

-    interdiction or blocking mechanisms even in the face of declarations by the user that made his or her minority evident, as well as the proposition of responses by the chatbot that were clearly in conflict with the protections that should be ensured to minors and, more generally, to all the most vulnerable subjects.

The Authority, in the act of initiating the proceeding, acknowledged that the Company implemented age verification mechanisms following the request of the Guarantor formulated in the context of the provisional limitation measure, adopted urgently on 2 February 2023. In particular, during the discussions that followed the adoption of the aforementioned measure and with specific reference to the age verification profile, the owner represented that it had implemented an age gate on all registration pages for the Services aimed at limiting access to adult users only and that the age verification mechanism includes a "cooling-off period" aimed at preventing the user, having ascertained the impossibility of accessing by entering real personal data, from immediately entering a different date of birth suitable to allow him to access the service. The Company also represented that a process was being developed to use language analysis in order to identify and prevent the use of the Services by persons under the age of 18.

In light of the above, the Authority has contested Luka for the possible violation of art. 5, par. 1, letter c), 6; 7; 8; 24 and 25, par. 1 of the Regulation for the failure to provide suitable systems to verify the age of the subjects as of 2 February 2023.

4. EXISTENCE OF EUROPEAN JURISDICTION AND COMPETENCE OF THE GUARANTOR

First of all, the Authority deems it appropriate to address the issues relating to the applicability of European legislation on the protection of personal data to the service offered by Luka and to the competence of the Guarantor, also taking into account the exceptions raised by the Company in the response dated 8 May 2023 to the request for information sent by the Authority. 

Art. 3 of the Regulation governs the territorial scope of application of the legislation by establishing different criteria depending on whether or not the data controller is established in the territory of the European Union. 

In the first hypothesis (art. 3, par. 1, so-called establishment criterion), the Regulation applies regardless of whether the processing is carried out in the Union and the competence is identified in compliance with the so-called one-stop-shop mechanism, pursuant to art. 56 of the Regulation itself.

In the second hypothesis (Article 3, paragraph 2, so-called targeting criterion), the Regulation applies to the processing of personal data of data subjects who are in the Union if the processing activities concern: i) the offering of goods or services to data subjects in the Union (Article 3, paragraph 2, letter a), of the Regulation); ii) the monitoring of the behavior of data subjects who are in the Union to the extent that such behavior takes place in the Union itself (Article 3, paragraph 2, letter b), of the Regulation).

The Company stated in the above-mentioned note that it has a sole establishment in the European Union in the Netherlands, reporting that it has “a group of employees located in the Netherlands, including several decision makers involved in cross-border data processing for the development of the LLMs and the product” and that “the Company’s employees located in the Netherlands are involved in decisions that affect the processing of personal data by the Company and the operation of the LLMs globally, including decisions that affect the smallest proportion of users located in Italy”. The existence of a Dutch establishment in the European Union would imply the application of the one-stop-shop mechanism and the competence, as lead supervisory authority and in cooperation with the authorities concerned, of the Dutch data protection authority.

However, this statement is not supported by any documents. In fact, both in the privacy policy published on Replika's website as of February 2, 2023 (version updated to July 5, 2022) and in subsequent versions thereof (including the current one updated to February 23, 2024) there is no mention of a Company establishment in the Netherlands; similarly, no mention can be found in the terms of service (neither in the version updated to September 14, 2022 nor in the current one, updated to February 7, 2023), where, on the contrary, it is stated that Luka is "a software company that designed and built Replika, incorporated in Delaware, and operating in San Francisco, CA".

Moreover, the declarations referred to in the note of 8 May 2023 are absolutely generic since the name and corporate name of the company that would be established in the European Union are not even indicated (thus making any verification through mutual assistance with the Dutch supervisory authority pursuant to art. 61 of the Regulation impossible) and are not supported by any documentary evidence (e.g. articles of association of the Dutch company or Chamber of Commerce certificate).

At present, therefore, the Authority believes that no element has been provided that can validly demonstrate the existence of an establishment of the Company in the European Union with the consequent applicability of the establishment criterion pursuant to art. 3, par. 1, of the Regulation and of the one-stop-shop mechanism for the benefit of the Dutch data protection authority.

In this case, the existence of the European jurisdiction and the competence of the Guarantor must be ascertained on the basis of the targeting criterion pursuant to art. 3, par. 2, of the Regulation: specifically, it is therefore necessary to preliminarily assess whether the Replika service can be considered offered to interested parties located in the European Union for the purposes of the applicability of letter a) of the aforementioned art. 3 of the Regulation.

In this regard, reference is made to the “Guidelines 3/2018 on territorial scope”, adopted by the European Data Protection Board (EDPB) on 12 November 2019, which provide that the “controller… demonstrates his intention to offer goods or services to a data subject who is in the Union” (see par. 2 (a) of the cited Guidelines) and the case law of the Court of Justice of the European Union (judgment Pammer/Reederei Karl Schlüter GmbH & Co and Hotel Alpenhof/Heller - joined cases C-585/08 and C-144/09), which has indicated some factors in the presence of which it can be considered that a commercial activity carried out by a subject is directed towards a Member State, including the circumstance that the European Union is mentioned in reference to the good or service offered, the international nature of the activity or the launch of advertising and marketing campaigns aimed at the public of an EU country.

In this case, the evidence that the Replika service was offered to data subjects who were in the European Union and, in particular, in Italy on 2 February 2023, emerges “per tabulas” from the Company’s first response to the temporary limitation order contained in the emergency provision of the Guarantor no. 39/2023, where it is stated (see note of 3 March 2023, page 1) that “the Company promptly complied with the request for temporary limitation of processing for users established in Italian territory, promptly inhibiting access to both the app and the website of the service from Italy”.

Having demonstrated in the ways and terms set out above the territorial applicability of the Regulation and the competence of the Guarantor, the following is observed.

The processing of personal data carried out by Luka can be classified as cross-border processing of personal data pursuant to art. 4, par. 1, no. 23 of the Regulation, as it is capable of affecting data subjects in more than one Member State.

For this type of processing, where the controller has identified a single or main establishment in the European Union, as already illustrated, the cooperation mechanism described in Articles 60 et seq. of the Regulation applies and the competence to exercise the tasks and powers referred to in Articles 57 and 58 of the Regulation is rooted, pursuant to Article 56, par. 1, of the Regulation, in the lead supervisory authority, i.e. the supervisory authority of the Member State in which the single or main establishment is located.

If, on the contrary, as in the present case, there is no establishment of the data controller in the European territory, the latter will have to "interface with the supervisory authorities of each Member State in which it operates through the designated representative" (see par. 3.3. of the "Guidelines on the Lead Supervisory Authority" adopted by the Article 29 Working Party on 13 December 2016, revised on 5 April 2017 and adopted by the EDPB on 25 May 2018).
In fact, where a controller does not have an establishment in the European Union (or rather in the EEA area), the special rule under art. 56 does not apply in favour of the general rule under art. 55, par. 1, of the Regulation according to which "each Supervisory Authority shall be competent to perform the tasks assigned to it and to exercise the powers conferred upon it under the (...) Regulation in the territory of the respective Member State".

In the case in question, as mentioned, Luka is a company based in the United States of America that has not demonstrated that it has an establishment in the territory of the European Union. Therefore, the Italian data protection authority is competent to assess, with regard to its territory, the compliance with the Regulation of the processing of personal data carried out by the Company and to exercise the powers granted to it by art. 58 of the Regulation.

5. THE CONFIRMED VIOLATIONS

5.1 ARTT. 5, PAR. 1, LETT. A) AND 6 OF THE REGULATION

The Office contested Luka's violation of art. 5, par. 1, lett. a) and 6 of the Regulation for not having identified, as of 2 February 2023, the legal bases of the various processing operations carried out through the Replika service, a service offered and available to the public in Italy on that date. 

Art. 5, par. 1, of the Regulation prescribes that “personal data shall be: a) processed lawfully, fairly and in a transparent manner in relation to the data subject («lawfulness, fairness and transparency»); b) collected for specified, explicit and legitimate purposes, and not further processed in a manner that is incompatible with those purposes; further processing of personal data for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes shall not, in accordance with Article 89, paragraph 1, be considered incompatible with the initial purposes («purpose limitation»); c) adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed («data minimisation»); d) accurate and, where necessary, kept up to date; every reasonable step shall be taken to ensure that personal data that are inaccurate, having regard to the purposes for which they are processed, are erased or rectified without delay («accuracy»); e) kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which they are processed; personal data may be retained for longer periods insofar as the personal data will be processed solely for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes in accordance with Article 89(1), subject to implementation of the appropriate technical and organisational measures required by this Regulation in order to safeguard the rights and freedoms of the data subject (‘storage limitation’); (f) processed in a manner that ensures appropriate security of the personal data, including protection against unauthorised or unlawful processing and against accidental loss, destruction or damage, using appropriate technical or organisational measures (‘integrity and confidentiality’). Paragraph 2 of the same provision provides that “The controller shall be responsible for and able to demonstrate compliance with paragraph 1 (‘accountability’)”.

Recital 39 clarifies that “Any processing of personal data should be lawful and fair. It should be transparent to natural persons how personal data relating to them are collected, used, consulted or otherwise processed, as well as to what extent the personal data are or will be processed. The principle of transparency requires that information and communication relating to the processing of such personal data be easily accessible and understandable and that clear and plain language be used. This principle concerns, in particular, the information of data subjects on the identity of the controller and the purposes of the processing and further information to ensure fair and transparent processing with regard to the natural persons concerned and their rights to obtain confirmation and communication of whether personal data relating to them are being processed”.

Article 6 of the Regulation prescribes the conditions of lawfulness of the processing by listing the six possible legal bases (consent, contract, legal obligation, vital interest, public interest, legitimate interest) on which the owner must rely in order to lawfully process the personal data necessary for the performance of his activity. The legal basis, as clarified by the EDPB, “must be identified before the processing is implemented and must be specified in the information provided to the data subjects in accordance with Articles 13 and 14. (see Guideline 2/2019 on the processing of personal data pursuant to Article 6, paragraph 1, letter b), of the General Data Protection Regulation in the context of the provision of online services to data subjects”)”.

The Company did not submit written defences or documents, pursuant to art. 166, paragraph 5 of the Code, following the notification of the notice of contestation and initiation of the proceedings by the Office and therefore did not provide any counter-arguments with respect to the hypothesis of infringement relating to the failure to indicate the legal basis for each of the processing activities carried out by Luka within the Replika service.

In this case, from the investigative documents, in particular from the text of the privacy policy published on the date of the adoption of the emergency measure of the Garante, updated to 5 July 2022, it clearly emerges that the Company has not identified in a granular manner the legal basis underlying the various processing operations carried out by the Company within the scope of the Replika service, including that for the processing of data used for the development of the LLM model.

The only references, in the introductory part of the text in question, are the following: “We care about the protection and confidentiality of your data. We therefore only process your data to the extent that:

•    It is necessary to provide the Replika services you are requesting,

•    You have given your consent to the processing, or

•    We are otherwise authorized to do so under the data protection laws”.

The legal bases of the execution of a contract (art. 6, par. 1, lett. b), of the Regulation), of the consent of the interested parties (art. 6, par. 1, lett. a), of the Regulation) and of a legal authorization (where, moreover, the Regulation provides among the legal bases a legal obligation, not a mere authorization), are recalled implicitly and generically, without referring to specific processing operations (so-called granularity principle), with the consequent impossibility of identifying and assessing their suitability.

Finally, neither the privacy policy nor the documentation in the files make reference to the legal basis underlying the processing of personal data aimed at developing the LLM model that powered the chatbot at the date of February 2, 2023.

Specifically, the findings provided by Luka, although pertinent, are not conclusive. In particular, the DPIA and the privacy policy produced on May 8, 2023, do not allow to overcome the objections raised by the Authority in the dispute with reference to the principle of lawfulness and the legal basis of the processing, respectively governed by Articles 5, par. 1, letter a) and 6 of the Regulation, since:

-    the privacy policy, even in the subsequent version updated on March 22, 2023, in the table in paragraph 2.A, does not expressly mention the purpose of the “Model Development” or the related legal basis;

-    the DPIA, although distinguishing the two processing purposes relating to the “Chatbot Interaction” and the “Model Development” (par. I) and analyzing the respective legal bases (par. II), does not present a certain date, therefore it does not prove that the identification of the aforementioned conditions of lawfulness referred to in Article 6 of the Regulation occurred prior to 2 February 2023. Moreover, the DPIA states the legitimate interest as a legal basis for processing for the purposes of "Model Development" without indicating any argument relating to the so-called "triple test" underlying the legitimate interest assessment. Finally, it is highlighted that the DPIA, although an excellent accountability tool, does not constitute the place elected by the legislator in order to provide data subjects with information relating to processing activities, information that must be provided through the privacy policy.

With reference to art. 5, par. 1, letter a), of the Regulation, reference is made here to the principle expressed by the EDPB in the binding decision no. 1/2021 on transparency, but also applicable with reference to lawfulness, according to which the principles set out in art. 5 of the Regulation must be considered as a general concept that is then concretely implemented in various specific provisions and obligations (in the event of lawfulness, in Articles 6, 7, 8, 9 and 10 of the Regulation).

Therefore, according to the EDPB, it is necessary to distinguish the specific obligations deriving from a principle (in this case, Article 6 of the Regulation) from the principle itself expressed in Article 5 of the Regulation, since the latter cannot be limited to the specific obligation, even if the latter is a concretization of the former.

The principle of lawfulness, in fact, is an all-encompassing principle that reinforces other principles (e.g. fairness, accountability). Confirmation of this reconstruction is given by the fact that Article 83, paragraph 5, of the Regulation provides for the possibility of sanctioning the violation of the obligations of lawfulness independently of the violation of the principle itself. In this case, the Authority considers that the violation of the principle of lawfulness referred to in Article 5, paragraph 1, letter a) of the Regulation can also be identified, taking into account the seriousness (failure to clearly and granularly identify the legal bases underlying to the various processing operations) of the nature (this is an essential element of the processing) and of the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the single specific violation of the obligation under Article 6 of the Regulation.

In light of the above, the Authority considers that Luka has not identified, as of 2 February 2023, the legal bases of the various processing operations carried out through the Replika service, offered and available to the public in Italy on that date, in violation of Articles 5, paragraph 1, letter a) and 6 of the Regulation.

With regard to the analysis and assessment of merit in relation to the legal bases under Article 6, paragraph 1, letter b) and letter f), of the Regulation allegedly underlying the use of the chatbot and the post-training of the LLM model underlying the Replika service, as well as, in general, the legal bases relating to the entire life cycle of the system of generative artificial intelligence implemented by the Company, the Authority reserves the right to open a separate and independent investigation.

5.2 ARTT. 5, PAR. 1, LETT. A), 12 AND 13 OF THE REGULATION

The Office contested Luka for violating articles 5, par. 1, lett. a), 12 and 13 of the Regulation for having provided, on 2 February 2023, a privacy policy relating to the Replika service with content that does not comply with the obligations and general principles of transparency provided for by the legislation.

Article 5, par. 1, lett. a), of the Regulation requires that personal data be processed lawfully, fairly and transparently in relation to the data subject (principle of lawfulness, fairness and transparency).
Article 12 of the Regulation establishes rules on transparency and methods of exercising rights, while Article 13 introduces specific indications regarding the information that the owner is required to provide if the personal data are collected from the data subject.

On the subject of transparency, Recital 58 of the Regulation provides that information intended for the public or the data subject must be concise, easily accessible and easy to understand, that clear and plain language must be used, and, with reference to the specific protection to which minors must be addressed, provides that "where the processing of data concerns them, any information and communication should use clear and plain language that a minor can easily understand".

On the subject of transparency, the indications of the Committee are also relevant, in particular Guidelines 2/2019 on the processing of personal data pursuant to Article 6, paragraph 1, letter b) of the General Data Protection Regulation in the context of the provision of online services to data subjects, where it is provided that the legal basis of the processing, in addition to having to be identified before the implementation of the processing, "must be specified in the information provided to data subjects in accordance with Articles 13 and 14"; furthermore, the Guidelines no. 1/2022 of the Committee on access are relevant, which prescribe, in paragraph 142, that “a controller offering a service in a given country should also respond in a language understood by data subjects in that country”.

Finally, the guidelines adopted by the Article 29 Working Party on 11 April 2018, clarified that “the concept of transparency in the Regulation is not legalistic, but rather user-centric and is embodied in several articles containing specific obligations imposed on controllers and processors. The concrete (information) obligations are set out in Articles 12-14 of the Regulation. (…) The transparency obligations imposed by the Regulation apply regardless of the legal basis of the processing and throughout the lifecycle of the processing. This is clear from Article 12, which states that transparency applies at the following stages of the data processing cycle: (i) before or at the beginning of the data processing cycle, i.e. when the personal data are collected from the data subject or otherwise obtained; (ii) throughout the lifecycle of the processing, i.e. when communicating with data subjects about their rights; (ii) at specific moments when processing is ongoing, for example when a data breach occurs or in the event of a significant modification of the processing”.

The Company did not submit any written defense or documents, pursuant to art. 166, paragraph 5 of the Code, in response to the Office's notice of challenge and initiation of the proceedings and therefore did not provide any counter-arguments with respect to the hypothesis of violation formulated relating to the obligations and general principles of transparency provided for by the legislation.

The Office's investigation, as already specified above, concerned the privacy policy adopted and published by Luka on 2 February 2023, i.e. the version of the same updated to 5 July 2022.

First of all, the investigation documents show that, from a formal point of view, on 2 February 2023, the privacy policy was only available in English, not considering the language of the country in which the service was offered, namely Italian. 

From a substantive perspective, it is noted that as of 2 February 2023, the privacy policy did not comply with the principles of correctness and transparency as it was incomplete and incorrect.

In particular, from the point of view of the completeness of the information provided to the interested parties, it was found that the privacy policy:

-    did not indicate in a granular manner the legal basis relating to each of the processing activities carried out, nor the type of data processed;

-    did not indicate the purposes of the two distinct types of processing activities, namely the processing of data through the “Chatbot Interaction”, aimed at allowing users to register for the service and interact with the platform, and the processing of data in the context of the “Model Development”, aimed at improving the security and performance of the Large Language Model (LLM) underlying the service offered (“Model Development”);

-    did not clarify that the service was offered exclusively to adults, while inviting minors under 13 not to use the service. In particular, paragraph 8 of the aforementioned privacy policy stated: “We do not knowingly collect Personal Data from children under the age of 13. If you are under the age of 13, please do not submit any Personal Data through the Services. We encourage parents and legal guardians to monitor their children’s Internet usage and to help enforce our Privacy Policy by instructing their children never to provide Personal Data on the Services without their permission. If you have reason to believe that a child under the age of 13 has provided Personal Data to us through the Services, please contact us, and we will endeavor to delete that information from our databases”. This information, while making clear the circumstance for which the service was not offered to subjects under the age of 13 (“If you are under the age of 13, please do not submit any Personal Data through the Service”), did not clarify that the chatbot was reserved only for adults, since users between the ages of 13 and 18 were also excluded.

This last circumstance was clarified by the Company only at a later time;

-    did not provide any specific indication regarding the period of retention of personal data or the criteria used to determine such period;

-    did not clarify whether there was a transfer of personal data outside the EEA and, if so, what the legal basis of the processing was and the adequacy guarantees adopted pursuant to Chapter V of the Regulation. In more detail, the information provided by the Company (specifically in the part of the privacy policy in which it stated that: “By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen”) was likely to generate an erroneous belief in the data subject regarding the transfer of his or her personal data to the USA. The absence of a transfer of data to third countries was confirmed by the Company itself in the note of 8 May 2023 (prot. 74173/23) relating to the applicability of the criterion of establishment in the European Union. Therefore, it is noted that the presence of misleading information was confirmed in the declarations of the owner;

-    in section 6 “Your data protection rights”, although not expressly referring to Article 22 of the Regulation, it provided specific information regarding this right, generating in the user the unfounded belief that his/her personal data were subject to an automated decision-making process. The absence of automated processing pursuant to Article 22 of the Regulation was confirmed by the owner himself in the feedback note (prot.74173/23), where he argued that “although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling that have legal effects or similar relevance pursuant to Article 22 of the Regulation”. Therefore, also in this case, it is found that the presence of misleading information has been confirmed in the same declarations of the owner.

With reference to art. 5, par. 1, letter a), of the Regulation, reference is made to the same binding decision of the EDPB cited in the previous paragraph (binding decision no. 1/2021), according to which transparency must be considered a general concept that finds concrete implementation in various provisions and specific obligations (for example, arts. 12, 13, 14, 25 and 35). It is therefore necessary to distinguish the specific obligations deriving from the principle of transparency (referred to in articles 12-14 of the Regulation) from the principle expressed in art. 5 of the Regulation, since although these obligations are a concretization of the general principle, the latter has a broader scope.

The principle of transparency, in fact, is an all-encompassing principle that strengthens other principles (e.g. fairness, accountability). Confirmation of this reconstruction is given by the fact that art. 83, par. 5, of the Regulation provides for the possibility of sanctioning the violation of transparency obligations independently of the violation of the principle itself. In other words, the transparency obligations do not define the entire scope of the transparency principle, it follows that the violation of the transparency obligations provided for by Articles 12-14 of the Regulation may also constitute a violation of the transparency principle only if characterized by elements of seriousness and systematicity.

In this case, the Authority considers that the violation of the transparency principle referred to in Article 5, par. 1, letter a) of the Regulation can also be identified, taking into account the seriousness (lack of information to the interested parties about the legal bases underlying the various processing operations of their personal data) of the nature (lack of clear information about the essential elements of the processing such as legal basis, purpose, principle of conservation, transfer outside the EU) and the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the individual specific violations of the obligations referred to in Articles. 12 and 13 of the Regulation.

For the reasons stated above, the Authority believes that Luka violated, as of 2 February 2023, Articles 5, paragraph 1, letter a), 12 and 13 of the Regulation. 

For the sake of completeness, it should be noted that further technical investigations have revealed that the data controller has again updated the privacy policy relating to the Replika service on 23 February 2024. In this version, some of the inaccurate and incorrect information indicated above has been modified. In particular, the privacy policy, in force on the date of adoption of this provision, reports in a granular manner the legal basis relating to each of the processing activities carried out by the data controller and the type of data processed; it expressly clarifies that the service is offered exclusively to adults, and does not contain any reference, not even implicit, to the automated decisions referred to in Article 22 of the Regulation. Nonetheless, the information provided pursuant to Articles 12 and 13 of the Regulation continue to be available only in English, do not provide specific references regarding the period of retention of personal data or the criteria used to determine this period and are potentially suitable to generate in the interested party an erroneous belief regarding the transfer of his/her personal data to the USA.

5.3 ARTT.   5, PAR. 1, LETT. C), 6, 7, 8, 24 AND 25, PAR. 1, OF THE REGULATION

The Office contested Luka for the violation of arts. 5, par. 1, lett. c); 6; 7; 8; 24 and 25, par. 1 of the Regulation for failure to prepare systems to verify the age of the subjects as of 2 February 2023. 

Pursuant to article 5, par. 1, lett. c), of the Regulation “Personal data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed”.

Pursuant to art. 24, par. 1, of the Regulation “Taking into account the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for the rights and freedoms of natural persons, the controller shall implement appropriate technical and organisational measures to ensure and to be able to demonstrate that processing is performed in accordance with this Regulation. Those measures shall be reviewed and updated where necessary”.

Pursuant to art. 25, par. 1, of the Regulation, the controller shall adopt such measures “taking into account the state of the art, the costs of implementation and the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for the rights and freedoms of natural persons posed by the processing, both at the time of determining the means for processing and at the time of the processing itself”.

In the guidelines no. 4/2019 on Article 25 of the Regulation, the EPDB clarified that “the core of the provision is to ensure adequate and effective data protection by design and protection by default, which means that controllers should be able to demonstrate that they incorporate appropriate measures and safeguards into the processing to ensure the effectiveness of the data protection principles and the rights and freedoms of data subjects” and invited controllers to take into account, in the context of designing and setting up the processing in a privacy-oriented perspective, also the obligations to provide specific protection to minors and other groups of vulnerable subjects.

In the same guidelines, the EDPB also underlined that “In line with Article 25, paragraph 1, the controller shall implement appropriate technical and organizational measures that are designed to implement the data protection principles and shall integrate the necessary safeguards into the processing to comply with the requirements and protect the rights and freedoms of data subjects. Both adequate measures and necessary safeguards aim to pursue the same purpose of protecting the rights of data subjects and ensuring that the protection of their personal data is integrated into the processing. The expressions technical and organizational measures and necessary safeguards can be understood in a broad sense as any method or means that a controller may employ in processing. The term adequate means that the measures and necessary safeguards must be suitable for achieving the intended purpose, i.e. they must effectively implement the data protection principles”.

The Company has not submitted written defenses or documents, pursuant to art. 166, co. 5 of the Code, in response to the Office's contestation and initiation of proceedings and therefore did not provide any counter-arguments with respect to the hypothesis of violation formulated in relation to the failure to prepare systems to verify the age of the subjects.

In light of the aforementioned rules and guidelines, the Authority observes that the data controller is obliged to implement suitable technical and organizational measures to guarantee and demonstrate that the processing is carried out in compliance with the Regulation and to process only adequate, relevant and limited data to what is necessary with respect to the purposes of the processing itself.

However, the investigation revealed that the Company had failed to prepare measures to guarantee specific protection for personal data processed within the Replika service relating to minors under 18 years of age. Indeed, the absence of procedures to verify the age of the user who intended to access the service, as well as of interdiction or blocking mechanisms in the event of declarations by the user that made his or her minority clear, highlighted how the owner had not assessed, ex ante, the risks that registration and use of the service by minors under 18 could give rise to, with the consequence that, on the one hand, it did not adopt any measures to counter such risks, minimize them or limit them, and on the other hand, it processed data in excess of those necessary to satisfy the purposes of the processing (i.e. offering the service to adult users).

The investigation revealed that as of 2 February 2023, the Company did not provide any mechanism to verify the age of users either when registering for the Replika service or during its use, even though it excluded minors among potential users.

In particular, the absence of:

-    a procedure for verifying the user's age (the system only required name, email and gender) was detected, with the consequent risk of proposing to minors responses that were unsuitable for their level of development and self-awareness, including sexually explicit content;

-    interdiction or blocking mechanisms even in the face of user declarations that made their minority evident, as well as the proposition of responses by the chatbot that were clearly in contrast with the protections that should be ensured to minors and, more generally, to all the most vulnerable subjects.

Therefore, until 2 February 2023, the Company has not adopted any technical and organizational measures to ensure compliance with the general principles of the Regulation and the protection of the rights and freedoms of minors, thus exposing minors to significant risks to their person that the legislation in question aims to limit, including responses that are unsuitable with respect to their level of psychophysical development and self-awareness. Luka adopted age verification mechanisms only following the request of the Guarantor formulated in the context of the provisional limitation measure, adopted urgently on 2 February 2023. In particular, during the discussions that followed the adoption of the aforementioned measure and with specific reference to the age gate profile, the Company represented that it had implemented an age gate on all registration pages to the Services that limits access to users who have turned 18 and that includes a "cooling off period" aimed at preventing people who have not been able to access by entering real personal data from entering, immediately afterwards, a different date of birth. 

The Company also represented that it had planned to develop a process to use language analysis in order to identify and prevent the use of the Services by people under the age of 18. 

Before the Authority's intervention, therefore, all users, including minors, could register for the Replika service and use it without being asked to undergo any age verification. As already clarified in the dispute, in the Authority's opinion, the absence of a common standard suitable for guaranteeing, in a certain and absolute manner, the effectiveness of a user age verification model and the discussion still underway at European level in this regard, cannot be considered sufficient reasons to exclude the fulfillment of the obligations to which the data controller is bound, in particular that of verifying the user's actual contractual capacity for the purposes of the validity of the contract.

From the foregoing it emerges that, as of 2 February 2023, the Company has neither implemented, pursuant to art. 24 of the Regulation, measures aimed at ensuring that the processing of data at the time of registration for the Replika service complies with art. 5, par. 1, lett. c), 24, 25 and of the Regulation, in particular that the same had adopted technical and organizational measures "aimed at effectively implementing the principles of data protection, such as minimization, and at integrating the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of the data subjects" with consequent processing of data that were superfluous, or rather unnecessary, with respect to the purposes of a service that, according to the declarations of the same owner as well as the documentation in the files, was offered only to users over 18 years of age.

With specific reference to the violation referred to in art. 5, par. 1, letter c), of the Regulation, it is noted that, in this case, the adoption of adequate technical and organizational measures, from the design stage, aimed "at effectively implementing the principles of data protection, such as minimization", as well as being a constituent element of art. 25, par. 1, of the Regulation also constitutes a quid pluris suitable to consider the violation of the principle of minimization as integrated, in line with what is clarified by recital 78 of the Regulation.

Specifically, the failure by the Company to adopt suitable measures to safeguard access to and use of the Replika service meant not only that Luka systematically processed personal data in addition to those actually necessary to achieve the purpose of the processing (i.e. offering the service to adult users), but also that such processing concerned data relating to vulnerable subjects (minors, potentially even under the age of 13) who, due to this deficiency and given the innovative technology underlying the service and the highly sensitive nature of the conversations provided by the chatbot, were exposed to a particularly high risk. 

The news reports, which led to the Authority's investigation, together with specific cases of self-harm related to the use of the chatbot reported by the foreign press and brought to the attention of the judicial authority, are elements that support the Office's challenge and which, on the basis of the principles expressed by the repeatedly recalled binding decision of the EDPB no. 1/2021, require the enhancement of the seriousness and impact of the violations, making it considered integrated both the violation of the principle referred to in art. 5, par. 1, of the Regulation, and the specific violation of the obligation referred to in arts. 24 and 25, par. 1, of the Regulation.

The Authority, on the contrary, does not believe that there are sufficient elements to declare the violations, contested pursuant to art. 166, par. 5, of the Code, relating to consent, in particular the positive act expressed by the minor in the context of digital services, pursuant to Articles 6, 7, 8 of the Regulation. In particular, it is noted that during the investigation it emerged that, unlike what was erroneously indicated in the privacy policy in the version dated 2 February 2023 (see § 5.2) - the Replika service was not - and is not - offered to minors, it follows that the owner was not required to comply with the obligation to identify a legal basis for processing that was presumed not to be carried out. 

In light of the above, the Authority believes that Luka violated, as of 2 February 2023, Articles 5, par. 1, letter c) and 24 and 25, par. 1, of the Regulation.

For the sake of completeness, it should be noted that, as of the date of adoption of this provision, further technical investigations have revealed that the age verification system currently implemented by the data controller continues to be deficient in several respects; in particular, it has been ascertained that:

- after the creation of the user profile, it is possible to change the date of birth in the "My Profile" section without any verification by the data controller, with the result that a minor who has registered for the service indicating a false age could promptly change it by entering the correct one without any consequences, continuing to be able to access the service;

- the cooling off period (of 24 hours) does not operate where the profile is created via incognito browsing; in fact, as far as it appears, once the first age check has failed, it is possible to successfully complete registration for the service by changing the email address entered with a new address (including non-existent and non-functioning email addresses);

- linguistic analysis mechanisms have not been set up that require users to reconfirm their age through the age gate process when they identify themselves as minors under 18, with the sole exception of the case in which the user himself provides specific input (e.g. unequivocally declares himself to be under 18). In such cases, the application responds by asking for confirmation of the age of majority. 

During the technical checks, it also emerged that the user is offered the possibility of marking some conversations as inappropriate, however it is not possible to detect the consequences of such reporting. 

6. CONCLUSIONS

In light of the assessments expressed above, the existence, in the terms indicated below, of the majority of the violations contested by the Office notified with the act initiating the procedure is confirmed and the unlawfulness of the processing of personal data carried out by the Company is declared, in violation of art. 5, par. 1, letter a) (with reference to both the principle of lawfulness and transparency) and letter c) of the GDPR. c), 6, 12, 13, 24 and 25, paragraph 1, of the Regulation.

The ascertainment of the violation of the aforementioned provisions of the Regulation requires the consequent adoption of corrective measures pursuant to art. 58, paragraph 2, of the Regulation, in particular the order for compliance pursuant to art. 58, paragraph 2, letter d) of the Regulation and also makes applicable, pursuant to art. 58, paragraph 2, letter i), of the Regulation, the administrative sanction provided for by art. 83, paragraphs 3 and 5, of the Regulation itself. 

Furthermore, taking into account the particular sensitivity of the data processed, it is also believed that the accessory sanction of publication on the website of the Guarantor of this provision, provided for by art. 166, paragraph 7 of the Code and art. 16 of the Guarantor Regulation no. 1/2019, should also be applied.

The Authority reserves the right, as specified above, to examine and verify in an independent proceeding the profiles concerning the lawfulness of the processing carried out by the Company with specific reference to the legal bases of the processing of personal data relating to the entire life cycle of the generative artificial intelligence system underlying the Replika service.

7. CORRECTIVE MEASURES PURSUANT TO ART. 58, PARAGRAPH 2, LETTER D) OF THE REGULATION

Article 58, par. 2, of the Regulation provides for the Guarantor a series of corrective powers, of a prescriptive and sanctioning nature, to be exercised in the event that unlawful processing of personal data is ascertained.

Among these powers, Article 58, par. 2, lett. d), of the Regulation provides for the power to "order the data controller ... to bring the processing into conformity with the provisions of this Regulation, where appropriate, in a specific manner and within a specific period".

From what has been found and considered in the preceding paragraphs, it has emerged that Luka has violated, as of 2 February 2023, Articles 5, par. 1, letter a) (with reference to both the principle of lawfulness and the principle of transparency) and letter c), 6, 12 and 13, Articles 24 and 25, par. 1 of the Regulation but that, following the emergency intervention of the Authority, it has adopted some measures to remedy the critical issues that have emerged and, subsequently, has adopted further measures with respect to the violations contested in the act of initiation of the proceeding, which ensure the compliance of the processing with the legislation on the protection of personal data.

In particular, Luka has remedied the violation referred to in Articles 5, par. 1, letter a) and 6 of the Regulation by amending the privacy policy (see latest version dated 23 February 2024) specifying, in detail, the legal bases of the various processing operations carried out through the Replika service.

In light of the amendment of the privacy policy in the terms just described, it is believed that the conditions for the adoption of further corrective measures pursuant to art. 58, par. 2, of the Regulation do not currently exist.

On the other hand, with reference to the violation of articles art. 5, par. 1, letter a), 12 and 13 of the Regulation, relating to the information obligations and the violation of articles 24 and 25, par. 1 and 5, par. 1, letter c), of the Regulation, relating to the age verification system, there remain profiles of non-compliance with the Regulation that are believed to be the subject of specific provisions.

In particular, with reference to the information obligations, the Authority has ascertained that, to date, Luka's privacy policy (latest version dated 23 February 2024) continues to not comply with the legislation on the protection of personal data to the extent that i) it is only available in English, ii) it does not indicate in a timely manner the periods of retention of personal data or the criteria used to determine such periods, iii) it is likely to generate in the interested parties an erroneous belief regarding the transfer of their personal data to the USA.

Therefore, pursuant to Article 58, paragraph 2, letter d), of the Regulation, the controller is required to conform the privacy policy to Articles 5, paragraph 1, letter a), 12 and 13 of the Regulation by remedying the gaps indicated above.

Furthermore, with reference to the age verification system, the Authority has ascertained that, on the date of adoption of this provision, the age verification system used by the data controller does not comply with the principle of data minimization and the principles of privacy by design and by default, given that:

- after creating the profile, the user can change the date of birth in the "My Profile" section without any verification by the data controller. It follows that a minor who has registered for the service indicating a false age could promptly change it by entering the correct one without any consequences and continue to be able to access the service;

- the 24-hour cooling off period does not apply where the user creates the profile while browsing in incognito mode. In fact, as far as it appears, once the first age check has failed, the user can successfully complete the registration for the service by changing the email address entered with a new address (including non-existent and non-functioning email addresses);

- the data controller has not set up linguistic analysis mechanisms that require users to reconfirm their age through the age gate process in the presence of clear signals that identify a user under 18, with the sole exception of the case in which the user provides specific input (e.g. unequivocally declares that he or she is under 18). Only in the presence of such cases does the application respond by asking for confirmation of the age of majority. 

The Authority, however, positively found Luka's implementation of a function that allows the user to mark certain conversations as inappropriate, although it was not possible to detect the effect of this reporting. 

That said, pursuant to Article 58, paragraph 2, letter d) of the Regulation, the data controller is required to bring the age verification system into line with Articles 5, paragraph 1, letter c), 24 and 25, paragraph 1 of the Regulation, remedying the shortcomings indicated above. 

8. INJUNCTION ORDER FOR THE APPLICATION OF THE PECUNIARY ADMINISTRATIVE SANCTIONS AND ANCILLARY SANCTIONS

The Authority, pursuant to art. 58, par. 2, letter i), and 83 of the Regulation as well as art. 166 of the Code, has the power to impose an administrative pecuniary sanction pursuant to art. 83, in addition to or in place of the other corrective measures provided for in the same paragraph.

In determining the sanction, the Authority takes into account the principles and interpretation on the matter provided by the EDPB in the Guidelines 4/2022 on the calculation of administrative pecuniary sanctions pursuant to the GDPR, version 2.1, adopted on 24 May 2023.

On the basis of the arguments put forward above, the Guarantor has ascertained the violation of the following provisions of the Regulation: art. 5, par. 1, letter a) and 6; art. 5, par. 1, letter a), 12 and 13; art. 5, par. 1, letter c), 24 and 25, par. 1, of the Regulation.

In this case, it should first be noted that the Company has implemented a series of conducts that have integrated multiple violations, as specifically outlined and motivated in the previous paragraphs. The violations relating to the legal basis (art. 5, par. 2 and 6 of the Regulation), transparency (art. 5, par. 1, letter a), 12 and 13) and age gate (art. 24 and 25, par. 1) can be traced back, due to the principle of unity of action, under the aegis of art. 83, par. 3, of the Regulation, according to which in the presence of multiple violations of the Regulation, relating to the same processing or to connected processing, the total amount of the administrative pecuniary sanction cannot exceed the amount envisaged for the most serious violation.In particular, with reference to such violations, a hypothesis of linked processing can be configured, as defined in paragraph 28 of the aforementioned guidelines (a single conduct consists of multiple actions that are carried out on the basis of a single will and are contextually, spatially and temporally correlated in such a close way that they can be considered, from an objective point of view, as a single coherent conduct). The most serious violation compared to those mentioned above must be identified in the violation of transparency obligations given that both art. 5, par. 1, letter a) (principle of transparency) and arts. 12 and 13 (rights of the interested parties) are sanctioned pursuant to art. 83, par. 5, which sets the maximum amount of 20 million euros or, for companies, 4% of the annual worldwide turnover of the previous financial year, if higher. 

Pursuant to art. 83, par. 1 of the Regulation, the administrative sanction must be effective, proportionate and dissuasive in relation to the individual case. In the aforementioned guidelines, the EDPB has specified that the calculation of administrative pecuniary sanctions must start from a harmonized starting point, which constitutes the initial basis for the further calculation of the amount of the sanction, in which all the circumstances of the case are taken into account and weighed (see paragraph 46). The harmonized starting point must take into account three factors: 1) nature of the infringement pursuant to Article 83, paragraphs 4 to 6, of the Regulation; 2) seriousness of the infringement; 3) turnover of the company (see paragraph 48). Starting from the first profile, in the case in question, there are two infringements, in theory, of a more serious nature (Article 83, paragraph 5, of the Regulation) and one less serious infringement (Article 83, paragraph 4, of the Regulation). The first two refer to the violation of the legal basis and transparency, while the third concerns the violation of art. 25 of the Regulation.

As for the specific gravity, the elements to be taken into consideration are: a) nature, gravity and duration of the violation (art. 83, par. 2, letter a), of the Regulation); b) intentional or negligent character of the violation (art. 83, par. 2, letter b), of the Regulation); c) categories of personal data affected by the violation (art. 83, par. 2, letter g), of the Regulation). 

In the case in question, with reference to the three violations linked by the principle of unity of action (legal basis, transparency and data protection by design and data protection by default), the seriousness of the violations must be considered to be high given that: i) the nature of the violations relates to two fundamental principles (accountability and transparency), namely, on the one hand, the inability of the controller to demonstrate that he had identified the legal bases of the processing before the processing began and, on the other hand, the failure to provide appropriate information to the data subject, in particular with regard to the purpose of the two distinct types of processing (“Chatbot Interaction” and “Model Development”, and the involvement in the processing of data, such as that of minors, in addition to those necessary to satisfy the purpose of the processing of providing the service; ii) the nature of the processing involves significantly high risks as it is connected to an innovative, disruptive and rapidly expanding technology; iii) the object of the processing is of a cross-border nature of global scope with effects that are practically uncontrollable by the data subjects; iv) the purpose of the processing falls within the Company's core business; v) the number of Italian data subjects involved cannot be quantified with certainty, but, in general terms, it can be reasonably assumed that it is a very high number, since the information on the Google App Store (Google Play) shows that the application has exceeded 10 million downloads (leaving room for the assumption that a similar figure, although not found, can be recorded for downloads made on the Apple Store), while academic sources (Shikhar Ghosh, Replika: Embodying AI, Harvard Business School, Faculty & Research) report that the Company had already reached 10 million users in January 2022; vi) the nature of the data concerned, given the very nature of the chatbot (which still today presents itself as “an AI companion always ready to chat when you need an empathetic friend”), also particular data and, in the absence of age verification mechanisms and data filtering systems, personal information relating to minors. The duration of the violation is significant given that the app was released to the public in November 2017; in fact, the circumstance that the success of the chatbot is to be placed temporally in a later period does not constitute an element suitable to counterbalance the judgment of high severity, since the end of the violation depended on and coincided with the emergency intervention of the Guarantor. All violations must be considered negligent in nature. As stated by the Art. 29 Working Group, in the guidelines regarding the application and provision of administrative pecuniary sanctions for the purposes of Regulation (EU) no. 2016/679, adopted on 3 October 2017 and implemented by the EDPB on 25 May 2018 (WP 253 guidelines), intentional conduct refers to both awareness and intent (consciousness and will) to commit an offence, while negligent conduct lacks the intention to cause the infringement despite the failure to comply with a duty of care. The Court of Justice of the European Union (CJEU), with a recent ruling (judgment C-807/21 of 5 December 2023), has established that it is the supervisory authority's responsibility to establish that an infringement has been committed intentionally or negligently by the data controller, as only unlawful infringements can lead to the imposition of an administrative pecuniary sanction. In this regard, it should be noted that, while it is true that the CJEU has established in the aforementioned decision that art. 83 of the Regulation does not allow for the imposition of an administrative pecuniary sanction unless it is established that such infringement was committed intentionally or negligently by the data controller (see par. 75), it is also true that the Court itself upheld the basic principle of “ignorantia legis non excusat”, stating that “a data controller may be sanctioned for conduct falling within the scope of the GDPR if the data controller could not have been unaware of the unlawful nature of his conduct, regardless of whether he was aware of violating the provisions of the GDPR” (see par. 76). This principle had already been stated by the Court of Justice in another case (judgment C-601/16 of 25 March 2021, paragraphs 97 and 98) in which it had held that “an undertaking may be penalised for conduct falling within the scope of Article 101(1) TFEU where that undertaking could not have been unaware of the anti-competitive nature of its conduct, regardless of whether or not it was aware that it was infringing the competition rules of the Treaty (see, to that effect, judgment of 18 June 2013, Schenker & Co. and Others, C-681/11, EU:C:2013:404, paragraph 37). It follows that the fact that that undertaking has wrongly characterised in law its conduct on which the finding of the infringement is based cannot have the effect of exempting it from the imposition of a fine, since it could not have been unaware of the anti-competitive nature of its conduct”. anti-competitive nature of such conduct” (judgment of 18 June 2013, Schenker & Co. and others, C 681/11, EU:C:2013:404, paragraph 38). In this case, it is considered that Luka could not, at the time when its service was made available (also) to users located in the European Union and in particular in Italy, evade a duty to know and apply the Regulation which, as is known, protects a fundamental right provided for and protected by art. 8 of the Charter of Fundamental Rights of the European Union. In light of the circumstances of the specific case, the context in which the owner operates and the disruptive and rapidly expanding technology that characterises the activity carried out by the same, it is considered that the failure to comply with the processing of personal data with the European Union legislation constitutes the negligence underlying the concept of fault and demonstrates the existence of such a subjective element on the part of the Company. Furthermore, such fault must be considered serious precisely because of the breadth and innovative nature of the service offered, which involves large-scale processing of personal data worldwide.

Also for the purposes of quantifying the administrative pecuniary sanction, the aggravating factors referred to in art. 83, par. 2, letters d) and f) of the Regulation are relevant.

With regard to the first profile, the degree of responsibility of the data controller must be considered high due to the failure to adopt, at the time of launching the service, suitable technical and organizational measures to mitigate the risks for the rights and freedoms of the data subjects and to attribute to them the exercise of the prerogatives referred to in Chapter III of the Regulation. With regard to the second circumstance, with reference to the degree of cooperation, it should be noted that the Company, despite having responded to the request for information, did not produce any defense brief following the notification of the dispute pursuant to art. 166 of the Code, thereby giving rise to poor cooperation with the Authority. 

For the purposes of adopting the administrative sanction, account is taken, as a mitigating factor, of the measures implemented by the data controller to remedy the violation and mitigate its possible negative effects (Article 83, paragraph 2, letter f, of the Regulation), in particular:

-    the updates to the privacy policy, both immediately following provision 39/2023 and subsequently, in particular with reference to the latest version of the same dated 23 February 2024, as described in the previous paragraph, although such updates are not considered exhaustive;

-    the implementation of age gate mechanisms described in the previous paragraph, although not exhaustive.

In light of the aforementioned elements, assessed as a whole, in the absence of data relating to the total annual worldwide turnover of the previous financial year of the Company, it is deemed appropriate to determine, pursuant to Article 83, paragraph 3, of the Regulation, the total amount of the administrative pecuniary sanction in Euro 5,000,000.00 (five million), equal to half of the maximum fixed fine provided for by art. 83, par. 5, of the Regulation. This amount is determined in the following terms:

•    pursuant to art. 83, par. 3, of the Regulation, considering the unique nature of the conduct, since it concerns connected treatments for the reasons stated above, the amount of the pecuniary sanction for the most serious violation of art. 5, par. 1, letter a), 12 and 13, of the Regulation, is calculated in the amount of Euro 3,000,000.00;

•    the sanction is increased for the violation of art. 5, par. 1, letter a) and 6 of the Regulation in an amount equal to Euro 1,000,000.00;

•    the penalty is increased for the violation of art. 5, par. 1, letter c) and art. 25, par. 1, of the Regulation in an amount equal to €1,000,000.00;

This administrative pecuniary penalty is considered, pursuant to art. 83, par. 1, of the Regulation, effective, proportionate and dissuasive.

Taking into account the particular sensitivity of the data processed, it is believed that the accessory sanction of publication on the website of the Guarantor of this provision, provided for by art. 166, paragraph 7 of the Code and by art. 16 of the Guarantor Regulation no. 1/2019, should be applied; this in light of the nature and severity of the violations ascertained, particularly taking into account that these are large-scale processing involving a large number of interested parties, the risks in terms of protection of personal data connected to making available to the public a service based on an innovative and complex technology in the absence of the necessary safeguards. Furthermore, it is believed that there is a general interest with respect to the topic of generative artificial intelligence which requires the widest possible knowledge of the position of the Authority on the matter.

Finally, it is believed that the conditions set out in art. 17 of Regulation no. 1/2019 concerning internal procedures with external relevance, aimed at carrying out the tasks and exercising the powers delegated to the Guarantor, for the annotation of the violations detected here in the internal register of the Authority, provided for by art. 57, par. 1, letter u) of the Regulation.

GIVEN ALL THE ABOVE, THE GUARANTOR

pursuant to art. 57, par. 1, letter f), of the Regulation, declares the processing described in the terms of the motivation, carried out by Luka Inc., with registered office in 490 Post St Suite 526, San Francisco, California, United States of America, to be unlawful for the violation of articles 5, par. 1, letter a) (with reference to both the principle of lawfulness and transparency), 6, 12, 13, 5, par. 1, letter c), 24 and 25, par. 1, of the Regulation and, consequently,

a)    pursuant to art. 58, par. 2, letter d) of the Regulation, orders the Company, within thirty days of notification of the provision, to conform the processing to the provisions of the Regulation, in particular to conform the privacy policy to art. 5, par. 1, letter a), 12 and 13 of the Regulation as well as to conform the age verification system to art. 5, par. 1, letter c), 24 and 25 of the Regulation, remedying the gaps respectively indicated in paragraphs 5 and 6 of this provision;

b)    pursuant to art. 157 of the Code, orders the Company to communicate to the Authority, within sixty days of notification of this provision, the initiatives undertaken in order to implement the corrective measure referred to in the preceding point; any failure to comply with the provisions of this point may result in the application of the administrative pecuniary sanction provided for by art. 83, par. 5, of the Regulation

ORDERS

to Luka Inc., with registered office at 490 Post St Suite 526, San Francisco, California, United States of America, to pay the total sum of Euro 5,000,000.00 (five million) as an administrative pecuniary sanction for violations of art. 5, par. 1, letter a), 6; art. 5, par. 1, letter a), 12, 13, 5, par. 1, letter c), 24 and 25, par.1, of the Regulation, representing that the offender, pursuant to art. 166, paragraph 8, of the Code has the right to settle the dispute by paying, within sixty days, an amount equal to half of the fine imposed.

ORDERS

a) the aforementioned Company, in the event of failure to settle the dispute pursuant to art. 166, paragraph 8, of the Code, to pay the sum of Euro 5,000,000.00 (five million), according to the methods indicated in the attachment, within 60 days of notification of this provision, under penalty of adopting the consequent executive acts pursuant to art. 27 of Law no. 689/1981.

ORDERS

a) the publication of this provision, pursuant to art. 154-bis of the Code and 37 of Regulation no. 1/2019;

b) the application of the accessory sanction of the publication on the website of the Guarantor of this injunction order, as provided for by art. 166, paragraph 7 of the Code and 16 of the Guarantor Regulation no. 1/2019;

c) the annotation of this provision in the internal register of the Authority - provided for by art. 57, paragraph 1, letter u), of the Regulation, as well as by art. 17 of Regulation no. 1/2019 concerning internal procedures having external relevance, aimed at carrying out the tasks and exercising the powers delegated to the Guarantor - relating to the violations and measures adopted in accordance with art. 58, paragraph 2, of the Regulation itself.

The Authority reserves the right to examine and verify in an independent proceeding the profiles concerning the lawfulness of the processing carried out by Luka Inc., with specific reference to the legal bases of the processing of personal data relating to the entire life cycle of the generative artificial intelligence system underlying the Replika service.

Pursuant to art. 78 of the Regulation, as well as art. 152 of the Code and 10 of Legislative Decree no. 150 of 1 September 2011, an objection to this provision may be lodged with the ordinary judicial authority, with an appeal filed with the ordinary court of the place where the data controller is resident, within thirty days from the date of communication of the provision itself, or sixty days if the appellant resides abroad.

Rome, 10 April 2025

THE PRESIDENT
Stanzione

THE REPORTER
Scorza

THE ACTING SECRETARY GENERAL
Filippi








GUARANTEE FOR THE PROTECTION OF PERSONAL DATA

AT today's meeting, attended by President Pasquale Stanzione, Vice-President Ginevra Cerrina Feroni, Board Members Agostino Ghiglia and Guido Scorza, members, and Acting Secretary General Claudio Filippi;

HAVING REGARD TO Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation, hereinafter referred to as the ‘Regulation’);

HAVING REGARD TO the Personal Data Protection Code (Legislative Decree No. 196 of 30 June 2003), as amended by Legislative Decree No. 101 of 10 August 2018, laying down provisions for the adaptation of the national legal system to the aforementioned Regulation (hereinafter referred to as the 'Code');

HAVING REGARD TO Regulation No. 1/2019 concerning internal procedures with external relevance, aimed at fulfilling the tasks and exercising the powers assigned to the Guarantor for the protection of personal data (Italian Data Protection Authority), approved by Resolution No. 98 of 4 April 2019, published in the Official Journal No. 106 of 8 May 2019 and at www.gpdp. it, web doc. No. 9107633 (hereinafter 'Guarantor's Regulation No. 1/2019');

HAVING REGARD TO the documentation on record;

HAVING REGARD TO the observations made by the Secretary-General pursuant to Article 15 of Garante's Regulation No. 1/2000;

REPORTER Guido Scorza;    

1.    INTRODUCTION

The proceedings originated from an investigation initiated by the Guarantor of its own motion following the publication of press reports and preliminary fact-finding conducted on the Replika service (https://replika.com/), a chatbot with a written and voice interface developed and managed by the US company Luka Inc.  (hereinafter 'Luka' or the 'Company') and based on a generative AI system.

Replika is described as a chatbot that can improve the user's mood and emotional well-being by helping them understand their thoughts and feelings, track their mood, learn coping skills, reduce anxiety, and work towards goals such as positive thinking, stress management, socializing, and finding love. Replika creates a 'virtual companion' that the user can decide to set up as a friend, therapist, romantic partner or mentor. 

Replika uses a Large Language Model (LLM) system that is constantly fed and improved through interaction with users.

For the purposes of this decision, 'generative artificial intelligence' means the field of artificial intelligence that focuses on creating new and original content from input data in response to user requests (prompts), through the use of predominantly neural algorithms. ‘Neural network’ means a standard computational model applicable in a wide variety of contexts that allows the recognition of objects, shapes or patterns within a given data set or data set (e.g. a human face in a photograph). Generative artificial intelligence algorithms are used in a wide range of applications, including the recognition and generation of images, voice or music tracks, text and videos.

An example of generative artificial intelligence is large language models. For the purposes of this measure, 'Large Language Model' means a probabilistic model of a natural language, such as English or Italian, based on the assumption that all natural languages are highly redundant and correlated; this gives LLM the ability to identify the word or symbol that is most likely to follow a given piece of data.

In light of the above, the Guarantor launched an investigation on its own initiative, noting that Luka's processing of personal data in the context of the Replika service could give rise to an infringement of personal data protection legislation, with particular reference to: the privacy policy and the transparency obligations; the absence in the privacy policy of a specific indication of the legal basis for the processing in relation to the various processing operations carried out; the legal basis for the processing of minors' personal data, since in this case it must be excluded that it could be based on the performance of a contract; the absence of any filter to verify the age of users, both when accessing the service (by registering an account) and during interaction with the chatbot; the delivery, through the chatbot, of content that conflicts with the protections that should be ensured to minors and, more generally, to all vulnerable individuals.

In this context, on 2 February 2023, having found that the processing of personal data by Luka as regards the Replika service could give rise to an infringement of Articles 5, 6, 8, 9 and 25 of the Regulation and posed concrete risks to minors, also due to the fact that the responses provided were not in line with the enhanced safeguards to be ensured for minors and vulnerable individuals, the President of the Garante, pursuant to Article 5(8) of the Garante's Regulation No. 1/2000,  adopted an urgent measure (No. 39/2023, Reg. No. 18321/2023) against Luka to temporarily limit the processing of personal data of data subjects in Italy, pursuant to Article 58(2)(f) of the Regulation.

Subsequently, by decision No. 280 of 22 June 2023 (Reg. No. 104960/23), the Garante decided to suspend decision No. 39/2023 temporarily limiting the processing, provided that the controller, pursuant to Article 58(2)(d) of the Regulation, adopted appropriate measures to ensure that the processing of personal data within the Replika service was carried out in accordance with the legislation on the protection of personal data. In particular, the Garante ordered the data controller to:

1.    present an updated privacy policy to all users in Italy before registration and before accessing the Replika service;

2.    implement an age gate mechanism on all service registration pages;

3.    implement a ‘cooling-off period’ to prevent minors from entering a different date of birth when they are denied access to the services;

4.    make it possible for users in Italy to easily and effectively exercise their rights regarding personal data protection, including the right to object to the processing of personal data and to request access, rectification and erasure of data;

5.    submit to the Garante, fifteen days before the date scheduled for the opening of the service to Italian users, a plan for the development of a process aimed at preventing access to the service by persons under the age of 18, possibly supported by a language analysis mechanism with subsequent blocking effect;

6.    submit to the Garante, fifteen days before the service being available again to Italian users, a plan for the implementation of functions that allow users to report inappropriate content to prevent the Replika chatbot from providing such content, such as the possibility of flagging specific responses as inappropriate and providing feedback on the user's experience during the session.

The Garante indicated specific deadlines for the implementation of the above requirements, establishing that those referred to in points 1 to 4 had to be fully complied with by 28 July 2023, and that those referred to in points 5 and 6 had to be implemented within fifteen days of the date of the service being available again to Italian users.

2.    LUKA'S REPLIES TO DECISIONS NO. 39/2023 AND NO. 280/2023 

In a letter dated 3 March 2023 (Reg. No. 38795/23), the Company replied that it had promptly taken steps to comply with the Garante's requests, in particular to comply with the request for temporary limitation of processing for users located in Italy, by immediately blocking access to the Replika service from Italy, both through the app and through its website.

Luka also stated that it had launched a series of initiatives aimed at implementing the Garante's requests in a concrete manner, including through the involvement of external consultants and experts in the field. In particular, the Company stated that it had initiated a number of assessments, actions and processes intended to:

-    implement more robust user age verification mechanisms to better ensure that minors in Italy do not use the ‘Replika’ service, which is reserved for adults; in addition to the age gate tools already in use, the Company undertook to introduce automated measures aimed at recognising underage users based on the analysis of indicators contained in conversations with the chatbot;

-    implement algorithms and processes for moderating inappropriate content, in line with best practice;

-    ensure compliance with the Regulation by, among other things, updating the register of processing activities, reviewing and updating data protection impact assessments (DPIAs), and updating the privacy policy relating to the service, in order to increase transparency for users.

By letter dated 31 March 2023 (Reg. No. 55533/23), the Company requested the corrective measure of the temporary limitation imposed by urgent measure No. 39/2023 be lifted, specifying:

-    that the Replika service was designed to limit the extent of personal data processing, in accordance with the principles set out in Article 5 of the Regulation, including i) minimising the collection of user registration data (name, email address, date of birth – to verify age – and any third-party login data); ii) the adoption of data retention and erasure procedures that strike a balance between the need to provide the user with a smooth experience and the need to minimise the personal information that remains accessible; iii) the design of proprietary artificial intelligence (AI) models to interact with users; iv) the non-sharing of user conversations with third parties other than the Company's essential service providers, who are bound by confidentiality obligations; v) the implementation of strict controls designed to limit access to personal data by its own staff; vi) the non-use of user conversations for advertising or marketing purposes; 

-    that it does not offer the service to minors and to base the processing of users' personal data on the legal basis of contract performance; 

-    that it implemented, following the provision of the Garante, several measures aimed at preventing minors from accessing the service in violation of the Company's terms;

-    that it included its mobile application in the Apple App Store with an age rating of 17 or older, which is the highest age rating allowed by Apple;

-    that it does not collect special categories of personal data, given that the sharing of special categories of personal data by users during their interaction with the chatbot is spontaneous and must therefore be qualified as covered by explicit consent to processing, in accordance with Article 9 of the Regulation;

-    that it takes its data protection obligations seriously and has integrated data protection into the design of the service, in accordance with Article 25 of the Regulation, and that it will continue to ‘develop and improve its policies and procedures to provide users with a consistent, secure and rewarding experience’. 

With specific reference to the Garante's decision, the Company stated:

-    that it promptly blocked access to the Replika service to individuals located in Italy;

-    that it strengthened measures to prevent access to the service by minors under the age of 18, in particular by:  i) introducing an age gate on all service registration pages requiring users to indicate a date of birth greater than or equal to 18 years of age in order to access the service; ii) providing for a ‘cooling-off period’, in line with the guidelines of data protection authorities and best practices, to prevent minors from entering a different date of birth when the system denies access to the service; iii) launching activities aimed at improving automated content control processes (reporting individuals likely to be under 18 and preventing use of the service until age verification is completed through more robust methods);

-    that it updated its privacy policy to address the transparency issues identified by the Garante;

-    that it continues to develop and fine-tune its content moderation practices to prevent harm to users, in particular by creating a trust and safety programme to prevent the chatbot from being involved in offensive or harmful conversations;

-    that it restricted access to conversations of a sexual nature or relating to other adult content to users active at 1 February 2023 and made such conversations unavailable to new users;

-    that it continues to endeavour to ensure compliance with the Regulation through the support of an external data protection consultant. The commitments undertaken by the Company include: i) updating and maintaining the Company's record of processing activities; ii) reviewing and updating data protection impact assessments (DPIAs), including documentation of data protection by design and by default processes; iii) refining and reviewing the Company's security policies and procedures; iv) reviewing the Company's data protection governance (including the possibility of appointing a DPO following the expansion of the Company's activities in the European Union).

The Company, by letter dated 26 April 2023 (Reg. No. 68896/23), submitted a second request for lifting the corrective measure of the temporary limitation imposed by urgent measure No. 39/2023, reiterating the measures taken, as already explained in the previous letter.

The Company, in a letter dated 14 June 2023 (Reg. No. 93675/23), further to the discussions held at the hearing on 31 May 2023, reaffirmed that it had responded promptly to Decision No. 39/2023, immediately blocking access to Replika in Italy and implementing adequate measures in response to the issues raised by the Garante in the aforementioned measure. The Company also expressed its commitment to prevent users located in Italy from engaging in conversations of a sexual nature by providing, once the service is reactivated, two versions of Replika: a free version and a paid version containing romantic but not sexual content. According to the Company, the introduction of a paid ‘romantic’ version will require additional age verification based on the user's payment card details, in line with the latest market standards for age verification mechanisms.

In a letter dated 14 July 2023 (Reg. No. 109176/23), the Company announced that it had complied with the requests set out in points 1-6 of decision No. 280/23 and, in particular, stated:

1.    in relation to the information referred to in point 1 of Decision No. 280/23, that it implemented an updated privacy policy in the registration process and prior to access to the service, and that this information would be displayed to Italian users upon reactivation of the service;

2.    with reference to the age gate mechanism referred to in point 2 of Decision No. 280/23, that it implemented an age verification system on all registration pages and that this system would be applied upon reactivation of the service;

3.    with reference to the cooling-off period referred to in point 3 of Decision No. 280/23, that it implemented a cooling-off period to prevent minors from attempting to access the service again by entering a different date of birth. This period—lasting 24 hours—is to be managed (i) by checking the credentials of a minor user's account and subsequently preventing them from entering a different date of birth and (ii) by installing a cookie to prevent minor users from re-entering a different date of birth from the same browser. The Company stated that this cooling-off period would be applied upon reactivation of the service;

4.    with regard to the exercise of the rights referred to in point 4 of Decision No. 280/23, that it provides users with a simple and effective method for exercising their data protection rights, including the right to object to the processing of their personal data and the rights to request access, rectification and erasure of their data, and that this mechanism would be applied upon reactivation of the service;

5.    with regard to the request to prepare a plan for the development of an age verification mechanism during registration referred to in point 5 of Decision No. 280/23, that it implemented processes to prevent access by minors under the age of 18, including a language analysis mechanism that requires users to reconfirm their age through the age gate process when users identify themselves as under the age of 18. If no date of birth that satisfies the age gate is provided, the user cannot access the service The Company stated that such processes would be applied in Italy upon reactivation of the service;

6.    with regard to the request to prepare a plan for the development of an age verification mechanism during the use of the service referred to in point 6 of Decision No. 280/23, the Company stated that it implemented features that allow users to report inappropriate content to prevent the Replika chatbot from presenting it again, such as the ability to flag specific responses as inappropriate and provide feedback on the user experience during the session. The Company stated that these features would be implemented in Italy upon reactivation of the service.

Luka submitted, together with the letter dated 14 July 2023, a copy of the privacy policy updated on 12 June 2023.

3. FACT-FINDING ACTIVITY  

Parallel to adoption of the precautionary measure, the Garante started gathering the info deemed necessary to carry out the fact-finding activity by sending a request for information, in line with Articles 58(1)(e) of the Regulation and 157 of the Code.

By letter dated 6 April 2023 (Reg. No. 58925/23), the Garante sent a request for information to Luka asking for details on how Replika works (categories of personal data processed and source from which they are collected; method used for collection; how the data collected is processed; where the data is stored; security measures adopted; processing of user data for system training purposes or for other purposes pursued by Luka), the processing of users' personal data (legal basis; storage period; minimum age for accessing the service provided by Replika; DPIA; appointment of a representative pursuant to Article 27 of the Regulation; procedures for managing rights pursuant to Articles 12–22 of the Regulation; legal basis and guarantees of adequacy pursuant to Chapter V of the Regulation, where applicable; clarifications regarding automated processing pursuant to Article 22 of the Regulation), and age verification measures for access to the service on the date of notification of urgent measure No. 39/23.

With regard to this request, in a letter dated 8 May 2023 (Reg. No. 74173/23), the Company, after initially claiming that it has a single establishment in the European Union in the Netherlands, stated that:

-    it uses the messages and content that the user sends to the chatbot to enable conversations with that user (the ‘Chatbot Interaction’). In relation to Chatbot Interaction, the content of the database may include basic profile information, conversation topics, questions that the user may ask, and selected preferences or interests. When a user sends a message, the model analyses the text to enable the chatbot to generate a response based on the latest messages in the conversation. The Company has also made it clear that it uses a database containing all the info sent through the chat to create de-identified data and fine-tune the LLM that forms the basis of the chatbot (‘Model Development’). The section of the database used as a source to create de-identified data is limited to: 1) user ‘Reactions’ (‘like’, ‘dislike’, ‘love’, ‘funny’, ‘meaningless’ or ‘offensive’), if the user chooses to make such a selection; 2) ‘Feedback’ on user satisfaction levels with the conversation (‘happy’, ‘neutral’ or ‘sad’); 3) ‘Snippets’, i.e. small parts of user conversations that provide context for interpreting Reactions and Feedback. The information used by the Company for Model Development does not identify specific individuals and cannot be associated with specific individuals (‘De-identified Data’) as any personal identifiers (such as names, addresses, email addresses, telephone numbers and identification numbers) that may be contained in conversation snippets are removed and the snippets are ‘shuffled’ in a randomised fashion;

-    it collects all personal data described above from users‘ interactions with the service;

-    it uses a system for collecting (‘Reactions’, ‘Feedback’ and ‘Snippets’) and processing in real time users’ interactions with the chatbot using webhooks, i.e. automated tools that capture such information and send it to the Company's servers;

-    it follows, in the processing of ‘De-identified Data’ for Model Development, the following steps: 1) data collection, as described above; 2) pre-processing consisting of cleaning, structuring and removing any personal identification data from such data, in order to safeguard privacy (through aggregation and randomisation techniques); 3) labelling of pre-processed data; 4) analysis and development to evaluate the performance of the LLM, identify patterns and develop filters that prevent the model from producing outputs with inappropriate content; 5) testing and validation (regular testing and validation against predefined criteria);

-    it stores personal data on encrypted databases hosted by Amazon Web Services, Inc. in the United States;

-    it does not use personal data provided by users for Model Development;

-    it employs technical and organisational measures to protect the security of personal data and ‘De-identified Data’ from unauthorised access, use and disclosure. These measures include encryption, access controls, vulnerability management, pre-processing and anonymisation of ‘Snippets’, ‘Reactions’ and ‘Feedback’, training and possible disciplinary measures in the event of non-compliance with the measures by the Company's personnel;

-    it relies on the contractual legal basis for ‘Chatbot Interaction’ as the processing of user data is necessary for the provision of the service, in accordance with the Terms of Service. This processing includes the creation and maintenance of user account profiles, the facilitation of payments and transactions, and the processing of data entered by users to generate the chatbot's response;

-    it relies on the legal basis of legitimate interest for ‘Model Development’;

-    it retains data for ‘as long as it deems reasonably necessary to provide users with a safe, enjoyable and successful experience on the platform’, in accordance with the principle of minimisation;

-    it retains ‘Chatbot Interaction’ data for ‘a period sufficient to enable the retrieval of information to ensure a seamless conversation experience for users with the chatbot, in line with user expectations’;

-    it retains [without further specification, editor's note] user data to create ‘De-identified Data’ for ‘Model Development’;

-    that the minimum age required to use the Replika service is 18 years;

-    that there is no contradiction between the previous point and the statement contained in the Company's privacy policy, which reads: ‘We do not knowingly collect Personal Data from children under the age of 13. If you are under 13, please do not submit any Personal Data through the Services’, as this statement has been included as required by US federal law (COPPA);

-    the Replika mobile application included an age gate that prevented minors under the age of 18 from accessing the service even before the Garante's Decision of 2 February 2023. The Company has also listed its application in the Apple App Store with an age rating of 17+, which is the highest age rating allowed by Apple;

-    all adult content has been placed behind a paywall, out of reach of minors;

-    following the Decision of 2 February 2023, the Company deliberately improved the measures aimed at preventing minors under the age of 18 from accessing the service;

-    it has not designated a representative pursuant to Article 27 of the Regulation as the Company has an establishment in the European Union;

-    with regard to the exercise of the rights of data subjects, the relevant information is provided through a privacy policy published on the Company's website and in the App. Access, rectification and erasure may be requested by users, who may also object to and restrict the processing of any personal data that is not necessary for the provision of the service. Requests are evaluated on a case-by-case basis;

-    it does not engage in any profiling of data subjects or take automated decisions that have legal or equally significant effects;

-    it collects personal data directly from users and does not transfer them from Italy or the European Union in accordance with Chapter V of the Regulation and has entered into data processing agreements with data processors, which include standard contractual clauses where required;

-    for the purpose of content control, it has trained its models to prevent the emergence and escalation of inappropriate content or inappropriate responses. As part of this process, the Company uses open-source data sets specifically designed and made available to the artificial intelligence research community for the purpose of improving the safety and robustness of machine learning models. The Company has also developed, and continues to improve and refine, filters that recognise keywords, phrases and patterns associated with harmful behaviour, such as self-harm, insults or murder. The filters trigger the LLM to respond appropriately to such content, for example by changing the topic of the conversation or providing users with self-help resources. The Company also uses human review in both the evaluation of the AI model and the development of filters;

-    it uses other methods to control content that is inappropriate or conflicts with the app's Terms of Service, including: 1) placing so-called romantic content behind a paywall and disabling sexually explicit content for new users; 2) allowing users to report specific content or conversations as offensive in real time and using such reports to improve the models and prevent them from developing similar content in the future; 3) prohibiting users, in the Terms of Service, from uploading illegal, harmful and threatening content.

Along with its reply of 8 May, the Company provided a copy of the privacy policy applicable on 2 February 2023, its updated version dated 22 March 2023, and a copy of the impact assessment (undated and unsigned).

By letter dated 27 February 2024 (Reg. No. 23744/24), the Garante informed the Company of the initiation of proceedings for the adoption of corrective measures and sanctions pursuant to Article 166(5) of the Code and Article 12 of the Garante's Regulation No. 1/2019, alleging that Luka had infringed Articles 5, 6, 7, 8, 12, 13, 24 and 25(1) of the Regulation in relation to the processing of personal data carried out by the Company through the Replika service as at 2 February 2023.

The Company did not reply to the notice of initiation of proceedings nor did it request to be heard pursuant to Article 166(6) of the Code and Article 13 of the Garante's Regulation No. 1/2019. 

In the notice of initiation of proceedings, which is hereby expressly and fully referred to, the Garante alleged three infringements against the Company on the basis of the critical issues identified in urgent measure No. 39/2023. The evaluation conducted by the Garante focused on the facts, the processing operations and the measures implemented by Luka as at 2 February 2023.

With regard to the failure to identify the conditions governing the lawfulness of the processing, the Garante found that the privacy policy that was online at the time of the adoption of the urgent measure—updated on 5 July 2022—did not provide a granular description of the legal basis for the various processing operations carried out by the Company in connection with the Replika service. The reference to the legal bases for the performance of a contract (Article 6(1)(b) of the Regulation) and the consent of the data subjects (Article 6(1)(a) of the Regulation), as well as to a generic authorisation (‘authorisation’, not obligation) under the law, were not in fact linked or attributable to specific processing operations (so-called granularity), making it impossible to identify and evaluate the suitability of those legal bases. Furthermore, the privacy policy dated 5 July 2022, effective on 2 February 2023, did not contain any reference to the legal basis underlying the processing of personal data for the development of the LLM that powers the chatbot, nor did the documentation subsequently produced—notably the privacy policy—even in the version updated on 22 March 2023, and the DPIA, include elements demonstrating that the Company had identified a legal basis for this purpose prior to 2 February 2023.

In light of the above, the Garante alleged that Luka had possibly infringed Article 5(1)(a) and Article 6 of the Regulation by failing to identify, as at 2 February 2023, the legal bases for the various processing operations carried out through the Replika service.

With regard to transparency obligations, the Garante's evaluation concerned the privacy policy applicable as at 2 February 2023, i.e. the version updated on 5 July 2022. From a formal point of view, the Garante, in the act initiating the proceedings, found that as at 2 February 2023 the privacy policy was only available in English (including for minors) and was not easily accessible.  From a content point of view, it was found that as at 2 February 2023, the privacy policy:

-    did not indicate the legal basis for each processing activity and type of data processed;

-    did not indicate the purposes of the processing with specific reference to the two distinct types of processing, namely processing for ‘Chatbot Interaction’ and processing for ‘Model Development’; 

-    in the sections ‘People mentioned in the chat’ and ‘Integration with your Instagram account’, two categories of personal data processed for the purpose of enabling user conversations were indicated; 

-    did not clarify that the service was offered exclusively to adults, since, as mentioned above, the privacy policy only included a reference to minors under the age of 13 in compliance with the requirements of COPPA (Children's Online Privacy Protection Act);

-    did not provide any specific information regarding the storage period of personal data or the criteria used to determine such period;

-    did not clarify whether personal data were transferred outside the EEA and, in such case, what the legal basis and guarantees of adequacy referred to in Chapter V of the Regulation were. In particular, the text of the privacy policy (see, in particular, the statement ‘By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen’) is in clear contradiction with the statement made by the same Company in its letter dated 8 May 2023 (Reg. No. 74173/23), where it is stated that, since the criterion of establishment in the European Union does not apply, no transfer of personal data from the European Union (in particular, from Italy) to the US is possible under Chapter V of the Regulation;

-    in section 6 entitled ‘Your data protection rights’, the privacy policy provided specific information on the right set out in Article 22 of the Regulation, even though the provision was not expressly referred to. This reference (no longer present in the version dated 22 March 2023) was sufficient to lead users to believe that their personal data were subject to automated decision-making in violation of the principles of transparency and fairness. This circumstance was denied by the same data controller in its reply (Reg. No. 74173/23), in which it argued that ‘although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling that have legal or similar effects within the meaning of Article 22 of the Regulation’.

In light of the above, the Garante alleged that Luka had possibly infringed Article 5(1)(a), 6, 12 and 13 of the Regulation, given that, as at 2 February 2023, the privacy policy relating to the Replika service did not comply with the general obligations and principles of transparency and was provided in such a way and at such a time that it could not be readily accessed by users.
Lastly, with regard to the lack of mechanisms for age verification of minors, the Garante alleged the failure to implement measures ensuring specific protection for minors in relation to access to and use of the Replika service as at 2 February 2023. In particular, the following were found to be lacking:

-    a user age verification procedure (the system only required name, email address and gender), with the consequent risk of minors being presented with responses that were unsuitable for their level of development and self-awareness, including sexually explicit content;

-    mechanisms to prohibit or block access even when the user clearly stated that they were a minor; in addition, the chatbot provided responses that were clearly contrary to the protections that should be ensured for minors and, more generally, for all vulnerable individuals.

The Garante, when initiating the proceedings, acknowledged that the Company had implemented age verification mechanisms following the request made by the Garante in the temporary limitation decision adopted as a matter of urgency on 2 February 2023. In particular, during the exchanges that followed the adoption of the aforementioned decision and with specific reference to age verification, the data controller stated that it had implemented an age gate on all registration pages for the Services aimed at restricting access to users who are at least 18 years of age and that the age verification mechanism includes a ‘cooling-off period’ aimed at preventing users—once they have ascertained that it is impossible to access the service by entering their real personal data—from immediately entering a different date of birth that would allow them to access the service. The Company also stated that a process was being developed to use language analysis to identify and prevent the use of the Services by persons under the age of 18.

In light of the above, the Garante alleged that Luka had possibly infringed Article 5(1)(c), 6; 7; 8; 24 and 25(1) of the Regulation for failing to put in place appropriate systems to verify the age of individuals as at 2 February 2023.

4. EXISTENCE OF EU JURISDICTION AND COMPETENCE OF THE GARANTE

As a preliminary observation, the Garante considers it appropriate to address the issues relating to the applicability of European data protection legislation to the service offered by Luka and to its own competence, also taking into account the objections raised by the Company in its reply dated 8 May 2023 to the request for information sent by the Garante.

Article 3 of the Regulation governs the territorial scope of application of the legislation, establishing different criteria depending on whether or not the data controller is established in the European Union. 

In the first case (Article 3(1), known as the establishment criterion), the Regulation applies regardless of whether the processing is carried out in the Union or not, and competence is determined in accordance with the one-stop-shop mechanism, pursuant to Article 56 of the Regulation.

In the second case (Article 3(2), known as the targeting criterion), the Regulation applies to the processing of personal data of data subjects who are in the Union insofar as the processing activities relate to: i) the provision of goods or services to data subjects in the Union (Article 3(2)(a) of the Regulation); ii) the monitoring of the behaviour of data subjects located in the Union insofar as such behaviour takes place in the same Union (Article 3(2)(b) of the Regulation).

The Company stated in the above-mentioned letter that it has a single establishment in the European Union in the Netherlands, reporting that it has ‘a group of employees located in the Netherlands, including a number of decision-makers involved in cross-border data processing for the development of LLM and of the product’ and that ‘the Company's employees located in the Netherlands are involved in decisions concerning the processing of personal data by the Company and the operation of LLM globally, including decisions concerning the minimum portion of users located in Italy’. The existence of a Dutch establishment in the European Union would entail the application of the one-stop-shop mechanism and the competence of the Dutch data protection authority as lead supervisory authority in cooperation with the authorities concerned.

However, this statement is not supported by any evidence. In fact, both in the privacy policy published on Replika's website as at 2 February 2023 (version updated on 5 July 2022) and in subsequent versions thereof (including the current version updated on 23 February 2024), there is no mention of an establishment of the company in the Netherlands; likewise, no mention can be found in the Terms of Service (neither in the version updated on 14 September 2022 nor in the current version, updated on 7 February 2023), which, on the contrary, states that Luka is ‘a software company who designed and built Replika, incorporated in Delaware, and operating in San Francisco, CA’.

Furthermore, the statements in the letter of 8 May 2023 are extremely vague, as they do not even indicate the name and registered office of the company allegedly established in the European Union (thus making it impossible to carry out any checks in cooperation with the Dutch supervisory authority pursuant to Article 61 of the Regulation) and are not supported by any document evidence (e.g. the Dutch company's articles of association or chamber of commerce registration).

As things stand, therefore, the Garante considers that no evidence has been provided to effectively demonstrate the existence of an establishment of the Company in the European Union and, as a result, the applicability of the establishment criterion under Article 3(1) of the Regulation and of the one-stop-shop mechanism with the competence of the Dutch data protection authority.

In the present case, the existence of EU jurisdiction and the competence of the Garante must be ascertained on the basis of the targeting criterion set out in Article 3(2), of the Regulation: more specifically, it must therefore be ascertained, as a preliminary matter, whether the Replika service can be considered as offered to data subjects located in the European Union for the purposes of the applicability of point (a) of the aforementioned Article 3 of the Regulation.

In this regard, reference is made to the ‘Guidelines 3/2018 on territorial scope’, adopted by the European Data Protection Board (EDPB) on 12 November 2019, which provide that the ‘controller [...] demonstrates its intention to offer goods or services to a data subject located in the Union’ (see paragraph 2(a) of the aforementioned Guidelines) and the case law of the Court of Justice of the European Union (judgment Pammer/Reederei Karl Schlüter GmbH & Co and Hotel Alpenhof/Heller – joined cases C-585/08 and C-144/09), which may be taken into account in order to determine whether a commercial activity carried out by an entity is directed to a Member State, among which the fact that the European Union is mentioned in connection with the goods or services offered, the international nature of the activity or the launch of advertising and marketing campaigns aimed at the public of an EU country.

In the present case, the evidence that the Replika service was offered to data subjects located in the European Union and, in particular, in Italy as at 2 February 2023, is clear from the Company's initial reply to the order for temporary limitation issued by the Garante in its urgent measure  No. 39/2023, which states (see letter dated 3 March 2023, p. 1) that ‘the Company promptly complied with the request for temporary limitation of processing for users established in Italy, immediately blocking access to both the app and the service website from Italy’.

Having demonstrated the territorial applicability of the Regulation and the competence of the Garante in the manner and within the terms set out above, the following remarks are made.

The processing of personal data carried out by Luka qualifies as cross-border processing of personal data within the meaning of Article 4(1)(23) of the Regulation, as it is likely to affect data subjects in more than one Member State.

For this type of processing, where the controller has identified a single or main establishment in the European Union, as already explained, the cooperation mechanism described in Articles 60 et seq. of the Regulation applies, and the competence to exercise the tasks and powers referred to in Articles 57 and 58 of the Regulation lies, pursuant to Article 56(1) of the Regulation, with the lead supervisory authority, i.e. the supervisory authority of the Member State in which the single or main establishment is located.

If, on the contrary, as in the present case, the data controller does not have an establishment in the European territory, the data controller shall ‘liaise with the supervisory authorities of each Member State in which it operates through the designated representative’ (see paragraph 3.3. of the ‘Guidelines on the Lead Supervisory Authority’ adopted by the Article 29 Working Party on 13 December 2016, revised on 5 April 2017 and endorsed by the EDPB on 25 May 2018).

In fact, where a controller does not have an establishment in the European Union (or, more precisely, in the EEA), the special rule in Article 56 does not apply and the general rule set out in Article 55(1) of the Regulation applies, according to which ‘each supervisory authority shall be competent for the performance of the tasks assigned to and the exercise of the powers conferred on it in accordance with this Regulation on the territory of its own Member State’.

In the present case, as mentioned above, Luka is a company based in the United States of America and has not demonstrated that it has an establishment in the territory of the European Union. Therefore, the Garante (Italian Data Protection Authority) is competent to evaluate, as regards its own territory, the compliance with the Regulation of the processing of personal data carried out by the Company and to exercise the powers conferred on it by Article 58 of the Regulation.

5. FINDINGS OF INFRINGEMENT

5.1 ARTICLES 5(1)(A) AND 6 OF THE REGULATION

The Garante notified Luka of the infringement of Articles 5(1)(a) and 6 of the Regulation for failing, as at the date of 2 February 2023, to identify the legal bases for the different processing operations carried out through the Replika service which was provided and made available to the public in Italy on that date.

Article 5(1) of the Regulation provides that ‘Personal data shall be: a)processed lawfully, fairly and in a transparent manner in relation to the data subject (‘lawfulness, fairness and transparency’); b) collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes; further processing for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes shall, in accordance with Article 89(1), not be considered to be incompatible with the initial purposes (‘purpose limitation’); c) adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’); d) accurate and, where necessary, kept up to date; every reasonable step must be taken to ensure that personal data that are inaccurate, having regard to the purposes for which they are processed, are erased or rectified without delay (‘accuracy’); e) kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed; personal data may be stored for longer periods insofar as the personal data will be processed solely for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes in accordance with Article 89(1) subject to implementation of the appropriate technical and organisational measures required by this Regulation in order to safeguard the rights and freedoms of the data subject (‘storage limitation); f) processed in a manner that ensures appropriate security of the personal data, including protection against unauthorised or unlawful processing and against accidental loss, destruction or damage, using appropriate technical or organisational measures (‘integrity and confidentiality’)’. Paragraph 2 of the same Article provides that ‘The controller shall be responsible for, and be able to demonstrate compliance with, paragraph 1 (‘accountability’)’.

Recital 39 clarifies that ’Any processing of personal data should be lawful and fair. It should be transparent to natural persons that personal data concerning them are collected, used, consulted or otherwise processed and to what extent the personal data are or will be processed. The principle of transparency requires that any information and communication relating to the processing of those personal data be easily accessible and easy to understand, and that clear and plain language be used. That principle concerns, in particular, information to the data subjects on the identity of the controller and the purposes of the processing and further information to ensure fair and transparent processing in respect of the natural persons concerned and their right to obtain confirmation and communication of personal data concerning them which are being processed’.

Article 6 of the Regulation sets out the conditions for lawful processing by listing six possible legal bases (consent, contract, legal obligation, vital interest, public interest and legitimate interest) on which the data controller must rely to lawfully process personal data necessary for carrying out its activities. As clarified by the EDPB ‘The legal basis must be identified at the outset of processing, and information given to data subjects in line with Articles 13 and 14 must specify the legal basis. (see Guidelines 2/2019 on the processing of personal data under Article 6(1)(b) GDPR in the context of the provision of online services to data subjects)’.

The Company did not submit any defence statements or documents, pursuant to Article 166(5) of the Code, following the Garante’s notice of infringement and initiation of proceedings, and thus did not provide any counter-arguments regarding the alleged infringement related to the failure to indicate the legal basis for each of the processing activities carried out by Luka within the scope of the Replika service.

In the present case, the documentation reviewed during the investigation—particularly the text of the privacy policy published on the date of the adoption of the Garante's urgent measure, as last updated on 5 July 2022—shows that the Company failed to identify, in a granular manner, the legal basis for the different processing operations carried out by the Company within the Replika service, including the processing of data used for the development of the LLM.

The only references provided in the introductory section of the privacy policy are as follows:
‘We care about the protection and confidentiality of your data. We therefore only process your data to the extent that:

•    It is necessary to provide the Replika services you are requesting,

•    You have given your consent to the processing, or

•    We are otherwise authorized to do so under the data protection laws’.

The legal bases referred to therein—namely, the performance of a contract (Art. 6(1)(b) of the Regulation), the consent of the data subjects (Art. 6(1)(a) of the Regulation), and a legal authorisation (although the Regulation in fact requires a legal obligation, not merely an authorisation, as a legal basis)—are cited only implicitly and generically. They are not referred to specific processing operations (the so-called principle of granularity), thereby making it impossible to identify and assess their appropriateness.

Finally, neither the privacy policy nor the documents on file contain any reference to the legal basis for the processing of personal data for the purpose of developing the LLM that powered the chatbot until 2 February 2023.

Specifically, while the evidence provided by Luka is relevant, it is not conclusive. In particular, the DPIA and the privacy policy submitted on 8 May 2023 do not overcome the concerns raised by the Garante regarding the principle of lawfulness and the identification of a valid legal basis for the processing, as required respectively under Articles 5(1)(a) and 6 of the Regulation, since:

-    The privacy policy, including the later version updated on 22 March 2023, does not explicitly mention the purpose of ‘Model Development’ nor its legal basis in the table set out in paragraph 2;

-    The DPIA, while distinguishing between the two processing purposes of ‘Chatbot Interaction’ and ‘Model Development’ (par. I) and analysing their respective legal bases (par. II), does not provide a clearly identified date and therefore does not demonstrate that the identification of the lawfulness conditions under Article 6 of the Regulation occurred prior to 2 February 2023. Moreover, the DPIA refers to legitimate interest as the legal basis for processing for the purposes of ‘Model Development’ without providing any arguments relating to the so-called ‘three-step test’ required in the legitimate interest assessment. Finally, it is pointed out that the DPIA, while being an excellent accountability tool, is not the document chosen by regulators for informing data subjects about processing activities; such information must instead be provided in the privacy policy.

With reference to Article 5(1)(a) of the Regulation, attention is drawn to the principle expressed by the EDPB in its binding decision 1/2021 on transparency—also applicable to the principle of lawfulness—according to which the principles set out in Article 5 of the Regulation must be considered as a general concept which is then implemented in various provisions and specific obligations (in the case of lawfulness, in Articles 6, 7, 8, 9 and 10 of the Regulation). Therefore, according to the EDPB, it is necessary to distinguish the specific obligations arising from a principle (in this case, Article 6 of the Regulation) from the principle itself as set out in Article 5 of the Regulation, since the principle cannot be circumscribed to the specific obligation, although the specific obligation is a concretisation of the principle.

The principle of lawfulness is indeed an overarching principle that reinforces other principles (such as fairness and accountability). This is confirmed by Article 83(5) of the Regulation which allows for separate sanctions for infringing the lawfulness obligations independently of any breach of the principle itself. In this specific case, the Garante considers that there has also been an infringement of the principle of lawfulness referred to in Article 5(1)(a) of the Regulation, taking into account the gravity (lack of a clear and granular identification of the legal bases underpinning the different processing operations), the nature (this is an essential element of data processing) and the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the single specific infringement of the obligation referred to in Article 6 of the Regulation.

Based on the foregoing, the Garante considers that Luka failed, as at 2 February 2023, to identify the legal bases applicable to the different processing operations carried out through the Replika service, provided and made available to the public in Italy on that date, in breach of Articles 5(1)(a) and 6 of the Regulation.

With regard to the substantive analysis and evaluation of the legal bases under Article 6(1)(b) and (f) of the Regulation—allegedly relied upon for the use of the chatbot and the post-training of the LLM underlying the Replika service—and, more generally, the legal bases applicable throughout the entire lifecycle of the generative AI system developed by the Company, the Garante reserves the right to initiate a separate and autonomous investigation.

5.2 ARTICLES 5(1)(A), 12 AND 13 OF THE REGULATION

The Garante notified Luka of the infringement of Articles 5(1)(a), 12 and 13 of the Regulation for having provided, as at the date of 2 February 2023, a privacy policy concerning the Replika service which did not comply with the obligations and general principles on transparency established under the Regulation. 

Article 5(1)(a) of the Regulation requires that personal data be processed lawfully, fairly, and in a transparent manner in relation to the data subject (principles of lawfulness, fairness and transparency).

Article 12 of the Regulation lays down rules regarding transparency and the modalities for the exercise of rights, while Article 13 specifies the information that a data controller must provide when personal data are collected from the data subject.

On the subject of transparency, Recital 58 of the Regulation requires that any information addressed to the public or the data subject be concise, easily accessible and easy to understand, and that clear and plain language be used, and, with reference to the specific protection of children, it provides that ‘where processing is addressed to a child, should be in such a clear and plain language that the child can easily understand’.

On the issue of transparency, the Committee's guidance is also relevant, particularly Guidelines 2/2019 on the processing of personal data under Article 6(1)(b) of the GDPR in the context of the provision of online services to data subjects, where it is provided that the legal basis for the processing must not only be identified at the outset of processing, but also explicitly specified in the ‘information given to data subjects in line with Articles 13 and 14’; the Committee's Guidelines 1/2022 on data subject rights – Right of access, are also applicable. Paragraph 142 of these Guidelines affirms that ‘a controller that offers a service in a country should also offer answers in the language that is understood by the data subjects in that country’.

Finally, the Guidelines adopted by the Article 29 Working Party on 11 April 2018, clarified that ‘The concept of transparency in the GDPR is user-centric rather than legalistic and is realised by way of specific practical requirements on data controllers and processors in a number of articles. The practical (information) requirements are outlined in Articles 12 - 14 of the GDPR. (…) The transparency requirements in the GDPR apply irrespective of the legal basis for processing and throughout the life cycle of processing. This is clear from Article 12 which provides that transparency applies at the following stages of the data processing cycle:  i) before or at the start of the data processing cycle, i.e. when the personal data is being collected either from the data subject or otherwise obtained; ii) throughout the whole processing period, i.e. when communicating with data subjects about their rights; and iii) at specific points while processing is ongoing, for example when data breaches occur or in the case of material changes to the processing’.

The Company did not submit any defence statements or documents, pursuant to Article 166(5) of the Code, following the Garante’s notice of infringement and initiation of proceedings, and thus did not provide any counter-arguments in relation to the alleged infringement of the transparency obligations and general principles of transparency as required by the Regulation.

The Garante's investigation, as already noted above, focused on the privacy policy adopted and published by Luka on 2 February 2023, that is to say the version updated on 5 July 2022.
First of all, from a formal point of view, the investigation established that, as at the date of 2 February 2023, the privacy policy was available only in English, not considering the language of the country in which the service was offered, namely Italian. 

From a substantive point of view, it is noted that as at the date of 2 February 2023, the privacy policy failed to comply with the principles of fairness and transparency as it was incomplete and inaccurate.

In particular, with regard to the accuracy of the information provided to data subjects, it was found that the privacy policy:

-    did not specify in a granular manner the legal basis for each processing operation carried out, nor the type of data processed;

-    did not distinguish the purposes of the two distinct types of processing activities, namely the processing of data through ‘Chatbot Interaction’, intended to allow users to register for the service and interact with the platform, and the processing of data for ‘Model Development’, aimed at improving the security and performance of the Large Language Model (LLM) underlying the service offered (‘Model Development’);

-    did not clearly state that the service was intended exclusively for users aged 18 and above, although it encouraged users under the age of 13 not to use the service. In particular, paragraph 8 of the aforementioned privacy policy stated: ‘We do not knowingly collect Personal Data from children under the age of 13. If you are under the age of 13, please do not submit any Personal Data through the Services. We encourage parents and legal guardians to monitor their children’s Internet usage and to help enforce our Privacy Policy by instructing their children never to provide Personal Data on the Services without their permission. If you have reason to believe that a child under the age of 13 has provided Personal Data to us through the Services, please contact us, and we will endeavour to delete that information from our databases’. Although this information made it clear that the service was not intended for persons under the age of 13 (‘If you are under the age of 13, please do not submit any Personal Data through the Service’), it did not clearly specify that access to the chatbot was restricted only to users aged 18 and over, and that users between the ages of 13 and 18 were excluded. This latter circumstance was clarified by the Company only at a later stage;

-    did not provide any precise indication as to the storage period of the personal data or the criteria used to determine that period;

-    did not clarify whether personal data were transferred outside the EEA, nor did it specify, where such transfers occurred, the legal basis for processing or the adequacy measures adopted under Chapter V of the Regulation. More specifically, the information provided by the Company (namely the section of the privacy policy reading: ‘By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen’) appeared likely to mislead data subjects as to the transfer of their personal data to the USA. The absence of any transfer of personal data to third countries was confirmed by the Company itself in its letter of 8 May 2023 (Reg. No. 74173/23) concerning the applicability of the criterion of establishment in the European Union. It is therefore noted that the controller's own statements confirm the presence of misleading information;

-    Section 6, ‘Your data protection rights’, while not expressly referring to Article 22 of the Regulation, provided specific information on that right, thus giving users the unfounded belief that their personal data were subject to automated decision-making. The absence of an automated processing within the meaning of Article 22 of the Regulation was confirmed by the data controller in its reply letter (Reg. No. 74173/23), where it claimed that ‘although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling which produce legal effects on data subjects or similarly affect them within the meaning of Article 22 of the Regulation’. This again confirms, through the controller's own statements, the presence of misleading information.

With reference to Article 5(1)(a) of the Regulation, reference is made to the same binding decision of the EDPB mentioned in the previous paragraph (binding decision 1/2021), according to which transparency is to be regarded as a general concept which is then concretised in various provisions and specific obligations (e.g. Articles 12, 13, 14, 25 and 35). It is therefore necessary to distinguish the specific obligations arising from the principle of transparency (set out in Articles 12-14 of the Regulation) from the principle expressed in Article 5 of the Regulation, since although these obligations are a concretisation of the general principle, the latter has a broader scope.

The principle of transparency, in fact, is an overarching principle that reinforces other principles (such as fairness and accountability). This interpretation is confirmed by the fact that Article 83(5) of the Regulation allows for distinct administrative fines for infringing transparency obligations independently of any breach of the principle itself. In other words, the transparency obligations do not define the entire scope of the principle of transparency, it follows that a breach of the transparency obligations laid down in Articles 12 to 14 of the Regulation may also constitute a violation of the principle of transparency where such a breach is marked by elements of gravity and systematicity.

In the present case, the Garante considers that there has also been an infringement of the principle of transparency laid down in Article 5(1)(a) of the Regulation, taking into account the gravity (failure to provide information to the data subjects on the legal bases underlying the different personal data processing operations), the nature (lack of clear information on the essential elements of the processing, such as the legal basis, purpose, storage principle, transfer outside the EU) and the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the single specific infringements of the obligations under Articles 12 and 13 of the Regulation.

For the above reasons, the Garante considers that Luka infringed, as at the date of 2 February 2023, Articles 5(1)(a), 12 and 13 of the Regulation.

For the sake of completeness, it should be noted that subsequent technical investigations revealed that the data controller updated the privacy policy for the Replika service again on 23 February 2024. In this latest version, certain inaccuracies previously identified were rectified. Notably, the privacy policy in force at the date of adoption of this decision now granularly mentions the legal basis for each processing activity carried out by the controller and the type of data processed; it expressly clarifies that the service is exclusively intended for users over the age of 18, and contains no reference, not even implicit, to automated decision-making within the meaning of Article 22 of the Regulation. Nonetheless, the information provided under Articles 12 and 13 of the Regulation remains available only in English, does not include specific information on the personal data storage period or the criteria used to determine such a period, and may still mislead data subjects as to the possible transfer of their personal data to the USA.

5.3 ARTICLES.  5(1)(C), 6, 7, 8, 24 AND 25(1) OF THE REGULATION

The Authority notified Luka of the infringement of Articles 5(1)(c); 6; 7; 8; 24 and 25(1) of the Regulation for failing to set up users’ age verification systems as at the date of 2 February 2023. 
Pursuant to Article 5(1)(c) of the Regulation: ‘Personal data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed’.

Pursuant to Article 24(1) of the Regulation: ‘Taking into account the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for the rights and freedoms of natural persons, the controller shall implement appropriate technical and organisational measures to ensure and to be able to demonstrate that processing is performed in accordance with this Regulation. Those measures shall be reviewed and updated where necessary’.

Pursuant to Article 25(1) of the Regulation, the controller shall implement those measures ‘Taking into account the state of the art, the cost of implementation and the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for rights and freedoms of natural persons posed by the processing, the controller shall, both at the time of the determination of the means for processing and at the time of the processing itself ‘.

In the Guidelines 4/2019 on Article 25 of the Regulation, the EPDB clarified that ‘The core of the provision is to ensure appropriate and effective data protection both by design and by default, which means that controllers should be able to demonstrate that they have the appropriate measures and safeguards in the processing to ensure that the data protection principles and the rights and freedoms of data subjects are effective’ and called on data controllers to take into account also the obligations to provide specific protection to children under 18 and other vulnerable groups with a privacy-oriented approach in the data processing design process and default settings.

In the same Guidelines, the EDPB also emphasised that: ‘In line with Article 25(1) the controller shall implement appropriate technical and organisational measures which are designed to implement the data protection principles and to integrate the necessary safeguards into the processing in order to meet the requirements and protect the rights and freedoms of data subjects. Both appropriate measures and necessary safeguards are meant to serve the same purpose of protecting the rights of data subjects and ensuring that the protection of their personal data is built into the processing. Technical and organizational measures and necessary safeguards can be understood in a broad sense as any method or means that a controller may employ in the processing. Being appropriate means that the measures and necessary safeguards should be suited to achieve the intended purpose, i.e. they must implement the data protection principles effectively’.

The Company did not submit any defence statements or documents pursuant to Article 166(5) of the Code, following the Garante’s notice of infringement and initiation of proceedings, and thus did not provide any counter-arguments in relation to the alleged infringement of failing to set up users’ age verification systems. 

In the light of the above-mentioned rules and guidelines, the Authority notes that the data controller is required to implement appropriate technical and organisational measures to ensure, and be able to demonstrate, that processing is carried out in accordance with the Regulation and to process only personal data that are adequate, relevant and limited to what is necessary for the purposes for which they are processed. 

However, the preliminary investigation revealed that the Company failed to adopt measures to ensure specific protection of personal data processed through the Replika service in relation to children under the age of 18. In particular, the absence of age verification procedures, as well as the lack of mechanism to block or restrict access following declarations by users indicating that they are under 18, showed that the data controller did not assess, ex ante, the risks likely to arise from minors registering for and using the service. As a result, on the one hand, the controller did not take any measures to prevent, minimise or mitigate such risks, and, on the other, it processed more data than those necessary for the intended purposes of the processing (i.e. offering the service to users over the age of 18).

The investigation showed that, as at 2 February 2023, the Company had not implemented any age verification mechanisms, either at the time of registration to the Replika service or during its use, despite excluding minors from potential users.

In particular, the following shortcomings were identified:

-    absence of an age verification procedure (the system only required name, email address and gender) with the consequent risk of exposing minors to answers inappropriate for their level of development and self-awareness, including sexually explicit content;

-    lack of banning or blocking mechanisms, even when users declared or otherwise made it clear that they were underage, as well as the provision by the chatbot of responses that were clearly incompatible with the level of protection that should be guaranteed to children and, more generally, to all vulnerable individuals.

Until 2 February 2023, therefore, the Company had not adopted any technical and organisational measures to ensure compliance with the general principles of the Regulation or to safeguard the rights and freedoms of minors, thereby exposing them to the significant personal risks that the legislation in question is intended to limit, including the risk of receiving responses inappropriate to their level of psychophysical development and self-awareness.

Luka implemented age verification mechanisms only after receiving the request from the Garante in the context of the temporary limitation measure, adopted as a matter of urgency on 2 February 2023. In particular, during the discussions that followed the adoption of the aforesaid decision and with specific reference to the age gate issue, the Company explained that it had introduced an age gate across all registration pages of the Services restricting access to users over the age of 18 and which includes a ‘cooling-off period’ aimed at preventing users who are initially denied access based on their real personal data from immediately reattempting registration using a different date of birth. 

The Company also stated its intention to develop a process based on language analysis to detect and prevent use of the Services by individuals under the age of 18.

Prior to the Garante's intervention, therefore, all users—including minors—could register to and use the Replika service without being asked to undergo any age verification. As already clarified in the notice of infringement, in the Garante's view, the lack of a common standard capable of guaranteeing, with absolute certainty, the effectiveness of age verification systems, and the ongoing debate at European level on this issue, are not sufficient to exempt the data controller from its obligations, in particular the obligation to verify the user's actual capacity to enter into a contract, which is essential for its validity.

It follows from the foregoing that, as at 2 February 2023, the Company had not implemented, in accordance with Article 24 of the Regulation, the necessary measures to ensure that the processing of personal data at the time of registration for the Replika service complied with Articles 5(1)(c), 24, and 25 of the Regulation. In particular, the Company failed to implement technical and organisational measures ‘which are designed to implement data-protection principles, such as data minimisation, in an effective manner and to integrate the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of data subjects’. As a result, personal data were processed in excess of what was necessary for the purposes of a service which, according to the declarations of the data controller and the documents on file, was intended to be offered only to users over the age of 18 years.

With specific reference to the infringement of Article 5(1)(c) of the Regulation, it should be noted that, in this case, the adoption of appropriate technical and organisational measures—by design—intended ‘to implement data-protection principles, such as data minimisation’, not only constitutes a core requirement under Article 25(1) of the Regulation, but also represents an additional element substantiating the infringement of the minimisation principle itself, in line with Recital 78 of the Regulation.

More specifically, the Company's failure to adopt appropriate measures to safeguard access to and use of the Replika service resulted not only in the systematic processing by Luka of personal data exceeding what was necessary for the intended purpose of the processing (i.e. providing the service to users over the age of 18), but also that such processing involved data concerning vulnerable individuals (children, even potentially under the age of 13) who, because of this shortcoming, combined with the innovative technology underlying the service and the highly sensitive nature of the interactions generated by the chatbot, were exposed to a particularly high risk. 

The news reports that prompted the Garante to initiate its investigation—together with documented cases of self-harm associated with the use of the chatbot reported in foreign media and brought to the attention of the judicial authorities—support the Garante's allegations. On the basis of the principles expressed by the oft-referred to EDPB binding decision 1/2021, the gravity and impact of the infringements require due consideration, leading to the conclusion that both the infringement of the principle laid down in Article 5(1) of the Regulation and the specific infringement of the obligations under Articles 24 and 25(1) of the Regulation are substantiated.

The Garante, on the contrary, does not consider there to be sufficient grounds to establish an infringement—pursuant to Article 166(5) of the Code—of the provisions regarding the consent of minors, specifically the requirement of a positive act on the part of the child in relation to information society services, as set out in Articles 6, 7 and 8 of the Regulation. In particular, it should be noted that, as emerged during the preliminary investigation—contrary to what was erroneously stated in the version of the privacy policy dated 2 February 2023 (see § 5.2)—the Replika service was not, and is not, offered to minors. Consequently, the data controller was not required to comply with the obligation to identify a legal basis for processing operations which were presumed not to be carried out. 

Based on the foregoing, the Garante considers that, as at 2 February 2023, Luka infringed Articles 5(1)(c), 24 and 25(1) of the Regulation.

For the sake of completeness, it should be noted that, on the date of adoption of this decision, further technical assessments have revealed continuing deficiencies in the age verification system currently implemented by the controller; in particular, it was found that:

- after the user profile is created, it is possible to change the date of birth in the ‘My Profile’ section without this being followed by any verification by the data controller. As a result, children who initially provided a false date of birth to register could subsequently enter their real age and still retain access to the service; 

- the cooling-off period (24 hours) does not apply when the creation of the profile occurs while browsing in incognito mode; in fact, it appears that following an initial failed age check, users may still successfully complete the registration process by entering a different (event fictious) email address;

- no language analysis mechanisms are in place to systematically prompt age confirmation through the age gate process, when users indicate that they are under 18 years of age—except in limited cases where they provide specific inputs (i.e. they unambiguously declare that they are under 18). In such cases, the application prompts users to confirm that they are over 18.

The technical investigation also showed that while users are given the possibility of flagging certain conversations as inappropriate, it is not possible to identify the subsequent actions resulting from such reports.

6. CONCLUSIONS

Based on the above considerations, the Garante confirms the existence of the majority of the infringements alleged and notified in the notice of initiation of proceedings, as detailed below. It also declares the unlawfulness of the personal data processing carried out by the Company, in breach of Articles 5(1)(a) (in relation to both the principles of lawfulness and transparency) 5(1)(c), 6, 12, 13, 24 and 25(1) of the Regulation.

Having established the aforementioned infringements of the Regulation, the Garante shall adopt consequent corrective measures pursuant to Article 58(2) of the Regulation, specifically an order to bring processing operations into compliance under Article 58(2)(d) of the Regulation, and the imposition, under Article 58(2)(i) of the Regulation, of an administrative fine pursuant to Article 83(3) and (5) of the same Regulation. 

Furthermore, given the high sensitivity of the personal data involved, the Garante considers it appropriate to apply the ancillary penalty of publishing this decision on its website, as provided for by Article 166(7) of the Code and Article 16 of the Garante's Regulation No. 1/2019.

As previously noted, the Garante reserves the right to initiate a separate and autonomous investigation to assess the lawfulness of the processing operations carried out by the Company with a particular focus on the legal bases applicable throughout the entire lifecycle of the generative AI system underlying the Replika service.

7.  CORRECTIVE MEASURES PURSUANT TO ARTICLE 58(2)(D) OF THE REGULATION

Pursuant to Article 58(2) of the Regulation, the Garante is granted a series of corrective powers—both of a prescriptive and sanctioning nature—to be exercised when unlawful processing of personal data is found.

Such powers include, pursuant to Article 58(2)(d) of the Regulation, the power ‘to order the controller […] to bring processing operations into compliance with the provisions of this Regulation, where appropriate, in a specified manner and within a specified period’.

From the findings and considerations set out in the preceding paragraphs, it emerges that, as at 2 February 2023, Luka infringed Articles 5(1)(a) (in relation to both the principle of lawfulness and the principle of transparency), 5(1)(c), 6, 12 and 13, as well as Articles 24 and 25(1) of the Regulation. However, following the urgent measure of the Garante, the Company implemented some measures to remedy the identified shortcomings and, subsequently, took additional measures in relation to the infringements notified in the notice of initiation of proceedings, which brought processing into compliance with data protection legislation.

Specifically, Luka remedied the infringement of Articles 5(1)(a) and 6 of the Regulation by amending the privacy policy (see latest version dated 23 February 2024), detailing the legal bases for the different processing operations carried out through the Replika service.

Following the amendments to the privacy policy as described above, it is considered that, at this stage, there are no grounds for adopting further corrective measures under Article 58(2) of the Regulation.

On the other hand, with regard to the infringement of Articles 5(1)(a), 12 and 13 of the Regulation, concerning information obligations, and of Articles 24 and 25(1) and 5(1)(c) of the Regulation, concerning the age verification system, certain aspects remain non-compliant with the Regulation, which the Garante considers must be addressed by specific corrective measures.

In particular, as regards the information obligations, the Garante found that, as of today, Luka’s privacy policy (latest version dated 23 February 2024) is still non-compliant with data protection legislation insofar as: i) it is available only in English; ii) it does not specify the storage periods of personal data or the criteria used to determine such periods; iii) it may still mislead data subjects as to the possible transfer of their personal data to the USA.

Therefore, pursuant to Article 58(2)(d) of the Regulation, the controller is ordered to bring the privacy policy into compliance with Articles 5(1)(a), 12 and 13 of the Regulation by remedying the above shortcomings.

Furthermore, regarding the age verification system, the Garante found that, at the date of adoption of this decision, the age verification system used by the controller does not comply with the principle of data minimisation and with the principles of privacy by design and by default, in that:

- after the user profile is created, users can change their date of birth in the ‘My Profile’ section without this being followed by any verification by the controller. As a result, children who initially provided a false date of birth to register could subsequently enter their real age and still retain access to the service;

- the 24-hour cooling-off period does not apply when the creation of the profile occurs while browsing in incognito mode; in fact, it appears that following an initial failed age check, users may still successfully complete the registration process by entering a different (even fictious) email address;

- the controller has not implemented any language analysis mechanisms prompting users to reconfirm their age through the age gate process when there are clear indications that the user is under the age of 18, except in limited cases where the user provides specific inputs (i.e. they unambiguously declare that they are under 18). Only in such cases, does the application prompt users to confirm that they are over 18.

However, the Garante positively noted the implementation by Luka of a function allowing users to flag certain conversations as inappropriate, although it was not possible to determine the effect of such reports.

In view of the above, pursuant to Article 58(2)(d) of the Regulation, the controller is ordered to bring the age verification system into compliance with Articles 5(1)(c), 24 and 25(1) of the Regulation by remedying the shortcomings identified above.

8. INJUNCTION ORDER FOR THE APPLICATION OF THE ADMINISTRATIVE FINE AND ANCILLARY PENALTIES

The Garante, pursuant to Articles 58(2)(i) and 83 of the Regulation and Article 166 of the Code,  shall have the power to impose an administrative fine pursuant to Article 83, in addition to, or instead of corrective measures referred to in the same paragraph.

In determining the amount of the administrative fine, the Garante shall take into account the principles and interpretation provided by the EDPB in their Guidelines 4/2022 on the calculation of administrative fines under the GDPR, version 2.1, adopted on 24 May 2023.

Based on the arguments set out above, the Garante found that the following provisions of the Regulation have been infringed: Articles 5(1)(a) and 6; Articles 5(1)(a), 12 and 13; Articles 5(1)(c), 24 and 25(1) of the Regulation.

In the present case, it should first be noted that the Company engaged in a number of conducts leading to multiple infringements, as specifically outlined and substantiated in the preceding paragraphs. The infringements relating to the legal basis (Articles 5(2) and 6 of the Regulation), transparency (Articles 5(1)(a), 12 and 13), and the age gate (Articles 24 and 25(1)) may, under the principle of unity of action, be considered together under Article 83(3) of the Regulation, which provides that, if a controller or processor infringes several provisions of the Regulation for the same or linked processing operations, the total amount of the administrative fine shall not exceed the amount specified for the gravest infringement. Specifically, in relation to such infringements, it may be assumed that the processing operations are linked, as defined in paragraph 28 of the above-mentioned guidelines (a unitary conduct consists of several parts that are carried out by a unitary will and are contextually, spatially and temporally related in such a close way that, from an objective standpoint, they would be considered as one coherent action). Among the aforementioned infringements, the most serious is considered to be the infringement of transparency obligations, given that both Article 5(1)(a) (Principle of transparency) and Articles 12 and 13 (Rights of the data subject) are subject to the administrative fines laid down in Article 83(5) up to a maximum amount of €20 million or, in the case of an undertaking, up to 4% of the total worldwide annual turnover of the preceding financial year, whichever is higher.

Pursuant to Article 83(1) of the Regulation, the administrative fine shall be effective, proportionate and dissuasive in each individual case. According to the aforementioned guidelines, the EDPB has clarified that the calculation of administrative fines should commence from a harmonised starting point which forms the beginning for further calculation, in which all circumstances of the case are taken into account and weighed (see par. 46). The harmonised starting point should consider three elements: 1) the categorisation of the infringements by nature under Articles 83(4)–(6) of the Regulation; 2) the seriousness of the infringement; 3) the turnover of the undertaking (see par. 48).
As regards the first aspect, two infringements are found, in abstract terms, of a more serious nature (Article 83(5) of the Regulation) and one of a less serious nature (Article 83(4) of the Regulation). The first two concern the infringement of the legal basis and transparency, while the third concerns the infringement of Article 25 of the Regulation.

Regarding concrete seriousness, the elements to be taken into account are: a) the nature, gravity and duration of the infringement (Article 83(2)(a) of the Regulation); b) the intentional or negligent character of the infringement (Article 83(2)(b) of the Regulation) ; c) the categories of personal data affected (Article 83(2)(g) of the Regulation).

In this case, regarding the three infringements linked under the principle of unity of action (legal basis, transparency, and data protection by design and by default), the level of seriousness of the infringements must be considered high, given that: i) the nature of the infringements concerns two fundamental principles (accountability and transparency), namely, on the one hand, the controller’s failure to demonstrate that it had identified the legal bases for processing prior to the commencement of such processing, and, on the other hand, the failure to provide appropriate information to the data subject, particularly regarding the purposes of the two distinct types of processing (‘Chatbot Interaction’ and ‘Model Development’), and the involvement of data such as minors’ data, in excess of what was necessary to achieve the service provision purpose; ii) the nature of the processing involves significantly high risks as it is linked to an innovative, disruptive and rapidly evolving technology; iii) the processing has a cross-border nature and global scope, with effects that are practically uncontrollable by data subjects; iv) the purpose of the processing falls within the core business of the Company; v) the number of Italian data subjects involved cannot be precisely quantified, but in general terms it can reasonably be presumed to be very high, as information available on Google’s App Store (Google Play) shows that the application has exceeded 10 million downloads (suggesting a similar figure may exist, though not verifiable, for downloads via Apple Store), while academic sources (Shikhar Ghosh, Replika: Embodying AI, Harvard Business School, Faculty & Research) indicate that the Company had already reached 10 million users by January 2022; vi) the nature of the data has also involved special category data—considering the nature of the chatbot itself (which is still presented as ‘an AI companion always ready to chat when you need an empathetic friend’)— and, in the absence of age verification mechanisms and data filtering systems, personal information relating to underage users. The duration of the infringement is significant, given that the app was released to the public in November 2017; indeed, the fact that the chatbot’s success materialised later in time does not counterbalance the judgement of high seriousness, since the end of the infringement was triggered by and coincided with the Garante’s urgent measure. 

All the infringements must be regarded as being unintentional. As stated by the Article 29 Working Party in the Guidelines on the application and setting of administrative fines for the purposes of Regulation (EU) 2016/679, adopted on 3 October 2017 and endorsed by the EDPB on 25 May 2018 (WP 253 guidelines), ‘intent’ includes both knowledge and wilfulness to commit a violation, whereas ‘unintentional’ means that there was no intention to cause the infringement, although there was a breach of the duty of care. The Court of Justice of the European Union (CJEU), in a recent ruling (Judgment C-807/21 of 5 December 2023), established that it is the responsibility of the supervisory authority to determine whether an infringement has been committed intentionally or negligently by the controller, as only an unlawful infringement constitutes a condition for an administrative fine to be imposed. In this regard, it should be noted that while the CJEU stated in the aforementioned judgment that Article 83 of the Regulation does not allow an administrative fine to be imposed without it being established that the infringement was committed intentionally or negligently by the controller (see par. 75), it also upheld the fundamental principle ‘ignorantia legis non excusat’, stating that ‘a controller can be penalised for conduct falling within the scope of the GDPR where that controller could not be unaware of the infringing nature of its conduct, whether or not it is aware that it is infringing the provisions of the GDPR’ (see par. 76). This principle had already been established by the CJEU in another case (Judgment C-601/16 of 25 March 2021, paragraphs 97 and 98) where it affirmed that ‘an undertaking may be punished for conduct falling within the scope of Article 101(1) TFEU where that undertaking could not have been unaware of the anticompetitive nature of its conduct, whether or not it was aware that it was infringing the competition rules of the Treaty (see, to that effect, judgment of 18 June 2013, Schenker & Co. and Others, C 681/11, EU:C:2013:404, paragraph 37). It follows that the fact that that undertaking has characterised wrongly in law its conduct upon which the finding of the infringement is based cannot have the effect of exempting it from imposition of a fine in so far as it could not be unaware of the anticompetitive nature of that conduct’ (Judgment of 18 June 2013, Schenker & Co. and Others, C 681/11, EU:C:2013:404, paragraph 38). In the present case, it is considered that Luka could not, at the time its service was made available (also) to users located within the European Union, and specifically in Italy, avoid a duty to be aware of and to apply the Regulation, which, as is well known, safeguards a fundamental right provided for and protected by Article 8 of the Charter of Fundamental Rights of the European Union. In light of the specific circumstances of the case, the context in which the controller operates, and the disruptive and rapidly evolving technology characterising its activities, it is considered that the failure to ensure compliance of the personal data processing with EU data protection law is indicative of the negligence underlying the concept of fault and demonstrates the existence of such a subjective element on the part of the Company. Furthermore, this fault must be regarded as serious, precisely due to the scale and innovative nature of the service offered, which entails large-scale processing of personal data at a global level. 
In order to determine the amount of the administrative fine, the aggravating factors set out in Article 83(2)(d) and (f) of the Regulation are also applicable. 

As regards the first aspect, the degree of responsibility of the controller must be considered high due to the failure, at the time of the service launch, to adopt appropriate technical and organisational measures to mitigate the risks to the rights and freedoms of data subjects and enable them to exercise the rights laid down in Chapter III of the Regulation. As for the second circumstance, regarding the degree of cooperation, it should be noted that although the Company responded to the request for information, it did not submit any defence statements in reply to the notice of infringement notified pursuant to Article 166 of the Code, thereby demonstrating limited cooperation with the Garante.

In order to determine the amount of the administrative fine, the following measures implemented by the controller to remedy the infringement and mitigate its possible adverse effects (Article 83(2)(f) of the Regulation), are considered as mitigating factors. Namely:

-    the updates to the privacy policy, both immediately following Decision No. 39/2023 and subsequently, particularly with reference to the latest version dated 23 February 2024, as described in the previous paragraph, although such updates are considered not exhaustive;

-    the implementation of age-gating mechanisms as described in the previous paragraph, although not exhaustive. 

Based on the above elements, assessed in their entirety, and in the absence of data regarding the total worldwide annual turnover of the Company for the preceding financial year, it is hereby decided that, pursuant to Article 83(3) of the Regulation, the total amount of the administrative fine is set at EUR 5,000,000.00 (five million), equal to half of the maximum amount provided for in Article 83(5) of the Regulation. This amount has been determined as follows:

•    pursuant to Article 83(3) of the Regulation, considering the conduct as a single instance due to the interrelated nature of the processing activities for the reasons stated above, the administrative fine for the most serious infringement—namely of Articles 5(1)(a), 12, and 13 of the Regulation—is set at EUR 3,000,000.00;

•    the fine is increased by EUR 1,000,000.00 for the infringement of Articles 5(1)(a) and 6 of the Regulation;

•    the fine is further increased by EUR 1,000,000.00 for the infringement of Articles 5(1)(c) and 25(1) of the Regulation.

This administrative fine is considered to be effective, proportionate and dissuasive, pursuant to Article 83(1) of the Regulation.

Taking into account the particular sensitivity of the data processed, it is considered that the ancillary penalty of publication of this decision on the Garante's website shall apply, as provided for by Article 166(7) of the Code and Article 16 of the Regulation of the Garante’s Regulation No. 1/2019; this in light of the nature and gravity of the established infringements, in particular considering that they involve large-scale processing operations affecting a high number of data subjects, the data protection risks associated with the provision  of a service based on innovative and complex technology in the absence of appropriate safeguards. It is also considered that there is a general interest in the topic of generative artificial intelligence that requires the widest possible awareness of the Garante’s position on the matter.

Finally, it is considered that the conditions set out in Article 17 of the Garante’s Regulation No. 1/2019 concerning internal procedures with external relevance aimed at performing the tasks and exercising the powers entrusted to the Garante, are met for recording the infringements identified herein in the Garante’s Internal Register, as provided for by Article 57(1)(u) of the Regulation.

BASED ON THE FOREGOING, THE GARANTE

pursuant to Article 57(1)(f) of the Regulation, declares unlawful the processing activities described and carried out by Luka Inc., based in 490 Post St Suite 526, San Francisco, California, United States of America, as set out in the reasoning, for infringing Articles 5(1)(a) (with regard to both the principles of lawfulness and transparency), 6, 12, 13, 5(1)(c), 24, and 25(1) of the Regulation and, consequently:

a)    pursuant to Article 58(2)(d) of the Regulation, orders the Company, within thirty days of notification of this decision, to bring its processing activities into compliance with the provisions of the Regulation, in particular by aligning its privacy policy with Articles 5(1)(a), 12, and 13 of the Regulation, and aligning its age verification system with Articles 5(1)(c), 24, and 25 of the Regulation, remedying the shortcomings identified in paragraphs 5 and 6 of this decision, respectively;

b)    pursuant to Article 157 of the Code, orders the Company to inform the Authority, within sixty days of notification of this decision, of the initiatives undertaken to implement the corrective measure referred to in the preceding point; failure to comply with the provisions set out in this point may result in the imposition of the administrative fine provided for in Article 83(5) of the Regulation

ORDERS

Luka Inc., based in 490 Post St Suite 526, San Francisco, California, United States of America, to pay the sum of EUR 5,000,000.00 (five million) as an administrative fine for infringing Articles 5(1)(a) and 6; Articles 5(1)(a), 12, 13, 5(1)(c), 24 and 25(1) of the Regulation, stating that the infringing party, pursuant to Article 166(8) of the Code, has the right to settle the dispute by paying, within sixty, an amount equal to half the imposed fine.

REQUIRES

a)    the aforesaid Company, in the event of failure to settle the dispute pursuant to Article 166(8) of the Code, to pay the sum of EUR 5,000,000.00 (five million), according to the modalities indicated in the annex, within sixty days of notification of this decision, under penalty of the adoption of the consequent executive actions pursuant to Article 27 of Law No. 689/1981.

PROVIDES

a)    that this decision be published, pursuant to Article 154-bis of the Code and Article 37 of the Garante’s Regulation No. 1/2019;

b)    that the application of the ancillary penalty of the publication of this injunction order on the Garante’s website, as provided for by Article 166(7) of the Code and Article 16 of the Garante’s Regulation No. 1/2019, be applied;

c)    that this decision be recorded in the Garante’s Internal Register—as laid down in Article 57(1)(u) of the Regulation and Article 17 of the Garante’s Regulation No. 1/2019 concerning internal procedures with external relevance, aimed at performing the tasks and exercising the powers entrusted to the Garante—regarding infringements and measures adopted in compliance with Article 58(2) of the Regulation.

The Garante reserves the right to investigate and assess in a separate and autonomous proceeding, the aspects concerning the lawfulness of the processing operations carried out by Luka Inc., with specific reference to the legal bases for processing applicable throughout the entire lifecycle of the generative AI system underlying the Replika service.

Under Article 78 of the Regulation, Article 152 of the Code and Article 10 of Legislative Decree No. 150/2011, this decision may be challenged before the ordinary judicial authority, by lodging an appeal with the ordinary court of the controller’s place of residence, within thirty days from the date the decision was notified, or within sixty days if the appellant resides abroad.

Rome, 10 April 2025             

 

THE PRESIDENT
Stanzione

THE RAPPORTEUR
Scorza

THE ACTING SECTRETARY GENERAL
Filippi

VEDI ANCHE 
Comunicato stampa del 19 maggio 2025
Provvedimento del 22 giugno 2023
Comunicato stampa del 3 febbraio 2023
Provvedimento del 2 febbraio 2023
 

- English version

[doc. web n. 10127930]

Provvedimento del 10 aprile 2025

Registro dei provvedimenti
n.  232 del 10 aprile

 

IL GARANTE PER LA PROTEZIONE DEI DATI PERSONALI

NELLA riunione odierna, alla quale hanno preso parte il prof. Pasquale Stanzione, presidente, la prof.ssa Ginevra Cerrina Feroni, vicepresidente, il dott. Agostino Ghiglia e l’avv. Guido Scorza, componenti, e il dott. Claudio Filippi - Segretario generale reggente;

VISTO il Regolamento (UE) 2016/679 del Parlamento europeo e del Consiglio, del 27 aprile 2016, relativo alla protezione delle persone fisiche con riguardo al trattamento dei dati personali, nonché alla libera circolazione di tali dati e che abroga la direttiva 95/46/CE (Regolamento generale sulla protezione dei dati personali, di seguito “Regolamento”);

VISTO il Codice in materia di protezione dei dati personali (d.lgs. 30 giugno 2003, n. 196), come modificato dal d.lgs. 10 agosto 2018, n. 101, recante disposizioni per l'adeguamento dell'ordinamento nazionale al citato Regolamento (di seguito “Codice”);

VISTO il Regolamento n. 1/2019 concernente le procedure interne aventi rilevanza esterna, finalizzate allo svolgimento dei compiti e all’esercizio dei poteri demandati al Garante per la protezione dei dati personali, approvato con deliberazione n. 98 del 4 aprile 2019, pubblicato in G.U. n. 106 dell’8 maggio 2019 e in www.gpdp.it, doc. web n. 9107633 (di seguito “Regolamento del Garante n. 1/2019”);

VISTA la documentazione in atti;

VISTE le osservazioni formulate dal segretario generale ai sensi dell’art. 15 del Regolamento del Garante n. 1/2000;

RELATORE l’avv. Guido Scorza;    

1.    INTRODUZIONE 

Il procedimento ha avuto origine da una attività istruttoria avviata d’ufficio dall’Autorità a seguito della pubblicazione di notizie stampa e di accertamenti preliminari condotti sul servizio Replika (https://replika.com/), un chatbot con interfaccia scritta e vocale, sviluppata e gestita dalla società statunitense Luka Inc. (d’ora in poi “Luka” o la “Società”) e basata su un sistema di intelligenza artificiale generativa.

Replika viene presentata come un chatbot in grado di migliorare l’umore ed il benessere emotivo dell’utente, aiutandolo a comprendere i suoi pensieri ed i suoi sentimenti, a tenere traccia del suo umore, ad apprendere capacità di coping (ossia controllo dello stress), a calmare l’ansia ed a lavorare verso obiettivi come il pensiero positivo, la gestione dello stress, la socializzazione e la ricerca dell’amore. Replika genera un “compagno/a virtuale” che l’utente può decidere di configurare come amico/a, terapista, partner romantico o mentore. 

Replika utilizza un sistema di LLM (Large Language Model) che viene costantemente alimentato e si perfeziona attraverso l’interazione con gli utenti.

Ai fini del presente provvedimento, per “intelligenza artificiale generativa” si intende il campo dell’intelligenza artificiale che si concentra sulla creazione di contenuti nuovi e originali rispetto ai dati di input in risposta alle richieste (prompt) dell’utente, attraverso l’utilizzo di algoritmi prevalentemente di tipo neurale. Per “rete neurale” si intende un modello computazionale standard applicabile nei contesti più diversificati che permette il riconoscimento di oggetti, forme o pattern all’interno di un dato o un insieme di dati (ad esempio, un volto umano in una fotografia). Gli algoritmi di intelligenza artificiale generativa sono impiegati in una vasta gamma di applicazioni, tra cui il riconoscimento e la generazione di immagini, di tracce vocali o musicali, di testi e di video.

Un esempio di intelligenza artificiale generativa sono i modelli linguistici di grandi dimensioni (Large Language Models). Ai fini del presente provvedimento per “Large Language Model” si intende un modello probabilistico di un linguaggio naturale, come la lingua inglese o italiana, che si fonda sull’assunto per cui tutti i linguaggi naturali sono fortemente ridondanti e correlati; da ciò deriva la capacità del LLM di individuare la parola o il simbolo che, probabilisticamente, sono immediatamente successivi ad un determinato dato.

Alla luce dei predetti elementi il Garante ha avviato un’istruttoria ex officio rilevando che il trattamento dei dati personali da parte di Luka nell’ambito del servizio Replika potesse dare luogo ad una violazione della normativa in materia di dati personali con particolare riferimento: alla policy privacy ed agli obblighi previsti in materia di trasparenza; all’assenza nella privacy policy di una indicazione puntuale delle basi giuridiche del trattamento in relazione alle varie operazioni di trattamento effettuate; alla base giuridica del trattamento dei dati personali dei minori dovendosi escludere che, in questo caso, potesse essere individuata nell’esecuzione di un contratto; all’assenza di un qualsivoglia filtro per la verifica dell’età degli utenti, sia in fase di accesso al servizio (mediante registrazione dell’account) sia durante l’interazione con il chatbot; alla proposizione, attraverso il chatbot, di contenuti in contrasto con le tutele che andrebbero assicurate ai minori e, più in generale, a tutti i soggetti più fragili.

In tale quadro, il 2 febbraio 2023, rilevato che il trattamento dei dati personali da parte di Luka nell’ambito del servizio Replika potesse dare luogo alla violazione degli artt. 5, 6, 8, 9 e 25 del Regolamento e presentasse concreti rischi per i minori d’età, anche in ragione della proposizione di risposte in contrasto con le tutele rafforzate da assicurare ai minori ed ai soggetti vulnerabili, il Presidente dell’Autorità ha adottato nei confronti di Luka, ex art. 5, comma 8, del regolamento del Garante n. 1/2000, un provvedimento d’urgenza (n. 39/2023, prot. n. 18321/23) di limitazione provvisoria del trattamento dei dati personali degli interessati stabiliti nel territorio italiano, ai sensi dell’art. 58, par. 2, lett. f), del Regolamento.

Successivamente, con il provvedimento n. 280 del 22 giugno 2023 (prot. n. 104960/23), l’Autorità ha deliberato la sospensione del provvedimento n. 39/2023 di limitazione provvisoria a condizione che il titolare, ex art. 58, par. 2, lett. d) del Regolamento, adottasse misure idonee a garantire che le attività di trattamento dei dati personali nell’ambito del servizio Replika avvenissero in modo conforme alla normativa in materia di protezione dei dati personali. Segnatamente l’Autorità ha ingiunto al titolare di:

1.    presentare, a tutti gli utenti in Italia, prima della registrazione e prima dell’accesso al servizio Replika, una policy privacy aggiornata;

2.    implementare un meccanismo di age gate in tutte le pagine di registrazione ai servizi;

3.    implementare un “periodo di raffreddamento” (cooling-off period) volto ad evitare che i minorenni inseriscano una data di nascita diversa quando viene loro negato l’accesso ai servizi;

4.    predisporre, a favore degli utenti in Italia, la possibilità di esercitare in modo semplice ed efficace i propri diritti in materia di protezione dei dati personali, tra cui quello di opporsi al trattamento dei dati personali e di richiedere l’accesso, la rettifica e la cancellazione dei dati;

5.    sottoporre al Garante, quindici giorni prima della data prevista per l’apertura del servizio all’utenza italiana, un piano per lo sviluppo di un processo volto ad impedire l’accesso al servizio a soggetti di età inferiore ai 18 anni, eventualmente corredato da un meccanismo di analisi del linguaggio avente efficacia interdittiva successiva;

6.    sottoporre al Garante, quindici giorni prima della riapertura all’utenza italiana, un piano per l’implementazione di funzioni che consentano agli utenti di segnalare i contenuti inappropriati per evitare che il chatbot Replika li riproponga, quali, ad esempio, la possibilità di contrassegnare specifiche risposte come inappropriate e di fornire un feedback sull’esperienza dell’utente durante la sessione.

The Guarantor has indicated different terms for the implementation of the above-mentioned provisions, establishing that those referred to in points 1 to 4 were to be fully fulfilled no later than 28 July 2023, and that those referred to in points 5 and 6 were to be implemented within fifteen days of the date of reopening of the service to Italian users.

2. LUKA'S RESPONSES TO PROVISIONS NOS. 39/2023 AND 280/2023

The Company, with a note dated 3 March 2023 (prot. no. 38795/23), communicated that it had promptly taken action to follow up on the Authority's requests, in particular to comply with the request for temporary limitation of processing for users established in Italian territory, promptly inhibiting access to the Replika service from Italy, both through the app and through its website.

Luka also reported that it has launched a series of initiatives aimed at concretely implementing the requests of the Guarantor, also through the involvement of external consultants and industry experts; in particular, the Company declared that it has launched a series of assessments, actions and processes aimed at:

-    implementing more robust user age verification mechanisms, in order to strengthen the guarantee that minors in Italy do not use the “Replika” service, a service reserved for adults; in addition to the age gate tools already in use, the Company has undertaken to introduce automated measures aimed at recognizing underage users based on the analysis of indicators contained in conversations with the chatbot;

-    implementing adequate algorithms and processes for the moderation of inappropriate content, according to the best state of the art;

-    ensure compliance with the Regulation by, inter alia, updating the register of processing activities, reviewing and updating data protection impact assessments (DPIA), as well as updating the privacy policy relating to the service, in order to increase the level of transparency for users.

The Company, with a note dated 31 March 2023 (prot. no. 55533/23), requested the revocation of the corrective measure of the temporary limitation ordered by emergency provision no. 39/2023, specifying:

-    that it has designed the Replika service in such a way as to limit the extent of personal data processing, in accordance with the principles set out in art. 5 of the Regulation, including i) minimising the collection of user registration data (name, email address, date of birth - to verify age - and any third-party log-in data); ii) adopting data retention and deletion procedures that strike a balance between the need to provide the user with a seamless experience and that of minimizing the personal data that remains accessible; iii) designing proprietary artificial intelligence (AI) models to respond to users; iv) not sharing user conversations with third parties other than the Company’s essential service providers, who are bound by confidentiality obligations; v) implementing strict controls designed to limit access to personal data by its staff; vi) not using user conversations for advertising or marketing purposes; 

-    not offering the service to minors and basing the processing of users’ personal data on the legal basis of contractual performance; 

-    having implemented, following the provision of the Guarantor, numerous measures aimed at preventing minors from accessing the service in violation of the Company’s terms;

-    to have entered its mobile application in the Apple App Store with an age classification equal to or above 17 years, which is the highest age classification allowed by Apple;

-    to not collect special categories of personal data, given that the sharing of special categories of personal data by users during the interaction with the chatbot occurs spontaneously and must therefore be qualified as covered by an explicit consent to the processing, in accordance with art. 9 of the Regulation;

-    to take seriously its data protection responsibilities and to have integrated data protection into the design of the service, in accordance with art. 25 of the Regulation and to continue to “develop and improve its policies and procedures to provide users with a consistent, safe and rewarding experience”. 

With specific reference to the provision of the Guarantor, the Company declared:

-    to have promptly blocked access to the Replika service to natural persons who are in Italy;

-    to have strengthened the measures aimed at preventing access to the service by minors under 18, in particular by:  i) introducing an age gate on all service registration pages that requires the indication of a date of birth greater than or equal to 18 years to access the service; ii) providing for a “cooling-off period”, in line with the indications of the data protection authorities and with best practices, to prevent minors from entering a different date of birth when the system denies access to the service; iii) starting activities aimed at improving the automated processes for controlling content (reporting individuals presumably under 18 and preventing use of the service until their age is verified through more robust means);

-    to have updated its privacy policy to resolve the transparency issues identified by the Guarantor;

-    to continue to develop and improve its content moderation practices to avoid harm to users, in particular by creating a trust and safety program to prevent the chatbot from engaging in offensive or harmful conversations;

-    to have limited access to conversations of a sexual nature or relating to other adult content to users active on 1 February 2023, excluding the availability of such types of conversations to new users;

-    to continue to make efforts to ensure compliance with the Regulation with the support of an external consultant in matters of personal data protection. The commitments undertaken by the Company include: i) updating and maintaining the register of the Company’s processing activities; ii) reviewing and updating the data protection impact assessments (DPIA), which include the documentation of the processes for data protection by design and by default; iii) refining and verifying the Company’s security policies and procedures; iv) the review of the Company's governance in terms of data protection (including the possibility of appointing a DPO following the expansion of the Company's activities in the European Union).

The Company, with a note dated 26 April 2023 (ref. 68896/23), submitted a second request for revocation of the corrective measure of the temporary limitation ordered by emergency measure no. 39/2023, reiterating the measures adopted, as already illustrated in the previous note.

The Company, with a note dated 14 June 2023 (ref. no. 93675/23), following up on what was discussed at the hearing held on 31 May 2023, reiterated that it had given timely feedback to measure no. 39/2023, immediately blocking access to Replika in Italy and that it had implemented adequate measures in response to the issues raised by the Authority in the aforementioned measure. The Company also represented its commitment to interrupt the possibility for users located in Italy to engage in sexual conversations, providing, once reactivated, two versions of the Replika service: a free version and a paid version containing romantic, but not sexual, content. The introduction of a paid “romantic” version, according to the Company, involves an additional age verification based on the insertion of the user’s payment card data, in line with the most recent market standards for age verification mechanisms.

The Company, with a note dated 14 July 2023 (prot. no. 109176/23), communicated that it had fulfilled the requests referred to in points 1-6 of provision no. 280/23 and, in particular, declared:

1.    in relation to the information referred to in point 1 of provision no. 280/23, to have implemented an updated privacy policy in the registration process and before accessing the service and that this information would be shown to Italian users when the service was reactivated;

2.    with reference to the age gate mechanism referred to in point 2 of provision no. 280/23, to have implemented an age verification system on all registration pages and that this system would be applied when the service was reactivated;

3.    with reference to the cooling-off period referred to in point 3 of provision no. 280/23, to have implemented a cooling-off period to prevent minors from attempting to access the service again by entering a different date of birth. This period - lasting 24 hours - is expected to be managed by recognizing the credentials of the account of a minor user and consequently inhibiting the entry of a different date of birth and ii) by installing a cookie suitable for preventing minors from entering a different date of birth again from the same browser. The Company stated that such cooling-off period would be applied when the service is reactivated;

4.    as regards the exercise of the rights referred to in point 4 of provision no. 280/23, to provide users with a simple and effective method to exercise their data protection rights, including the right to object to the processing of their personal data and the rights to request access, rectification and erasure of their data and that such mechanism would be applied when the service is reactivated;

5.    as regards the request to prepare a plan for the development of an age verification mechanism during registration referred to in point 5 of provision no. 280/23, to have implemented processes to prevent access to minors under 18, including a linguistic analysis mechanism that requires users to reconfirm their age through the age gate process when users identify themselves as minors under 18. in the absence of a birth date that satisfies the age gate, the user cannot access the service. The Company stated that such processes would be applied in Italy upon reactivation of the service;

6.    with regard to the request to prepare a plan for the development of an age verification mechanism during the use of the service referred to in point 6 of provision no. 280/23, to have implemented functions that allow users to report inappropriate content to prevent the Replika chatbot from re-proposing it, such as, for example, the ability to mark specific responses as inappropriate and to provide feedback on the user's experience during the session. The Company stated that such functions would be applied in Italy upon reactivation of the service.

Luka produced, together with the note of 14 July 2023, a copy of the privacy policy updated to 12 June 2023.

3. INVESTIGATIVE ACTIVITY

In parallel with the adoption of the precautionary measure, the Authority proceeded to acquire the elements deemed necessary for carrying out the investigation through a request for information, pursuant to Articles 58, paragraph 1, letter e), of the Regulation and 157 of the Code.

With a note dated 6 April 2023 (prot. no. 58925/23), the Guarantor sent a request for information to Luka asking for clarifications regarding the functioning of Replika (categories of personal data processed and source from which they are collected; methodology applied for the collection; methods of processing the collected data; place of data storage; security measures adopted; processing of user data for system training purposes or for other purposes pursued by Luka), the processing of users' personal data (legal basis; retention period; minimum age to access the service provided by Replika; DPIA; appointment of a representative pursuant to art. 27 of the Regulation; procedures for managing rights pursuant to art. 12–22 of the Regulation; legal basis and guarantees of adequacy pursuant to Chapter V of the Regulation, where applicable; clarifications regarding automated processing pursuant to art. 22 of the Regulation), and the age verification measures for access to the service on the date of notification of emergency measure no. 39/23.

With respect to this request, with a note dated 8 May 2023 (ref. no. 74173/23), the Company, after having preliminarily claimed to have a sole establishment in the European Union in the Netherlands, represented:

-    to use the messages and contents that the user sends to the chatbot to enable that user's conversations (the “Chatbot Interaction”). With reference to the Chatbot Interaction, the content of the database may include basic profile information, conversation topics, questions that the user can ask and selected preferences or interests. When a user sends a message, the model analyzes the text to allow the chatbot to generate a response based on the latest messages in the conversation. The Company also specified that it uses a database that contains all the information sent through the chat to create de-identified data and refine the LLM model that forms the basis of the chatbot (“Model Development”). The part of the database used as a source to create de-identified data is limited to: 1) user “Reactions” (“like”, “dislike”, “love”, “funny” “meaningless” or “offensive”), if the user chooses to make such a selection; 2) user “Feedback” of the satisfaction levels of the conversation (“happy”, “neutral” or “sad”); 3) “Snippets”, which are small parts of the user conversations that provide context for the interpretation of the Reactions and Feedback. The information used by the Company for the Development of the Model does not identify specific individuals and cannot be associated with specific individuals (“De-identified Data”) as any personal identifiers (such as names, addresses, emails, phone numbers and identification numbers) that may be contained in the conversation fragments are removed and the fragments are “shuffled” in a randomized manner;

-    to collect all the personal data described above from the interaction of users with the service;

-    to employ a system for collecting (of “Reactions”, “Feedbacks” and “Fragments”) and processing in real time the interactions of users with the chatbot using webhooks, i.e. automated tools that capture such information and send it to the Company’s servers;

-    to follow, in the processing of the “De-identified Data” for the Development of the Model, the following phases: 1) data collection, as illustrated above; 2) pre-processing consisting in the cleaning, structuring and elimination of any personally identifiable data from the data itself, in order to safeguard privacy (through aggregation and randomization techniques); 3) labeling of the pre-processed data; 4) analysis and development to evaluate the performance of the LLM model, identify patterns and develop filters that prevent the model from producing outputs with inappropriate content; 5) testing and validation (regular testing and validation against predefined criteria);

-    to store personal data on encrypted databases hosted by Amazon Web Services, Inc. in the United States;

-    not to use personal data provided by users for Model Development;

-    to employ technical and organizational measures to protect the security of personal data and “De-identified Data” from unauthorized access, use and disclosure. Such measures include encryption, access controls, vulnerability management, pre-processing and anonymization of “Fragments”, “Reactions” and “Feedback”, training and possible disciplinary measures in case of non-compliance with the measures by Company personnel;

-    to rely on the contractual legal basis for “Chatbot Interaction” as the processing of user data is necessary for the provision of the service, in accordance with the Terms of Service. Such processing includes creating and maintaining user account profiles, facilitating payments and transactions and processing data entered by users to generate the chatbot response;

-    to rely on the legal basis of legitimate interest for the “Model Development”;

-    to retain the data for “the time it deems reasonably necessary to offer users a safe, enjoyable and effective experience on the platform”, in compliance with the principle of minimization;

-    to retain the data of the “Chatbot Interaction” for “a period sufficient to facilitate the recall of information to ensure users seamless conversations with the chatbot, in line with user expectations”;

-    to retain [without further specification, editor’s note] the user data to create “De-identified Data” for the “Model Development”;

-    that the minimum age required to use the Replika service is 18 years;

-    that there is no contradiction between the preceding point and the provision in the Company’s privacy policy, which states: “we do not knowingly collect Personal Data from children under the age of 13. If you are under the age of 13, please do not submit any Personal Data through the Services”, as this statement was included because it is required under the US federal law (COPPA);

-    Replika’s mobile application included an age gate that prevented minors under the age of 18 from accessing the service even before the provision of the Guarantor of February 2, 2023. The Company also placed its application in the Apple App Store with an age rating of 17 or above, which is the highest age rating allowed by Apple;

-    all adult content has been placed behind a paywall, out of reach of minors;

-    following the provision of 2 February 2023, the Company has voluntarily improved the measures aimed at preventing subjects under 18 years of age from accessing the service;

-    not to have appointed a representative pursuant to art. 27 of the Regulation as the Company has an establishment in the European Union;

-    as regards the exercise of the rights of the interested parties, the relevant information is provided through a privacy policy published on the Company's website and in the App. Access, rectification and deletion can be requested by users, who can also object and limit the processing of any personal data not necessary for the provision of the service. Requests are assessed individually;

-    not to carry out any profiling activity of the interested parties or to take automated decisions that have legal effects or similar relevance;

-    to directly collect personal data from users and not to transfer them from Italy or the European Union pursuant to Chapter V of the Regulation and to have entered into data processing agreements with data processors, which include standard contractual clauses, where required;

-    for content control purposes, to have trained its models to avoid the emergence and escalation of inappropriate content or inappropriate responses. As part of this process, the Company uses open-source data sets specifically designed and made available to the AI research community to improve the safety and robustness of machine learning models. The Company has also developed, and continues to refine and improve, filters that recognize keywords, phrases and patterns associated with harmful behavior, such as self-harm, insult or murder. The filters trigger the LLM model to respond appropriately to such content, for example by changing the topic of the conversation or providing users with self-help resources. The Company also uses human reviewers in both evaluating the AI model and developing filters;

-    to use other methods to control inappropriate content or content that violates the application's terms of service, including: 1) placing so-called romantic content behind a paywall and disabling sexually explicit content for new users; 2) allowing users to flag certain content or conversations as offensive in real time and using those flags to improve the models and prevent them from developing similar content in the future; 3) prohibiting users, in the terms of service, from uploading illegal, harmful and threatening content.

Together with the response of May 8, the Company produced a copy of the privacy policy in force on February 2, 2023, the updated version of the same dated March 22, 2023, as well as a copy of the impact assessment (without date and signature).

With a note dated February 27, 2024 (prot. no. 23744/24), the Authority notified the Company of the communication of the initiation of the procedure for the adoption of corrective and sanctioning measures pursuant to art. 166, paragraph 5, of the Code and art. 12 of the Internal Regulation of the Guarantor no. 1/2019, contesting Luka's alleged violation of arts. 5; 6; 7; 8; 12; 13; 24; 25, par. 1, of the Regulation in relation to the processing of personal data carried out by the Company, through the Replika service on 2 February 2023.

The Company did not provide feedback to the communication note of initiation of the proceeding nor did it request to be heard pursuant to art. 166, paragraph 6, of the Code and art. 13 of the regulation of the Guarantor no. 1/2019. 

In the act of initiation of the proceeding, which is intended to be fully and expressly referred to here, the Authority contested the Company with three violations on the basis of the critical issues identified in the emergency provision no. 39/2023. The analysis carried out by the Authority focused on the state of the facts, processing and obligations implemented by Luka as of 2 February 2023.

With reference to the failure to identify the condition of lawfulness of the processing, the Authority noted that in the text of the privacy policy published on the date of adoption of the emergency provision of the Guarantor, updated to 5 July 2022, the legal basis underlying the various processing operations carried out by the Company within the Replika service had not been identified in a granular manner. The reference to the legal bases of the execution of a contract (art. 6, par. 1, letter b), of the Regulation) and of the consent of the interested parties (art. 6, par. 1, letter a), of the Regulation) as well as the reference to a generic authorization (“authorization”, not obligation) by law, were not in fact referred, nor referable, to specific processing operations (so-called granularity), with the consequent impossibility of identifying and evaluating the suitability of the legal bases themselves. Furthermore, the privacy policy dated 5 July 2022, in force as of 2 February 2023, did not reveal any reference to the legal basis underlying the processing of personal data for the purpose of developing the LLM model that powers the chatbot, nor did the documentation subsequently produced, in particular the privacy policy, also in the version updated to 22 March 2023, and the DPIA, provide elements from which to derive evidence that the Company had identified a legal basis for this purpose at a time prior to 2 February 2023.

In light of the above, the Authority contested Luka for the possible violation of art. 5, par. 1, letter a) and art. 6 of the Regulation for having failed to identify, as of 2 February 2023, the legal bases of the various processing operations carried out through the Replika service.

With reference to transparency obligations, the Authority's assessment concerned the privacy policy in force as of 2 February 2023, i.e. the version updated as of 5 July 2022. From a formal point of view, the Authority, in the act of initiating the proceeding, noted that as of 2 February 2023 the privacy policy was only available in English (including for minors) and was not easy to find. From a content point of view, it was noted that as of 2 February 2023 the privacy policy:

-    did not report any indication of the relevant legal basis in correspondence with the processing activities carried out and the type of data processed;

-    did not indicate the purposes of the processing with reference, in particular to the two distinct types of processing, i.e. the processing aimed at "Chatbot Interaction" and that aimed at "Model Development";

-    in the sections “People mentioned in the chat” and “Integration with your Instagram account” two categories of personal data processed in order to allow users to have conversations were indicated; 

-    it did not clarify that the service was offered exclusively to adults given that, as reported above, the privacy policy only included a reference to minors under 13 years of age in compliance with the requirements imposed by COPPA (Children's Online Privacy Protection Act);

-    it did not provide any specific indication regarding the period of retention of personal data or the criteria used to determine this period;

-    it did not clarify whether there was a transfer of personal data outside the EEA and, if so, what the legal basis and the guarantees of adequacy were as per Chapter V of the Regulation. In particular, the text of the privacy policy (see, in particular, the wording “By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen” is in open contradiction with what was stated by the Company itself in the note of 8 May 2023 (protocol no. 74173/23), where it is stated that, since the criterion of establishment in the European Union does not apply, no transfer of personal data from the European Union (specifically, from Italy) to the United States of America would be possible, pursuant to Chapter V of the Regulation;

-    in section 6 entitled “Your data protection rights”, the privacy policy provided specific information regarding the right under Article 22 of the Regulation, even though the provision was not expressly referred to. This reference (no longer present in the version dated 22 March 2023), was able to make the user believe that his/her personal data were the subject of an automated decision-making process in violation of the principles of transparency and fairness. This circumstance was denied by the owner himself in the response note (prot.74173/23), where he argued that "although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling that have legal effects or similar relevance pursuant to Article 22 of the Regulation".

In light of the above, the Authority contested Luka for the possible violation of art. 5, par. 1, letter a), 6, 12 and 13 of the Regulation given that, as of 2 February 2023, the privacy policy relating to the Replika service was not compliant with the obligations and general principles regarding transparency and was provided in ways and times that did not allow users to promptly view it.

Finally, with reference to the absence of mechanisms for verifying the age of minors, the Authority contested the absence of measures aimed at ensuring specific protection for minors in relation to access and use of the Replika service as of 2 February 2023. In particular, the absence of:

-    a procedure for verifying the user's age (the system only required name, email and gender) with the consequent risk of proposing to minors responses that were unsuitable for their level of development and self-awareness, including sexually explicit content;

-    interdiction or blocking mechanisms even in the face of declarations by the user that made his or her minority evident, as well as the proposition of responses by the chatbot that were clearly in conflict with the protections that should be ensured to minors and, more generally, to all the most vulnerable subjects.

The Authority, in the act of initiating the proceeding, acknowledged that the Company implemented age verification mechanisms following the request of the Guarantor formulated in the context of the provisional limitation measure, adopted urgently on 2 February 2023. In particular, during the discussions that followed the adoption of the aforementioned measure and with specific reference to the age verification profile, the owner represented that it had implemented an age gate on all registration pages for the Services aimed at limiting access to adult users only and that the age verification mechanism includes a "cooling-off period" aimed at preventing the user, having ascertained the impossibility of accessing by entering real personal data, from immediately entering a different date of birth suitable to allow him to access the service. The Company also represented that a process was being developed to use language analysis in order to identify and prevent the use of the Services by persons under the age of 18.

In light of the above, the Authority has contested Luka for the possible violation of art. 5, par. 1, letter c), 6; 7; 8; 24 and 25, par. 1 of the Regulation for the failure to provide suitable systems to verify the age of the subjects as of 2 February 2023.

4. EXISTENCE OF EUROPEAN JURISDICTION AND COMPETENCE OF THE GUARANTOR

First of all, the Authority deems it appropriate to address the issues relating to the applicability of European legislation on the protection of personal data to the service offered by Luka and to the competence of the Guarantor, also taking into account the exceptions raised by the Company in the response dated 8 May 2023 to the request for information sent by the Authority. 

Art. 3 of the Regulation governs the territorial scope of application of the legislation by establishing different criteria depending on whether or not the data controller is established in the territory of the European Union. 

In the first hypothesis (art. 3, par. 1, so-called establishment criterion), the Regulation applies regardless of whether the processing is carried out in the Union and the competence is identified in compliance with the so-called one-stop-shop mechanism, pursuant to art. 56 of the Regulation itself.

In the second hypothesis (Article 3, paragraph 2, so-called targeting criterion), the Regulation applies to the processing of personal data of data subjects who are in the Union if the processing activities concern: i) the offering of goods or services to data subjects in the Union (Article 3, paragraph 2, letter a), of the Regulation); ii) the monitoring of the behavior of data subjects who are in the Union to the extent that such behavior takes place in the Union itself (Article 3, paragraph 2, letter b), of the Regulation).

The Company stated in the above-mentioned note that it has a sole establishment in the European Union in the Netherlands, reporting that it has “a group of employees located in the Netherlands, including several decision makers involved in cross-border data processing for the development of the LLMs and the product” and that “the Company’s employees located in the Netherlands are involved in decisions that affect the processing of personal data by the Company and the operation of the LLMs globally, including decisions that affect the smallest proportion of users located in Italy”. The existence of a Dutch establishment in the European Union would imply the application of the one-stop-shop mechanism and the competence, as lead supervisory authority and in cooperation with the authorities concerned, of the Dutch data protection authority.

However, this statement is not supported by any documents. In fact, both in the privacy policy published on Replika's website as of February 2, 2023 (version updated to July 5, 2022) and in subsequent versions thereof (including the current one updated to February 23, 2024) there is no mention of a Company establishment in the Netherlands; similarly, no mention can be found in the terms of service (neither in the version updated to September 14, 2022 nor in the current one, updated to February 7, 2023), where, on the contrary, it is stated that Luka is "a software company that designed and built Replika, incorporated in Delaware, and operating in San Francisco, CA".

Moreover, the declarations referred to in the note of 8 May 2023 are absolutely generic since the name and corporate name of the company that would be established in the European Union are not even indicated (thus making any verification through mutual assistance with the Dutch supervisory authority pursuant to art. 61 of the Regulation impossible) and are not supported by any documentary evidence (e.g. articles of association of the Dutch company or Chamber of Commerce certificate).

At present, therefore, the Authority believes that no element has been provided that can validly demonstrate the existence of an establishment of the Company in the European Union with the consequent applicability of the establishment criterion pursuant to art. 3, par. 1, of the Regulation and of the one-stop-shop mechanism for the benefit of the Dutch data protection authority.

In this case, the existence of the European jurisdiction and the competence of the Guarantor must be ascertained on the basis of the targeting criterion pursuant to art. 3, par. 2, of the Regulation: specifically, it is therefore necessary to preliminarily assess whether the Replika service can be considered offered to interested parties located in the European Union for the purposes of the applicability of letter a) of the aforementioned art. 3 of the Regulation.

In this regard, reference is made to the “Guidelines 3/2018 on territorial scope”, adopted by the European Data Protection Board (EDPB) on 12 November 2019, which provide that the “controller… demonstrates his intention to offer goods or services to a data subject who is in the Union” (see par. 2 (a) of the cited Guidelines) and the case law of the Court of Justice of the European Union (judgment Pammer/Reederei Karl Schlüter GmbH & Co and Hotel Alpenhof/Heller - joined cases C-585/08 and C-144/09), which has indicated some factors in the presence of which it can be considered that a commercial activity carried out by a subject is directed towards a Member State, including the circumstance that the European Union is mentioned in reference to the good or service offered, the international nature of the activity or the launch of advertising and marketing campaigns aimed at the public of an EU country.

In this case, the evidence that the Replika service was offered to data subjects who were in the European Union and, in particular, in Italy on 2 February 2023, emerges “per tabulas” from the Company’s first response to the temporary limitation order contained in the emergency provision of the Guarantor no. 39/2023, where it is stated (see note of 3 March 2023, page 1) that “the Company promptly complied with the request for temporary limitation of processing for users established in Italian territory, promptly inhibiting access to both the app and the website of the service from Italy”.

Having demonstrated in the ways and terms set out above the territorial applicability of the Regulation and the competence of the Guarantor, the following is observed.

The processing of personal data carried out by Luka can be classified as cross-border processing of personal data pursuant to art. 4, par. 1, no. 23 of the Regulation, as it is capable of affecting data subjects in more than one Member State.

For this type of processing, where the controller has identified a single or main establishment in the European Union, as already illustrated, the cooperation mechanism described in Articles 60 et seq. of the Regulation applies and the competence to exercise the tasks and powers referred to in Articles 57 and 58 of the Regulation is rooted, pursuant to Article 56, par. 1, of the Regulation, in the lead supervisory authority, i.e. the supervisory authority of the Member State in which the single or main establishment is located.

If, on the contrary, as in the present case, there is no establishment of the data controller in the European territory, the latter will have to "interface with the supervisory authorities of each Member State in which it operates through the designated representative" (see par. 3.3. of the "Guidelines on the Lead Supervisory Authority" adopted by the Article 29 Working Party on 13 December 2016, revised on 5 April 2017 and adopted by the EDPB on 25 May 2018).
In fact, where a controller does not have an establishment in the European Union (or rather in the EEA area), the special rule under art. 56 does not apply in favour of the general rule under art. 55, par. 1, of the Regulation according to which "each Supervisory Authority shall be competent to perform the tasks assigned to it and to exercise the powers conferred upon it under the (...) Regulation in the territory of the respective Member State".

In the case in question, as mentioned, Luka is a company based in the United States of America that has not demonstrated that it has an establishment in the territory of the European Union. Therefore, the Italian data protection authority is competent to assess, with regard to its territory, the compliance with the Regulation of the processing of personal data carried out by the Company and to exercise the powers granted to it by art. 58 of the Regulation.

5. THE CONFIRMED VIOLATIONS

5.1 ARTT. 5, PAR. 1, LETT. A) AND 6 OF THE REGULATION

The Office contested Luka's violation of art. 5, par. 1, lett. a) and 6 of the Regulation for not having identified, as of 2 February 2023, the legal bases of the various processing operations carried out through the Replika service, a service offered and available to the public in Italy on that date. 

Art. 5, par. 1, of the Regulation prescribes that “personal data shall be: a) processed lawfully, fairly and in a transparent manner in relation to the data subject («lawfulness, fairness and transparency»); b) collected for specified, explicit and legitimate purposes, and not further processed in a manner that is incompatible with those purposes; further processing of personal data for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes shall not, in accordance with Article 89, paragraph 1, be considered incompatible with the initial purposes («purpose limitation»); c) adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed («data minimisation»); d) accurate and, where necessary, kept up to date; every reasonable step shall be taken to ensure that personal data that are inaccurate, having regard to the purposes for which they are processed, are erased or rectified without delay («accuracy»); e) kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which they are processed; personal data may be retained for longer periods insofar as the personal data will be processed solely for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes in accordance with Article 89(1), subject to implementation of the appropriate technical and organisational measures required by this Regulation in order to safeguard the rights and freedoms of the data subject (‘storage limitation’); (f) processed in a manner that ensures appropriate security of the personal data, including protection against unauthorised or unlawful processing and against accidental loss, destruction or damage, using appropriate technical or organisational measures (‘integrity and confidentiality’). Paragraph 2 of the same provision provides that “The controller shall be responsible for and able to demonstrate compliance with paragraph 1 (‘accountability’)”.

Recital 39 clarifies that “Any processing of personal data should be lawful and fair. It should be transparent to natural persons how personal data relating to them are collected, used, consulted or otherwise processed, as well as to what extent the personal data are or will be processed. The principle of transparency requires that information and communication relating to the processing of such personal data be easily accessible and understandable and that clear and plain language be used. This principle concerns, in particular, the information of data subjects on the identity of the controller and the purposes of the processing and further information to ensure fair and transparent processing with regard to the natural persons concerned and their rights to obtain confirmation and communication of whether personal data relating to them are being processed”.

Article 6 of the Regulation prescribes the conditions of lawfulness of the processing by listing the six possible legal bases (consent, contract, legal obligation, vital interest, public interest, legitimate interest) on which the owner must rely in order to lawfully process the personal data necessary for the performance of his business. The legal basis, as clarified by the EDPB, “must be identified before the processing is implemented and must be specified in the information provided to the data subjects in accordance with Articles 13 and 14. (see Guideline 2/2019 on the processing of personal data pursuant to Article 6, paragraph 1, letter b), of the General Data Protection Regulation in the context of the provision of online services to data subjects”)”.

The Company did not submit written defences or documents, pursuant to art. 166, paragraph 5 of the Code, following the notification of the notice of contestation and initiation of the proceedings by the Office and therefore did not provide any counter-arguments with respect to the hypothesis of infringement relating to the failure to indicate the legal basis for each of the processing activities carried out by Luka within the Replika service.

In this case, from the investigative documents, in particular from the text of the privacy policy published on the date of the adoption of the emergency measure of the Garante, updated to 5 July 2022, it clearly emerges that the Company has not identified in a granular manner the legal basis underlying the various processing operations carried out by the Company within the scope of the Replika service, including that for the processing of data used for the development of the LLM model.

The only references, in the introductory part of the text in question, are the following: “We care about the protection and confidentiality of your data. We therefore only process your data to the extent that:

•    It is necessary to provide the Replika services you are requesting,

•    You have given your consent to the processing, or

•    We are otherwise authorized to do so under the data protection laws”.

The legal bases of the execution of a contract (art. 6, par. 1, lett. b), of the Regulation), of the consent of the interested parties (art. 6, par. 1, lett. a), of the Regulation) and of a legal authorization (where, moreover, the Regulation provides among the legal bases a legal obligation, not a mere authorization), are recalled implicitly and generically, without referring to specific processing operations (so-called granularity principle), with the consequent impossibility of identifying and assessing their suitability.

Finally, neither the privacy policy nor the documentation in the files make reference to the legal basis underlying the processing of personal data aimed at developing the LLM model that powered the chatbot at the date of February 2, 2023.

Specifically, the findings provided by Luka, although pertinent, are not conclusive. In particular, the DPIA and the privacy policy produced on May 8, 2023, do not allow to overcome the objections raised by the Authority in the dispute with reference to the principle of lawfulness and the legal basis of the processing, respectively governed by Articles 5, par. 1, letter a) and 6 of the Regulation, since:

-    the privacy policy, even in the subsequent version updated on March 22, 2023, in the table in paragraph 2.A, does not expressly mention the purpose of the “Model Development” or the related legal basis;

-    the DPIA, although distinguishing the two processing purposes relating to the “Chatbot Interaction” and the “Model Development” (par. I) and analyzing the respective legal bases (par. II), does not present a certain date, therefore it does not prove that the identification of the aforementioned conditions of lawfulness referred to in Article 6 of the Regulation occurred prior to 2 February 2023. Moreover, the DPIA states the legitimate interest as a legal basis for processing for the purposes of "Model Development" without indicating any argument relating to the so-called "triple test" underlying the legitimate interest assessment. Finally, it is highlighted that the DPIA, although an excellent accountability tool, does not constitute the place elected by the legislator in order to provide data subjects with information relating to processing activities, information that must be provided through the privacy policy.

With reference to art. 5, par. 1, letter a), of the Regulation, reference is made here to the principle expressed by the EDPB in the binding decision no. 1/2021 on transparency, but also applicable with reference to lawfulness, according to which the principles set out in art. 5 of the Regulation must be considered as a general concept that is then concretely implemented in various specific provisions and obligations (in the event of lawfulness, in Articles 6, 7, 8, 9 and 10 of the Regulation).

Therefore, according to the EDPB, it is necessary to distinguish the specific obligations deriving from a principle (in this case, Article 6 of the Regulation) from the principle itself expressed in Article 5 of the Regulation, since the latter cannot be limited to the specific obligation, even if the latter is a concretization of the former.

The principle of lawfulness, in fact, is an all-encompassing principle that reinforces other principles (e.g. fairness, accountability). Confirmation of this reconstruction is given by the fact that Article 83, paragraph 5, of the Regulation provides for the possibility of sanctioning the violation of the obligations of lawfulness independently of the violation of the principle itself. In this case, the Authority considers that the violation of the principle of lawfulness referred to in Article 5, paragraph 1, letter a) of the Regulation can also be identified, taking into account the seriousness (failure to clearly and granularly identify the legal bases underlying to the various processing operations) of the nature (this is an essential element of the processing) and of the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the single specific violation of the obligation under Article 6 of the Regulation.

In light of the above, the Authority considers that Luka has not identified, as of 2 February 2023, the legal bases of the various processing operations carried out through the Replika service, offered and available to the public in Italy on that date, in violation of Articles 5, paragraph 1, letter a) and 6 of the Regulation.

With regard to the analysis and assessment of merit in relation to the legal bases under Article 6, paragraph 1, letter b) and letter f), of the Regulation allegedly underlying the use of the chatbot and the post-training of the LLM model underlying the Replika service, as well as, in general, the legal bases relating to the entire life cycle of the system of generative artificial intelligence implemented by the Company, the Authority reserves the right to open a separate and independent investigation.

5.2 ARTT. 5, PAR. 1, LETT. A), 12 AND 13 OF THE REGULATION

The Office contested Luka for violating articles 5, par. 1, lett. a), 12 and 13 of the Regulation for having provided, on 2 February 2023, a privacy policy relating to the Replika service with content that does not comply with the obligations and general principles of transparency provided for by the legislation.

Article 5, par. 1, lett. a), of the Regulation requires that personal data be processed lawfully, fairly and transparently in relation to the data subject (principle of lawfulness, fairness and transparency).
Article 12 of the Regulation establishes rules on transparency and methods of exercising rights, while Article 13 introduces specific indications regarding the information that the owner is required to provide if the personal data are collected from the data subject.

On the subject of transparency, Recital 58 of the Regulation provides that information intended for the public or the data subject must be concise, easily accessible and easy to understand, that clear and plain language must be used, and, with reference to the specific protection to which minors must be addressed, provides that "where the processing of data concerns them, any information and communication should use clear and plain language that a minor can easily understand".

On the subject of transparency, the indications of the Committee are also relevant, in particular Guidelines 2/2019 on the processing of personal data pursuant to Article 6, paragraph 1, letter b) of the General Data Protection Regulation in the context of the provision of online services to data subjects, where it is provided that the legal basis of the processing, in addition to having to be identified before the implementation of the processing, "must be specified in the information provided to data subjects in accordance with Articles 13 and 14"; furthermore, the Guidelines no. 1/2022 of the Committee on access are relevant, which prescribe, in paragraph 142, that “a controller offering a service in a given country should also respond in a language understood by data subjects in that country”.

Finally, the guidelines adopted by the Article 29 Working Party on 11 April 2018, clarified that “the concept of transparency in the Regulation is not legalistic, but rather user-centric and is embodied in several articles containing specific obligations imposed on controllers and processors. The concrete (information) obligations are set out in Articles 12-14 of the Regulation. (…) The transparency obligations imposed by the Regulation apply regardless of the legal basis of the processing and throughout the lifecycle of the processing. This is clear from Article 12, which states that transparency applies at the following stages of the data processing cycle: (i) before or at the beginning of the data processing cycle, i.e. when the personal data are collected from the data subject or otherwise obtained; (ii) throughout the lifecycle of the processing, i.e. when communicating with data subjects about their rights; (ii) at specific moments when processing is ongoing, for example when a data breach occurs or in the event of a significant modification of the processing”.

The Company did not submit any written defense or documents, pursuant to art. 166, paragraph 5 of the Code, in response to the Office's notice of challenge and initiation of the proceedings and therefore did not provide any counter-arguments with respect to the hypothesis of violation formulated relating to the obligations and general principles of transparency provided for by the legislation.

The Office's investigation, as already specified above, concerned the privacy policy adopted and published by Luka on 2 February 2023, i.e. the version of the same updated to 5 July 2022.

First of all, the investigation documents show that, from a formal point of view, on 2 February 2023, the privacy policy was only available in English, not considering the language of the country in which the service was offered, namely Italian. 

From a substantive perspective, it is noted that as of 2 February 2023, the privacy policy did not comply with the principles of correctness and transparency as it was incomplete and incorrect.

In particular, from the point of view of the completeness of the information provided to the interested parties, it was found that the privacy policy:

-    did not indicate in a granular manner the legal basis relating to each of the processing activities carried out, nor the type of data processed;

-    did not indicate the purposes of the two distinct types of processing activities, namely the processing of data through the “Chatbot Interaction”, aimed at allowing users to register for the service and interact with the platform, and the processing of data in the context of the “Model Development”, aimed at improving the security and performance of the Large Language Model (LLM) underlying the service offered (“Model Development”);

-    did not clarify that the service was offered exclusively to adults, while inviting minors under 13 not to use the service. In particular, paragraph 8 of the aforementioned privacy policy stated: “We do not knowingly collect Personal Data from children under the age of 13. If you are under the age of 13, please do not submit any Personal Data through the Services. We encourage parents and legal guardians to monitor their children’s Internet usage and to help enforce our Privacy Policy by instructing their children never to provide Personal Data on the Services without their permission. If you have reason to believe that a child under the age of 13 has provided Personal Data to us through the Services, please contact us, and we will endeavor to delete that information from our databases”. This information, while making clear the circumstance for which the service was not offered to subjects under the age of 13 (“If you are under the age of 13, please do not submit any Personal Data through the Service”), did not clarify that the chatbot was reserved only for adults, since users between the ages of 13 and 18 were also excluded.

This last circumstance was clarified by the Company only at a later time;

-    did not provide any specific indication regarding the period of retention of personal data or the criteria used to determine such period;

-    did not clarify whether there was a transfer of personal data outside the EEA and, if so, what the legal basis of the processing was and the adequacy guarantees adopted pursuant to Chapter V of the Regulation. In more detail, the information provided by the Company (specifically in the part of the privacy policy in which it stated that: “By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen”) was likely to generate an erroneous belief in the data subject regarding the transfer of his or her personal data to the USA. The absence of a transfer of data to third countries was confirmed by the Company itself in the note of 8 May 2023 (prot. 74173/23) relating to the applicability of the criterion of establishment in the European Union. Therefore, it is noted that the presence of misleading information was confirmed in the declarations of the owner;

-    in section 6 “Your data protection rights”, although not expressly referring to Article 22 of the Regulation, it provided specific information regarding this right, generating in the user the unfounded belief that his/her personal data were subject to an automated decision-making process. The absence of automated processing pursuant to Article 22 of the Regulation was confirmed by the owner himself in the feedback note (prot.74173/23), where he argued that “although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling that have legal effects or similar relevance pursuant to Article 22 of the Regulation”. Therefore, also in this case, it is found that the presence of misleading information has been confirmed in the same declarations of the owner.

With reference to art. 5, par. 1, letter a), of the Regulation, reference is made to the same binding decision of the EDPB cited in the previous paragraph (binding decision no. 1/2021), according to which transparency must be considered a general concept that finds concrete implementation in various provisions and specific obligations (for example, arts. 12, 13, 14, 25 and 35). It is therefore necessary to distinguish the specific obligations deriving from the principle of transparency (referred to in articles 12-14 of the Regulation) from the principle expressed in art. 5 of the Regulation, since although these obligations are a concretization of the general principle, the latter has a broader scope.

The principle of transparency, in fact, is an all-encompassing principle that strengthens other principles (e.g. fairness, accountability). Confirmation of this reconstruction is given by the fact that art. 83, par. 5, of the Regulation provides for the possibility of sanctioning the violation of transparency obligations independently of the violation of the principle itself. In other words, the transparency obligations do not define the entire scope of the transparency principle, it follows that the violation of the transparency obligations provided for by Articles 12-14 of the Regulation may also constitute a violation of the transparency principle only if characterized by elements of seriousness and systematicity.

In this case, the Authority considers that the violation of the transparency principle referred to in Article 5, par. 1, letter a) of the Regulation can also be identified, taking into account the seriousness (lack of information to the interested parties about the legal bases underlying the various processing operations of their personal data) of the nature (lack of clear information about the essential elements of the processing such as legal basis, purpose, principle of conservation, transfer outside the EU) and the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the individual specific violations of the obligations referred to in Articles. 12 and 13 of the Regulation.

For the reasons stated above, the Authority believes that Luka violated, as of 2 February 2023, Articles 5, paragraph 1, letter a), 12 and 13 of the Regulation. 

For the sake of completeness, it should be noted that further technical investigations have revealed that the data controller has again updated the privacy policy relating to the Replika service on 23 February 2024. In this version, some of the inaccurate and incorrect information indicated above has been modified. In particular, the privacy policy, in force on the date of adoption of this provision, reports in a granular manner the legal basis relating to each of the processing activities carried out by the data controller and the type of data processed; it expressly clarifies that the service is offered exclusively to adults, and does not contain any reference, not even implicit, to the automated decisions referred to in Article 22 of the Regulation. Nonetheless, the information provided pursuant to Articles 12 and 13 of the Regulation continue to be available only in English, do not provide specific references regarding the period of retention of personal data or the criteria used to determine this period and are potentially suitable to generate in the interested party an erroneous belief regarding the transfer of his/her personal data to the USA.

5.3 ARTT.   5, PAR. 1, LETT. C), 6, 7, 8, 24 AND 25, PAR. 1, OF THE REGULATION

The Office contested Luka for the violation of arts. 5, par. 1, lett. c); 6; 7; 8; 24 and 25, par. 1 of the Regulation for failure to prepare systems to verify the age of the subjects as of 2 February 2023. 

Pursuant to article 5, par. 1, lett. c), of the Regulation “Personal data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed”.

Pursuant to art. 24, par. 1, of the Regulation “Taking into account the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for the rights and freedoms of natural persons, the controller shall implement appropriate technical and organisational measures to ensure and to be able to demonstrate that processing is performed in accordance with this Regulation. Those measures shall be reviewed and updated where necessary”.

Pursuant to art. 25, par. 1, of the Regulation, the controller shall adopt such measures “taking into account the state of the art, the costs of implementation and the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for the rights and freedoms of natural persons posed by the processing, both at the time of determining the means for processing and at the time of the processing itself”.

In the guidelines no. 4/2019 on Article 25 of the Regulation, the EPDB clarified that “the core of the provision is to ensure adequate and effective data protection by design and protection by default, which means that controllers should be able to demonstrate that they incorporate appropriate measures and safeguards into the processing to ensure the effectiveness of the data protection principles and the rights and freedoms of data subjects” and invited controllers to take into account, in the context of designing and setting up the processing in a privacy-oriented perspective, also the obligations to provide specific protection to minors and other groups of vulnerable subjects.

In the same guidelines, the EDPB also underlined that “In line with Article 25, paragraph 1, the controller shall implement appropriate technical and organizational measures that are designed to implement the data protection principles and shall integrate the necessary safeguards into the processing to comply with the requirements and protect the rights and freedoms of data subjects. Both adequate measures and necessary safeguards aim to pursue the same purpose of protecting the rights of data subjects and ensuring that the protection of their personal data is integrated into the processing. The expressions technical and organizational measures and necessary safeguards can be understood in a broad sense as any method or means that a controller may employ in processing. The term adequate means that the measures and necessary safeguards must be suitable for achieving the intended purpose, i.e. they must effectively implement the data protection principles”.

The Company has not submitted written defenses or documents, pursuant to art. 166, co. 5 of the Code, in response to the Office's contestation and initiation of proceedings and therefore did not provide any counter-arguments with respect to the hypothesis of violation formulated in relation to the failure to prepare systems to verify the age of the subjects.

In light of the aforementioned rules and guidelines, the Authority observes that the data controller is obliged to implement suitable technical and organizational measures to guarantee and demonstrate that the processing is carried out in compliance with the Regulation and to process only adequate, relevant and limited data to what is necessary with respect to the purposes of the processing itself.

However, the investigation revealed that the Company had failed to prepare measures to guarantee specific protection for personal data processed within the Replika service relating to minors under 18 years of age. Indeed, the absence of procedures to verify the age of the user who intended to access the service, as well as of interdiction or blocking mechanisms in the event of declarations by the user that made his or her minority clear, highlighted how the owner had not assessed, ex ante, the risks that registration and use of the service by minors under 18 could give rise to, with the consequence that, on the one hand, it did not adopt any measures to counter such risks, minimize them or limit them, and on the other hand, it processed data in excess of those necessary to satisfy the purposes of the processing (i.e. offering the service to adult users).

The investigation revealed that as of 2 February 2023, the Company did not provide any mechanism to verify the age of users either when registering for the Replika service or during its use, even though it excluded minors among potential users.

In particular, the absence of:

-    a procedure for verifying the user's age (the system only required name, email and gender) was detected, with the consequent risk of proposing to minors responses that were unsuitable for their level of development and self-awareness, including sexually explicit content;

-    interdiction or blocking mechanisms even in the face of user declarations that made their minority evident, as well as the proposition of responses by the chatbot that were clearly in contrast with the protections that should be ensured to minors and, more generally, to all the most vulnerable subjects.

Therefore, until 2 February 2023, the Company has not adopted any technical and organizational measures to ensure compliance with the general principles of the Regulation and the protection of the rights and freedoms of minors, thus exposing minors to significant risks to their person that the legislation in question aims to limit, including responses that are unsuitable with respect to their level of psychophysical development and self-awareness. Luka adopted age verification mechanisms only following the request of the Guarantor formulated in the context of the provisional limitation measure, adopted urgently on 2 February 2023. In particular, during the discussions that followed the adoption of the aforementioned measure and with specific reference to the age gate profile, the Company represented that it had implemented an age gate on all registration pages to the Services that limits access to users who have turned 18 and that includes a "cooling off period" aimed at preventing people who have not been able to access by entering real personal data from entering, immediately afterwards, a different date of birth. 

The Company also represented that it had planned to develop a process to use language analysis in order to identify and prevent the use of the Services by people under the age of 18. 

Before the Authority's intervention, therefore, all users, including minors, could register for the Replika service and use it without being asked to undergo any age verification. As already clarified in the dispute, in the Authority's opinion, the absence of a common standard suitable for guaranteeing, in a certain and absolute manner, the effectiveness of a user age verification model and the discussion still underway at European level in this regard, cannot be considered sufficient reasons to exclude the fulfillment of the obligations to which the data controller is bound, in particular that of verifying the user's actual contractual capacity for the purposes of the validity of the contract.

From the foregoing it emerges that, as of 2 February 2023, the Company has neither implemented, pursuant to art. 24 of the Regulation, measures aimed at ensuring that the processing of data at the time of registration for the Replika service complies with art. 5, par. 1, lett. c), 24, 25 and of the Regulation, in particular that the same had adopted technical and organizational measures "aimed at effectively implementing the principles of data protection, such as minimization, and at integrating the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of the data subjects" with consequent processing of data that were superfluous, or rather unnecessary, with respect to the purposes of a service that, according to the declarations of the same owner as well as the documentation in the files, was offered only to users over 18 years of age.

With specific reference to the violation referred to in art. 5, par. 1, letter c), of the Regulation, it is noted that, in this case, the adoption of adequate technical and organizational measures, from the design stage, aimed "at effectively implementing the principles of data protection, such as minimization", as well as being a constituent element of art. 25, par. 1, of the Regulation also constitutes a quid pluris suitable to consider the violation of the principle of minimization as integrated, in line with what is clarified by recital 78 of the Regulation.

Specifically, the failure by the Company to adopt suitable measures to safeguard access to and use of the Replika service meant not only that Luka systematically processed personal data in addition to those actually necessary to achieve the purpose of the processing (i.e. offering the service to adult users), but also that such processing concerned data relating to vulnerable subjects (minors, potentially even under the age of 13) who, due to this deficiency and given the innovative technology underlying the service and the highly sensitive nature of the conversations provided by the chatbot, were exposed to a particularly high risk. 

The news reports, which led to the Authority's investigation, together with specific cases of self-harm related to the use of the chatbot reported by the foreign press and brought to the attention of the judicial authority, are elements that support the Office's challenge and which, on the basis of the principles expressed by the repeatedly recalled binding decision of the EDPB no. 1/2021, require the enhancement of the seriousness and impact of the violations, making it considered integrated both the violation of the principle referred to in art. 5, par. 1, of the Regulation, and the specific violation of the obligation referred to in arts. 24 and 25, par. 1, of the Regulation.

The Authority, on the contrary, does not believe that there are sufficient elements to declare the violations, contested pursuant to art. 166, par. 5, of the Code, relating to consent, in particular the positive act expressed by the minor in the context of digital services, pursuant to Articles 6, 7, 8 of the Regulation. In particular, it is noted that during the investigation it emerged that, unlike what was erroneously indicated in the privacy policy in the version dated 2 February 2023 (see § 5.2) - the Replika service was not - and is not - offered to minors, it follows that the owner was not required to comply with the obligation to identify a legal basis for processing that was presumed not to be carried out. 

In light of the above, the Authority believes that Luka violated, as of 2 February 2023, Articles 5, par. 1, letter c) and 24 and 25, par. 1, of the Regulation.

For the sake of completeness, it should be noted that, as of the date of adoption of this provision, further technical investigations have revealed that the age verification system currently implemented by the data controller continues to be deficient in several respects; in particular, it has been ascertained that:

- after the creation of the user profile, it is possible to change the date of birth in the "My Profile" section without any verification by the data controller, with the result that a minor who has registered for the service indicating a false age could promptly change it by entering the correct one without any consequences, continuing to be able to access the service;

- the cooling off period (of 24 hours) does not operate where the profile is created via incognito browsing; in fact, as far as it appears, once the first age check has failed, it is possible to successfully complete registration for the service by changing the email address entered with a new address (including non-existent and non-functioning email addresses);

- linguistic analysis mechanisms have not been set up that require users to reconfirm their age through the age gate process when they identify themselves as minors under 18, with the sole exception of the case in which the user himself provides specific input (e.g. unequivocally declares himself to be under 18). In such cases, the application responds by asking for confirmation of the age of majority. 

During the technical checks, it also emerged that the user is offered the possibility of marking some conversations as inappropriate, however it is not possible to detect the consequences of such reporting. 

6. CONCLUSIONS

In light of the assessments expressed above, the existence, in the terms indicated below, of the majority of the violations contested by the Office notified with the act initiating the procedure is confirmed and the unlawfulness of the processing of personal data carried out by the Company is declared, in violation of art. 5, par. 1, letter a) (with reference to both the principle of lawfulness and transparency) and letter c) of the GDPR. c), 6, 12, 13, 24 and 25, paragraph 1, of the Regulation.

The ascertainment of the violation of the aforementioned provisions of the Regulation requires the consequent adoption of corrective measures pursuant to art. 58, paragraph 2, of the Regulation, in particular the order for compliance pursuant to art. 58, paragraph 2, letter d) of the Regulation and also makes applicable, pursuant to art. 58, paragraph 2, letter i), of the Regulation, the administrative sanction provided for by art. 83, paragraphs 3 and 5, of the Regulation itself. 

Furthermore, taking into account the particular sensitivity of the data processed, it is also believed that the accessory sanction of publication on the website of the Guarantor of this provision, provided for by art. 166, paragraph 7 of the Code and art. 16 of the Guarantor Regulation no. 1/2019, should also be applied.

The Authority reserves the right, as specified above, to examine and verify in an independent proceeding the profiles concerning the lawfulness of the processing carried out by the Company with specific reference to the legal bases of the processing of personal data relating to the entire life cycle of the generative artificial intelligence system underlying the Replika service.

7. CORRECTIVE MEASURES PURSUANT TO ART. 58, PARAGRAPH 2, LETTER D) OF THE REGULATION

Article 58, par. 2, of the Regulation provides for the Guarantor a series of corrective powers, of a prescriptive and sanctioning nature, to be exercised in the event that unlawful processing of personal data is ascertained.

Among these powers, Article 58, par. 2, lett. d), of the Regulation provides for the power to "order the data controller ... to bring the processing into conformity with the provisions of this Regulation, where appropriate, in a specific manner and within a specific period".

From what has been found and considered in the preceding paragraphs, it has emerged that Luka has violated, as of 2 February 2023, Articles 5, par. 1, letter a) (with reference to both the principle of lawfulness and the principle of transparency) and letter c), 6, 12 and 13, Articles 24 and 25, par. 1 of the Regulation but that, following the emergency intervention of the Authority, it has adopted some measures to remedy the critical issues that have emerged and, subsequently, has adopted further measures with respect to the violations contested in the act of initiation of the proceeding, which ensure the compliance of the processing with the legislation on the protection of personal data.

In particular, Luka has remedied the violation referred to in Articles 5, par. 1, letter a) and 6 of the Regulation by amending the privacy policy (see latest version dated 23 February 2024) specifying, in detail, the legal bases of the various processing operations carried out through the Replika service.

In light of the amendment of the privacy policy in the terms just described, it is believed that the conditions for the adoption of further corrective measures pursuant to art. 58, par. 2, of the Regulation do not currently exist.

On the other hand, with reference to the violation of articles art. 5, par. 1, letter a), 12 and 13 of the Regulation, relating to the information obligations and the violation of articles 24 and 25, par. 1 and 5, par. 1, letter c), of the Regulation, relating to the age verification system, there remain profiles of non-compliance with the Regulation that are believed to be the subject of specific provisions.

In particular, with reference to the information obligations, the Authority has ascertained that, to date, Luka's privacy policy (latest version dated 23 February 2024) continues to not comply with the legislation on the protection of personal data to the extent that i) it is only available in English, ii) it does not indicate in a timely manner the periods of retention of personal data or the criteria used to determine such periods, iii) it is likely to generate in the interested parties an erroneous belief regarding the transfer of their personal data to the USA.

Therefore, pursuant to Article 58, paragraph 2, letter d), of the Regulation, the controller is required to conform the privacy policy to Articles 5, paragraph 1, letter a), 12 and 13 of the Regulation by remedying the gaps indicated above.

Furthermore, with reference to the age verification system, the Authority has ascertained that, on the date of adoption of this provision, the age verification system used by the data controller does not comply with the principle of data minimization and the principles of privacy by design and by default, given that:

- after creating the profile, the user can change the date of birth in the "My Profile" section without any verification by the data controller. It follows that a minor who has registered for the service indicating a false age could promptly change it by entering the correct one without any consequences and continue to be able to access the service;

- the 24-hour cooling off period does not apply where the user creates the profile while browsing in incognito mode. In fact, as far as it appears, once the first age check has failed, the user can successfully complete the registration for the service by changing the email address entered with a new address (including non-existent and non-functioning email addresses);

- the data controller has not set up linguistic analysis mechanisms that require users to reconfirm their age through the age gate process in the presence of clear signals that identify a user under 18, with the sole exception of the case in which the user provides specific input (e.g. unequivocally declares that he or she is under 18). Only in the presence of such cases does the application respond by asking for confirmation of the age of majority. 

The Authority, however, positively found Luka's implementation of a function that allows the user to mark certain conversations as inappropriate, although it was not possible to detect the effect of this reporting. 

That said, pursuant to Article 58, paragraph 2, letter d) of the Regulation, the data controller is required to bring the age verification system into line with Articles 5, paragraph 1, letter c), 24 and 25, paragraph 1 of the Regulation, remedying the shortcomings indicated above. 

8. INJUNCTION ORDER FOR THE APPLICATION OF THE PECUNIARY ADMINISTRATIVE SANCTIONS AND ANCILLARY SANCTIONS

The Authority, pursuant to art. 58, par. 2, letter i), and 83 of the Regulation as well as art. 166 of the Code, has the power to impose an administrative pecuniary sanction pursuant to art. 83, in addition to or in place of the other corrective measures provided for in the same paragraph.

In determining the sanction, the Authority takes into account the principles and interpretation on the matter provided by the EDPB in the Guidelines 4/2022 on the calculation of administrative pecuniary sanctions pursuant to the GDPR, version 2.1, adopted on 24 May 2023.

On the basis of the arguments put forward above, the Guarantor has ascertained the violation of the following provisions of the Regulation: art. 5, par. 1, letter a) and 6; art. 5, par. 1, letter a), 12 and 13; art. 5, par. 1, letter c), 24 and 25, par. 1, of the Regulation.

In this case, it should first be noted that the Company has implemented a series of conducts that have integrated multiple violations, as specifically outlined and motivated in the previous paragraphs. The violations relating to the legal basis (art. 5, par. 2 and 6 of the Regulation), transparency (art. 5, par. 1, letter a), 12 and 13) and age gate (art. 24 and 25, par. 1) can be traced back, due to the principle of unity of action, under the aegis of art. 83, par. 3, of the Regulation, according to which in the presence of multiple violations of the Regulation, relating to the same processing or to connected processing, the total amount of the administrative pecuniary sanction cannot exceed the amount envisaged for the most serious violation.In particular, with reference to such violations, a hypothesis of linked processing can be configured, as defined in paragraph 28 of the aforementioned guidelines (a single conduct consists of multiple actions that are carried out on the basis of a single will and are contextually, spatially and temporally correlated in such a close way that they can be considered, from an objective point of view, as a single coherent conduct). The most serious violation compared to those mentioned above must be identified in the violation of transparency obligations given that both art. 5, par. 1, letter a) (principle of transparency) and arts. 12 and 13 (rights of the interested parties) are sanctioned pursuant to art. 83, par. 5, which sets the maximum amount of 20 million euros or, for companies, 4% of the annual worldwide turnover of the previous financial year, if higher. 

Pursuant to art. 83, par. 1 of the Regulation, the administrative sanction must be effective, proportionate and dissuasive in relation to the individual case. In the aforementioned guidelines, the EDPB has specified that the calculation of administrative pecuniary sanctions must start from a harmonized starting point, which constitutes the initial basis for the further calculation of the amount of the sanction, in which all the circumstances of the case are taken into account and weighed (see paragraph 46). The harmonized starting point must take into account three factors: 1) nature of the infringement pursuant to Article 83, paragraphs 4 to 6, of the Regulation; 2) seriousness of the infringement; 3) turnover of the company (see paragraph 48). Starting from the first profile, in the case in question, there are two infringements, in theory, of a more serious nature (Article 83, paragraph 5, of the Regulation) and one less serious infringement (Article 83, paragraph 4, of the Regulation). The first two refer to the violation of the legal basis and transparency, while the third concerns the violation of art. 25 of the Regulation.

As for the specific gravity, the elements to be taken into consideration are: a) nature, gravity and duration of the violation (art. 83, par. 2, letter a), of the Regulation); b) intentional or negligent character of the violation (art. 83, par. 2, letter b), of the Regulation); c) categories of personal data affected by the violation (art. 83, par. 2, letter g), of the Regulation). 

In the case in question, with reference to the three violations linked by the principle of unity of action (legal basis, transparency and data protection by design and data protection by default), the seriousness of the violations must be considered to be high given that: i) the nature of the violations relates to two fundamental principles (accountability and transparency), namely, on the one hand, the inability of the controller to demonstrate that he had identified the legal bases of the processing before the processing began and, on the other hand, the failure to provide appropriate information to the data subject, in particular with regard to the purpose of the two distinct types of processing (“Chatbot Interaction” and “Model Development”, and the involvement in the processing of data, such as that of minors, in addition to those necessary to satisfy the purpose of the processing of providing the service; ii) the nature of the processing involves significantly high risks as it is connected to an innovative, disruptive and rapidly expanding technology; iii) the object of the processing is of a cross-border nature of global scope with effects that are practically uncontrollable by the data subjects; iv) the purpose of the processing falls within the Company's core business; v) the number of Italian data subjects involved cannot be quantified with certainty, but, in general terms, it can be reasonably assumed that it is a very high number, since the information on the Google App Store (Google Play) shows that the application has exceeded 10 million downloads (leaving room for the assumption that a similar figure, although not found, can be recorded for downloads made on the Apple Store), while academic sources (Shikhar Ghosh, Replika: Embodying AI, Harvard Business School, Faculty & Research) report that the Company had already reached 10 million users in January 2022; vi) the nature of the data concerned, given the very nature of the chatbot (which still today presents itself as “an AI companion always ready to chat when you need an empathetic friend”), also particular data and, in the absence of age verification mechanisms and data filtering systems, personal information relating to minors. The duration of the violation is significant given that the app was released to the public in November 2017; in fact, the circumstance that the success of the chatbot is to be placed temporally in a later period does not constitute an element suitable to counterbalance the judgment of high severity, since the end of the violation depended on and coincided with the emergency intervention of the Guarantor. All violations must be considered negligent in nature. As stated by the Art. 29 Working Group, in the guidelines regarding the application and provision of administrative pecuniary sanctions for the purposes of Regulation (EU) no. 2016/679, adopted on 3 October 2017 and implemented by the EDPB on 25 May 2018 (WP 253 guidelines), intentional conduct refers to both awareness and intent (consciousness and will) to commit an offence, while negligent conduct lacks the intention to cause the infringement despite the failure to comply with a duty of care. The Court of Justice of the European Union (CJEU), with a recent ruling (judgment C-807/21 of 5 December 2023), has established that it is the supervisory authority's responsibility to establish that an infringement has been committed intentionally or negligently by the data controller, as only unlawful infringements can lead to the imposition of an administrative pecuniary sanction. In this regard, it should be noted that, while it is true that the CJEU has established in the aforementioned decision that art. 83 of the Regulation does not allow for the imposition of an administrative pecuniary sanction unless it is established that such infringement was committed intentionally or negligently by the data controller (see par. 75), it is also true that the Court itself upheld the basic principle of “ignorantia legis non excusat”, stating that “a data controller may be sanctioned for conduct falling within the scope of the GDPR if the data controller could not have been unaware of the unlawful nature of his conduct, regardless of whether he was aware of violating the provisions of the GDPR” (see par. 76). This principle had already been stated by the Court of Justice in another case (judgment C-601/16 of 25 March 2021, paragraphs 97 and 98) in which it had held that “an undertaking may be penalised for conduct falling within the scope of Article 101(1) TFEU where that undertaking could not have been unaware of the anti-competitive nature of its conduct, regardless of whether or not it was aware that it was infringing the competition rules of the Treaty (see, to that effect, judgment of 18 June 2013, Schenker & Co. and Others, C-681/11, EU:C:2013:404, paragraph 37). It follows that the fact that that undertaking has wrongly characterised in law its conduct on which the finding of the infringement is based cannot have the effect of exempting it from the imposition of a fine, since it could not have been unaware of the anti-competitive nature of its conduct”. anti-competitive nature of such conduct” (judgment of 18 June 2013, Schenker & Co. and others, C 681/11, EU:C:2013:404, paragraph 38). In this case, it is considered that Luka could not, at the time when its service was made available (also) to users located in the European Union and in particular in Italy, evade a duty to know and apply the Regulation which, as is known, protects a fundamental right provided for and protected by art. 8 of the Charter of Fundamental Rights of the European Union. In light of the circumstances of the specific case, the context in which the owner operates and the disruptive and rapidly expanding technology that characterises the activity carried out by the same, it is considered that the failure to comply with the processing of personal data with the European Union legislation constitutes the negligence underlying the concept of fault and demonstrates the existence of such a subjective element on the part of the Company. Furthermore, such fault must be considered serious precisely because of the breadth and innovative nature of the service offered, which involves large-scale processing of personal data worldwide.

Also for the purposes of quantifying the administrative pecuniary sanction, the aggravating factors referred to in art. 83, par. 2, letters d) and f) of the Regulation are relevant.

With regard to the first profile, the degree of responsibility of the data controller must be considered high due to the failure to adopt, at the time of launching the service, suitable technical and organizational measures to mitigate the risks for the rights and freedoms of the data subjects and to attribute to them the exercise of the prerogatives referred to in Chapter III of the Regulation. With regard to the second circumstance, with reference to the degree of cooperation, it should be noted that the Company, despite having responded to the request for information, did not produce any defense brief following the notification of the dispute pursuant to art. 166 of the Code, thereby giving rise to poor cooperation with the Authority. 

For the purposes of adopting the administrative sanction, account is taken, as a mitigating factor, of the measures implemented by the data controller to remedy the violation and mitigate its possible negative effects (Article 83, paragraph 2, letter f, of the Regulation), in particular:

-    the updates to the privacy policy, both immediately following provision 39/2023 and subsequently, in particular with reference to the latest version of the same dated 23 February 2024, as described in the previous paragraph, although such updates are not considered exhaustive;

-    the implementation of age gate mechanisms described in the previous paragraph, although not exhaustive.

In light of the aforementioned elements, assessed as a whole, in the absence of data relating to the total annual worldwide turnover of the previous financial year of the Company, it is deemed appropriate to determine, pursuant to Article 83, paragraph 3, of the Regulation, the total amount of the administrative pecuniary sanction in Euro 5,000,000.00 (five million), equal to half of the maximum fixed fine provided for by art. 83, par. 5, of the Regulation. This amount is determined in the following terms:

•    pursuant to art. 83, par. 3, of the Regulation, considering the unique nature of the conduct, since it concerns connected treatments for the reasons stated above, the amount of the pecuniary sanction for the most serious violation of art. 5, par. 1, letter a), 12 and 13, of the Regulation, is calculated in the amount of Euro 3,000,000.00;

•    the sanction is increased for the violation of art. 5, par. 1, letter a) and 6 of the Regulation in an amount equal to Euro 1,000,000.00;

•    the penalty is increased for the violation of art. 5, par. 1, letter c) and art. 25, par. 1, of the Regulation in an amount equal to €1,000,000.00;

This administrative pecuniary penalty is considered, pursuant to art. 83, par. 1, of the Regulation, effective, proportionate and dissuasive.

Taking into account the particular sensitivity of the data processed, it is believed that the accessory sanction of publication on the website of the Guarantor of this provision, provided for by art. 166, paragraph 7 of the Code and by art. 16 of the Guarantor Regulation no. 1/2019, should be applied; this in light of the nature and severity of the violations ascertained, particularly taking into account that these are large-scale processing involving a large number of interested parties, the risks in terms of protection of personal data connected to making available to the public a service based on an innovative and complex technology in the absence of the necessary safeguards. Furthermore, it is believed that there is a general interest with respect to the topic of generative artificial intelligence which requires the widest possible knowledge of the position of the Authority on the matter.

Finally, it is believed that the conditions set out in art. 17 of Regulation no. 1/2019 concerning internal procedures with external relevance, aimed at carrying out the tasks and exercising the powers delegated to the Guarantor, for the annotation of the violations detected here in the internal register of the Authority, provided for by art. 57, par. 1, letter u) of the Regulation.

GIVEN ALL THE ABOVE, THE GUARANTOR

pursuant to art. 57, par. 1, letter f), of the Regulation, declares the processing described in the terms of the motivation, carried out by Luka Inc., with registered office in 490 Post St Suite 526, San Francisco, California, United States of America, to be unlawful for the violation of articles 5, par. 1, letter a) (with reference to both the principle of lawfulness and transparency), 6, 12, 13, 5, par. 1, letter c), 24 and 25, par. 1, of the Regulation and, consequently,

a)    pursuant to art. 58, par. 2, letter d) of the Regulation, orders the Company, within thirty days of notification of the provision, to conform the processing to the provisions of the Regulation, in particular to conform the privacy policy to art. 5, par. 1, letter a), 12 and 13 of the Regulation as well as to conform the age verification system to art. 5, par. 1, letter c), 24 and 25 of the Regulation, remedying the gaps respectively indicated in paragraphs 5 and 6 of this provision;

b)    pursuant to art. 157 of the Code, orders the Company to communicate to the Authority, within sixty days of notification of this provision, the initiatives undertaken in order to implement the corrective measure referred to in the preceding point; any failure to comply with the provisions of this point may result in the application of the administrative pecuniary sanction provided for by art. 83, par. 5, of the Regulation

ORDERS

to Luka Inc., with registered office at 490 Post St Suite 526, San Francisco, California, United States of America, to pay the total sum of Euro 5,000,000.00 (five million) as an administrative pecuniary sanction for violations of art. 5, par. 1, letter a), 6; art. 5, par. 1, letter a), 12, 13, 5, par. 1, letter c), 24 and 25, par.1, of the Regulation, representing that the offender, pursuant to art. 166, paragraph 8, of the Code has the right to settle the dispute by paying, within sixty days, an amount equal to half of the fine imposed.

ORDERS

a) the aforementioned Company, in the event of failure to settle the dispute pursuant to art. 166, paragraph 8, of the Code, to pay the sum of Euro 5,000,000.00 (five million), according to the methods indicated in the attachment, within 60 days of notification of this provision, under penalty of adopting the consequent executive acts pursuant to art. 27 of Law no. 689/1981.

ORDERS

a) the publication of this provision, pursuant to art. 154-bis of the Code and 37 of Regulation no. 1/2019;

b) the application of the accessory sanction of the publication on the website of the Guarantor of this injunction order, as provided for by art. 166, paragraph 7 of the Code and 16 of the Guarantor Regulation no. 1/2019;

c) the annotation of this provision in the internal register of the Authority - provided for by art. 57, paragraph 1, letter u), of the Regulation, as well as by art. 17 of Regulation no. 1/2019 concerning internal procedures having external relevance, aimed at carrying out the tasks and exercising the powers delegated to the Guarantor - relating to the violations and measures adopted in accordance with art. 58, paragraph 2, of the Regulation itself.

The Authority reserves the right to examine and verify in an independent proceeding the profiles concerning the lawfulness of the processing carried out by Luka Inc., with specific reference to the legal bases of the processing of personal data relating to the entire life cycle of the generative artificial intelligence system underlying the Replika service.

Pursuant to art. 78 of the Regulation, as well as art. 152 of the Code and 10 of Legislative Decree no. 150 of 1 September 2011, an objection to this provision may be lodged with the ordinary judicial authority, with an appeal filed with the ordinary court of the place where the data controller is resident, within thirty days from the date of communication of the provision itself, or sixty days if the appellant resides abroad.

Rome, 10 April 2025

THE PRESIDENT
Stanzione

THE REPORTER
Scorza

THE ACTING SECRETARY GENERAL
Filippi








GUARANTEE FOR THE PROTECTION OF PERSONAL DATA

AT today's meeting, attended by President Pasquale Stanzione, Vice-President Ginevra Cerrina Feroni, Board Members Agostino Ghiglia and Guido Scorza, members, and Acting Secretary General Claudio Filippi;

HAVING REGARD TO Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation, hereinafter referred to as the ‘Regulation’);

HAVING REGARD TO the Personal Data Protection Code (Legislative Decree No. 196 of 30 June 2003), as amended by Legislative Decree No. 101 of 10 August 2018, laying down provisions for the adaptation of the national legal system to the aforementioned Regulation (hereinafter referred to as the 'Code');

HAVING REGARD TO Regulation No. 1/2019 concerning internal procedures with external relevance, aimed at fulfilling the tasks and exercising the powers assigned to the Guarantor for the protection of personal data (Italian Data Protection Authority), approved by Resolution No. 98 of 4 April 2019, published in the Official Journal No. 106 of 8 May 2019 and at www.gpdp. it, web doc. No. 9107633 (hereinafter 'Guarantor's Regulation No. 1/2019');

HAVING REGARD TO the documentation on record;

HAVING REGARD TO the observations made by the Secretary-General pursuant to Article 15 of Garante's Regulation No. 1/2000;

REPORTER Guido Scorza;    

1.    INTRODUCTION

The proceedings originated from an investigation initiated by the Guarantor of its own motion following the publication of press reports and preliminary fact-finding conducted on the Replika service (https://replika.com/), a chatbot with a written and voice interface developed and managed by the US company Luka Inc.  (hereinafter 'Luka' or the 'Company') and based on a generative AI system.

Replika is described as a chatbot that can improve the user's mood and emotional well-being by helping them understand their thoughts and feelings, track their mood, learn coping skills, reduce anxiety, and work towards goals such as positive thinking, stress management, socializing, and finding love. Replika creates a 'virtual companion' that the user can decide to set up as a friend, therapist, romantic partner or mentor. 

Replika uses a Large Language Model (LLM) system that is constantly fed and improved through interaction with users.

For the purposes of this decision, 'generative artificial intelligence' means the field of artificial intelligence that focuses on creating new and original content from input data in response to user requests (prompts), through the use of predominantly neural algorithms. ‘Neural network’ means a standard computational model applicable in a wide variety of contexts that allows the recognition of objects, shapes or patterns within a given data set or data set (e.g. a human face in a photograph). Generative artificial intelligence algorithms are used in a wide range of applications, including the recognition and generation of images, voice or music tracks, text and videos.

An example of generative artificial intelligence is large language models. For the purposes of this measure, 'Large Language Model' means a probabilistic model of a natural language, such as English or Italian, based on the assumption that all natural languages are highly redundant and correlated; this gives LLM the ability to identify the word or symbol that is most likely to follow a given piece of data.

In light of the above, the Guarantor launched an investigation on its own initiative, noting that Luka's processing of personal data in the context of the Replika service could give rise to an infringement of personal data protection legislation, with particular reference to: the privacy policy and the transparency obligations; the absence in the privacy policy of a specific indication of the legal basis for the processing in relation to the various processing operations carried out; the legal basis for the processing of minors' personal data, since in this case it must be excluded that it could be based on the performance of a contract; the absence of any filter to verify the age of users, both when accessing the service (by registering an account) and during interaction with the chatbot; the delivery, through the chatbot, of content that conflicts with the protections that should be ensured to minors and, more generally, to all vulnerable individuals.

In this context, on 2 February 2023, having found that the processing of personal data by Luka as regards the Replika service could give rise to an infringement of Articles 5, 6, 8, 9 and 25 of the Regulation and posed concrete risks to minors, also due to the fact that the responses provided were not in line with the enhanced safeguards to be ensured for minors and vulnerable individuals, the President of the Garante, pursuant to Article 5(8) of the Garante's Regulation No. 1/2000,  adopted an urgent measure (No. 39/2023, Reg. No. 18321/2023) against Luka to temporarily limit the processing of personal data of data subjects in Italy, pursuant to Article 58(2)(f) of the Regulation.

Subsequently, by decision No. 280 of 22 June 2023 (Reg. No. 104960/23), the Garante decided to suspend decision No. 39/2023 temporarily limiting the processing, provided that the controller, pursuant to Article 58(2)(d) of the Regulation, adopted appropriate measures to ensure that the processing of personal data within the Replika service was carried out in accordance with the legislation on the protection of personal data. In particular, the Garante ordered the data controller to:

1.    present an updated privacy policy to all users in Italy before registration and before accessing the Replika service;

2.    implement an age gate mechanism on all service registration pages;

3.    implement a ‘cooling-off period’ to prevent minors from entering a different date of birth when they are denied access to the services;

4.    make it possible for users in Italy to easily and effectively exercise their rights regarding personal data protection, including the right to object to the processing of personal data and to request access, rectification and erasure of data;

5.    submit to the Garante, fifteen days before the date scheduled for the opening of the service to Italian users, a plan for the development of a process aimed at preventing access to the service by persons under the age of 18, possibly supported by a language analysis mechanism with subsequent blocking effect;

6.    submit to the Garante, fifteen days before the service being available again to Italian users, a plan for the implementation of functions that allow users to report inappropriate content to prevent the Replika chatbot from providing such content, such as the possibility of flagging specific responses as inappropriate and providing feedback on the user's experience during the session.

The Garante indicated specific deadlines for the implementation of the above requirements, establishing that those referred to in points 1 to 4 had to be fully complied with by 28 July 2023, and that those referred to in points 5 and 6 had to be implemented within fifteen days of the date of the service being available again to Italian users.

2.    LUKA'S REPLIES TO DECISIONS NO. 39/2023 AND NO. 280/2023 

In a letter dated 3 March 2023 (Reg. No. 38795/23), the Company replied that it had promptly taken steps to comply with the Garante's requests, in particular to comply with the request for temporary limitation of processing for users located in Italy, by immediately blocking access to the Replika service from Italy, both through the app and through its website.

Luka also stated that it had launched a series of initiatives aimed at implementing the Garante's requests in a concrete manner, including through the involvement of external consultants and experts in the field. In particular, the Company stated that it had initiated a number of assessments, actions and processes intended to:

-    implement more robust user age verification mechanisms to better ensure that minors in Italy do not use the ‘Replika’ service, which is reserved for adults; in addition to the age gate tools already in use, the Company undertook to introduce automated measures aimed at recognising underage users based on the analysis of indicators contained in conversations with the chatbot;

-    implement algorithms and processes for moderating inappropriate content, in line with best practice;

-    ensure compliance with the Regulation by, among other things, updating the register of processing activities, reviewing and updating data protection impact assessments (DPIAs), and updating the privacy policy relating to the service, in order to increase transparency for users.

By letter dated 31 March 2023 (Reg. No. 55533/23), the Company requested the corrective measure of the temporary limitation imposed by urgent measure No. 39/2023 be lifted, specifying:

-    that the Replika service was designed to limit the extent of personal data processing, in accordance with the principles set out in Article 5 of the Regulation, including i) minimising the collection of user registration data (name, email address, date of birth – to verify age – and any third-party login data); ii) the adoption of data retention and erasure procedures that strike a balance between the need to provide the user with a smooth experience and the need to minimise the personal information that remains accessible; iii) the design of proprietary artificial intelligence (AI) models to interact with users; iv) the non-sharing of user conversations with third parties other than the Company's essential service providers, who are bound by confidentiality obligations; v) the implementation of strict controls designed to limit access to personal data by its own staff; vi) the non-use of user conversations for advertising or marketing purposes; 

-    that it does not offer the service to minors and to base the processing of users' personal data on the legal basis of contract performance; 

-    that it implemented, following the provision of the Garante, several measures aimed at preventing minors from accessing the service in violation of the Company's terms;

-    that it included its mobile application in the Apple App Store with an age rating of 17 or older, which is the highest age rating allowed by Apple;

-    that it does not collect special categories of personal data, given that the sharing of special categories of personal data by users during their interaction with the chatbot is spontaneous and must therefore be qualified as covered by explicit consent to processing, in accordance with Article 9 of the Regulation;

-    that it takes its data protection obligations seriously and has integrated data protection into the design of the service, in accordance with Article 25 of the Regulation, and that it will continue to ‘develop and improve its policies and procedures to provide users with a consistent, secure and rewarding experience’. 

With specific reference to the Garante's decision, the Company stated:

-    that it promptly blocked access to the Replika service to individuals located in Italy;

-    that it strengthened measures to prevent access to the service by minors under the age of 18, in particular by:  i) introducing an age gate on all service registration pages requiring users to indicate a date of birth greater than or equal to 18 years of age in order to access the service; ii) providing for a ‘cooling-off period’, in line with the guidelines of data protection authorities and best practices, to prevent minors from entering a different date of birth when the system denies access to the service; iii) launching activities aimed at improving automated content control processes (reporting individuals likely to be under 18 and preventing use of the service until age verification is completed through more robust methods);

-    that it updated its privacy policy to address the transparency issues identified by the Garante;

-    that it continues to develop and fine-tune its content moderation practices to prevent harm to users, in particular by creating a trust and safety programme to prevent the chatbot from being involved in offensive or harmful conversations;

-    that it restricted access to conversations of a sexual nature or relating to other adult content to users active at 1 February 2023 and made such conversations unavailable to new users;

-    that it continues to endeavour to ensure compliance with the Regulation through the support of an external data protection consultant. The commitments undertaken by the Company include: i) updating and maintaining the Company's record of processing activities; ii) reviewing and updating data protection impact assessments (DPIAs), including documentation of data protection by design and by default processes; iii) refining and reviewing the Company's security policies and procedures; iv) reviewing the Company's data protection governance (including the possibility of appointing a DPO following the expansion of the Company's activities in the European Union).

The Company, by letter dated 26 April 2023 (Reg. No. 68896/23), submitted a second request for lifting the corrective measure of the temporary limitation imposed by urgent measure No. 39/2023, reiterating the measures taken, as already explained in the previous letter.

The Company, in a letter dated 14 June 2023 (Reg. No. 93675/23), further to the discussions held at the hearing on 31 May 2023, reaffirmed that it had responded promptly to Decision No. 39/2023, immediately blocking access to Replika in Italy and implementing adequate measures in response to the issues raised by the Garante in the aforementioned measure. The Company also expressed its commitment to prevent users located in Italy from engaging in conversations of a sexual nature by providing, once the service is reactivated, two versions of Replika: a free version and a paid version containing romantic but not sexual content. According to the Company, the introduction of a paid ‘romantic’ version will require additional age verification based on the user's payment card details, in line with the latest market standards for age verification mechanisms.

In a letter dated 14 July 2023 (Reg. No. 109176/23), the Company announced that it had complied with the requests set out in points 1-6 of decision No. 280/23 and, in particular, stated:

1.    in relation to the information referred to in point 1 of Decision No. 280/23, that it implemented an updated privacy policy in the registration process and prior to access to the service, and that this information would be displayed to Italian users upon reactivation of the service;

2.    with reference to the age gate mechanism referred to in point 2 of Decision No. 280/23, that it implemented an age verification system on all registration pages and that this system would be applied upon reactivation of the service;

3.    with reference to the cooling-off period referred to in point 3 of Decision No. 280/23, that it implemented a cooling-off period to prevent minors from attempting to access the service again by entering a different date of birth. This period—lasting 24 hours—is to be managed (i) by checking the credentials of a minor user's account and subsequently preventing them from entering a different date of birth and (ii) by installing a cookie to prevent minor users from re-entering a different date of birth from the same browser. The Company stated that this cooling-off period would be applied upon reactivation of the service;

4.    with regard to the exercise of the rights referred to in point 4 of Decision No. 280/23, that it provides users with a simple and effective method for exercising their data protection rights, including the right to object to the processing of their personal data and the rights to request access, rectification and erasure of their data, and that this mechanism would be applied upon reactivation of the service;

5.    with regard to the request to prepare a plan for the development of an age verification mechanism during registration referred to in point 5 of Decision No. 280/23, that it implemented processes to prevent access by minors under the age of 18, including a language analysis mechanism that requires users to reconfirm their age through the age gate process when users identify themselves as under the age of 18. If no date of birth that satisfies the age gate is provided, the user cannot access the service The Company stated that such processes would be applied in Italy upon reactivation of the service;

6.    with regard to the request to prepare a plan for the development of an age verification mechanism during the use of the service referred to in point 6 of Decision No. 280/23, the Company stated that it implemented features that allow users to report inappropriate content to prevent the Replika chatbot from presenting it again, such as the ability to flag specific responses as inappropriate and provide feedback on the user experience during the session. The Company stated that these features would be implemented in Italy upon reactivation of the service.

Luka submitted, together with the letter dated 14 July 2023, a copy of the privacy policy updated on 12 June 2023.

3. FACT-FINDING ACTIVITY  

Parallel to adoption of the precautionary measure, the Garante started gathering the info deemed necessary to carry out the fact-finding activity by sending a request for information, in line with Articles 58(1)(e) of the Regulation and 157 of the Code.

By letter dated 6 April 2023 (Reg. No. 58925/23), the Garante sent a request for information to Luka asking for details on how Replika works (categories of personal data processed and source from which they are collected; method used for collection; how the data collected is processed; where the data is stored; security measures adopted; processing of user data for system training purposes or for other purposes pursued by Luka), the processing of users' personal data (legal basis; storage period; minimum age for accessing the service provided by Replika; DPIA; appointment of a representative pursuant to Article 27 of the Regulation; procedures for managing rights pursuant to Articles 12–22 of the Regulation; legal basis and guarantees of adequacy pursuant to Chapter V of the Regulation, where applicable; clarifications regarding automated processing pursuant to Article 22 of the Regulation), and age verification measures for access to the service on the date of notification of urgent measure No. 39/23.

With regard to this request, in a letter dated 8 May 2023 (Reg. No. 74173/23), the Company, after initially claiming that it has a single establishment in the European Union in the Netherlands, stated that:

-    it uses the messages and content that the user sends to the chatbot to enable conversations with that user (the ‘Chatbot Interaction’). In relation to Chatbot Interaction, the content of the database may include basic profile information, conversation topics, questions that the user may ask, and selected preferences or interests. When a user sends a message, the model analyses the text to enable the chatbot to generate a response based on the latest messages in the conversation. The Company has also made it clear that it uses a database containing all the info sent through the chat to create de-identified data and fine-tune the LLM that forms the basis of the chatbot (‘Model Development’). The section of the database used as a source to create de-identified data is limited to: 1) user ‘Reactions’ (‘like’, ‘dislike’, ‘love’, ‘funny’, ‘meaningless’ or ‘offensive’), if the user chooses to make such a selection; 2) ‘Feedback’ on user satisfaction levels with the conversation (‘happy’, ‘neutral’ or ‘sad’); 3) ‘Snippets’, i.e. small parts of user conversations that provide context for interpreting Reactions and Feedback. The information used by the Company for Model Development does not identify specific individuals and cannot be associated with specific individuals (‘De-identified Data’) as any personal identifiers (such as names, addresses, email addresses, telephone numbers and identification numbers) that may be contained in conversation snippets are removed and the snippets are ‘shuffled’ in a randomised fashion;

-    it collects all personal data described above from users‘ interactions with the service;

-    it uses a system for collecting (‘Reactions’, ‘Feedback’ and ‘Snippets’) and processing in real time users’ interactions with the chatbot using webhooks, i.e. automated tools that capture such information and send it to the Company's servers;

-    it follows, in the processing of ‘De-identified Data’ for Model Development, the following steps: 1) data collection, as described above; 2) pre-processing consisting of cleaning, structuring and removing any personal identification data from such data, in order to safeguard privacy (through aggregation and randomisation techniques); 3) labelling of pre-processed data; 4) analysis and development to evaluate the performance of the LLM, identify patterns and develop filters that prevent the model from producing outputs with inappropriate content; 5) testing and validation (regular testing and validation against predefined criteria);

-    it stores personal data on encrypted databases hosted by Amazon Web Services, Inc. in the United States;

-    it does not use personal data provided by users for Model Development;

-    it employs technical and organisational measures to protect the security of personal data and ‘De-identified Data’ from unauthorised access, use and disclosure. These measures include encryption, access controls, vulnerability management, pre-processing and anonymisation of ‘Snippets’, ‘Reactions’ and ‘Feedback’, training and possible disciplinary measures in the event of non-compliance with the measures by the Company's personnel;

-    it relies on the contractual legal basis for ‘Chatbot Interaction’ as the processing of user data is necessary for the provision of the service, in accordance with the Terms of Service. This processing includes the creation and maintenance of user account profiles, the facilitation of payments and transactions, and the processing of data entered by users to generate the chatbot's response;

-    it relies on the legal basis of legitimate interest for ‘Model Development’;

-    it retains data for ‘as long as it deems reasonably necessary to provide users with a safe, enjoyable and successful experience on the platform’, in accordance with the principle of minimisation;

-    it retains ‘Chatbot Interaction’ data for ‘a period sufficient to enable the retrieval of information to ensure a seamless conversation experience for users with the chatbot, in line with user expectations’;

-    it retains [without further specification, editor's note] user data to create ‘De-identified Data’ for ‘Model Development’;

-    that the minimum age required to use the Replika service is 18 years;

-    that there is no contradiction between the previous point and the statement contained in the Company's privacy policy, which reads: ‘We do not knowingly collect Personal Data from children under the age of 13. If you are under 13, please do not submit any Personal Data through the Services’, as this statement has been included as required by US federal law (COPPA);

-    the Replika mobile application included an age gate that prevented minors under the age of 18 from accessing the service even before the Garante's Decision of 2 February 2023. The Company has also listed its application in the Apple App Store with an age rating of 17+, which is the highest age rating allowed by Apple;

-    all adult content has been placed behind a paywall, out of reach of minors;

-    following the Decision of 2 February 2023, the Company deliberately improved the measures aimed at preventing minors under the age of 18 from accessing the service;

-    it has not designated a representative pursuant to Article 27 of the Regulation as the Company has an establishment in the European Union;

-    with regard to the exercise of the rights of data subjects, the relevant information is provided through a privacy policy published on the Company's website and in the App. Access, rectification and erasure may be requested by users, who may also object to and restrict the processing of any personal data that is not necessary for the provision of the service. Requests are evaluated on a case-by-case basis;

-    it does not engage in any profiling of data subjects or take automated decisions that have legal or equally significant effects;

-    it collects personal data directly from users and does not transfer them from Italy or the European Union in accordance with Chapter V of the Regulation and has entered into data processing agreements with data processors, which include standard contractual clauses where required;

-    for the purpose of content control, it has trained its models to prevent the emergence and escalation of inappropriate content or inappropriate responses. As part of this process, the Company uses open-source data sets specifically designed and made available to the artificial intelligence research community for the purpose of improving the safety and robustness of machine learning models. The Company has also developed, and continues to improve and refine, filters that recognise keywords, phrases and patterns associated with harmful behaviour, such as self-harm, insults or murder. The filters trigger the LLM to respond appropriately to such content, for example by changing the topic of the conversation or providing users with self-help resources. The Company also uses human review in both the evaluation of the AI model and the development of filters;

-    it uses other methods to control content that is inappropriate or conflicts with the app's Terms of Service, including: 1) placing so-called romantic content behind a paywall and disabling sexually explicit content for new users; 2) allowing users to report specific content or conversations as offensive in real time and using such reports to improve the models and prevent them from developing similar content in the future; 3) prohibiting users, in the Terms of Service, from uploading illegal, harmful and threatening content.

Along with its reply of 8 May, the Company provided a copy of the privacy policy applicable on 2 February 2023, its updated version dated 22 March 2023, and a copy of the impact assessment (undated and unsigned).

By letter dated 27 February 2024 (Reg. No. 23744/24), the Garante informed the Company of the initiation of proceedings for the adoption of corrective measures and sanctions pursuant to Article 166(5) of the Code and Article 12 of the Garante's Regulation No. 1/2019, alleging that Luka had infringed Articles 5, 6, 7, 8, 12, 13, 24 and 25(1) of the Regulation in relation to the processing of personal data carried out by the Company through the Replika service as at 2 February 2023.

The Company did not reply to the notice of initiation of proceedings nor did it request to be heard pursuant to Article 166(6) of the Code and Article 13 of the Garante's Regulation No. 1/2019. 

In the notice of initiation of proceedings, which is hereby expressly and fully referred to, the Garante alleged three infringements against the Company on the basis of the critical issues identified in urgent measure No. 39/2023. The evaluation conducted by the Garante focused on the facts, the processing operations and the measures implemented by Luka as at 2 February 2023.

With regard to the failure to identify the conditions governing the lawfulness of the processing, the Garante found that the privacy policy that was online at the time of the adoption of the urgent measure—updated on 5 July 2022—did not provide a granular description of the legal basis for the various processing operations carried out by the Company in connection with the Replika service. The reference to the legal bases for the performance of a contract (Article 6(1)(b) of the Regulation) and the consent of the data subjects (Article 6(1)(a) of the Regulation), as well as to a generic authorisation (‘authorisation’, not obligation) under the law, were not in fact linked or attributable to specific processing operations (so-called granularity), making it impossible to identify and evaluate the suitability of those legal bases. Furthermore, the privacy policy dated 5 July 2022, effective on 2 February 2023, did not contain any reference to the legal basis underlying the processing of personal data for the development of the LLM that powers the chatbot, nor did the documentation subsequently produced—notably the privacy policy—even in the version updated on 22 March 2023, and the DPIA, include elements demonstrating that the Company had identified a legal basis for this purpose prior to 2 February 2023.

In light of the above, the Garante alleged that Luka had possibly infringed Article 5(1)(a) and Article 6 of the Regulation by failing to identify, as at 2 February 2023, the legal bases for the various processing operations carried out through the Replika service.

With regard to transparency obligations, the Garante's evaluation concerned the privacy policy applicable as at 2 February 2023, i.e. the version updated on 5 July 2022. From a formal point of view, the Garante, in the act initiating the proceedings, found that as at 2 February 2023 the privacy policy was only available in English (including for minors) and was not easily accessible.  From a content point of view, it was found that as at 2 February 2023, the privacy policy:

-    did not indicate the legal basis for each processing activity and type of data processed;

-    did not indicate the purposes of the processing with specific reference to the two distinct types of processing, namely processing for ‘Chatbot Interaction’ and processing for ‘Model Development’; 

-    in the sections ‘People mentioned in the chat’ and ‘Integration with your Instagram account’, two categories of personal data processed for the purpose of enabling user conversations were indicated; 

-    did not clarify that the service was offered exclusively to adults, since, as mentioned above, the privacy policy only included a reference to minors under the age of 13 in compliance with the requirements of COPPA (Children's Online Privacy Protection Act);

-    did not provide any specific information regarding the storage period of personal data or the criteria used to determine such period;

-    did not clarify whether personal data were transferred outside the EEA and, in such case, what the legal basis and guarantees of adequacy referred to in Chapter V of the Regulation were. In particular, the text of the privacy policy (see, in particular, the statement ‘By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen’) is in clear contradiction with the statement made by the same Company in its letter dated 8 May 2023 (Reg. No. 74173/23), where it is stated that, since the criterion of establishment in the European Union does not apply, no transfer of personal data from the European Union (in particular, from Italy) to the US is possible under Chapter V of the Regulation;

-    in section 6 entitled ‘Your data protection rights’, the privacy policy provided specific information on the right set out in Article 22 of the Regulation, even though the provision was not expressly referred to. This reference (no longer present in the version dated 22 March 2023) was sufficient to lead users to believe that their personal data were subject to automated decision-making in violation of the principles of transparency and fairness. This circumstance was denied by the same data controller in its reply (Reg. No. 74173/23), in which it argued that ‘although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling that have legal or similar effects within the meaning of Article 22 of the Regulation’.

In light of the above, the Garante alleged that Luka had possibly infringed Article 5(1)(a), 6, 12 and 13 of the Regulation, given that, as at 2 February 2023, the privacy policy relating to the Replika service did not comply with the general obligations and principles of transparency and was provided in such a way and at such a time that it could not be readily accessed by users.
Lastly, with regard to the lack of mechanisms for age verification of minors, the Garante alleged the failure to implement measures ensuring specific protection for minors in relation to access to and use of the Replika service as at 2 February 2023. In particular, the following were found to be lacking:

-    a user age verification procedure (the system only required name, email address and gender), with the consequent risk of minors being presented with responses that were unsuitable for their level of development and self-awareness, including sexually explicit content;

-    mechanisms to prohibit or block access even when the user clearly stated that they were a minor; in addition, the chatbot provided responses that were clearly contrary to the protections that should be ensured for minors and, more generally, for all vulnerable individuals.

The Garante, when initiating the proceedings, acknowledged that the Company had implemented age verification mechanisms following the request made by the Garante in the temporary limitation decision adopted as a matter of urgency on 2 February 2023. In particular, during the exchanges that followed the adoption of the aforementioned decision and with specific reference to age verification, the data controller stated that it had implemented an age gate on all registration pages for the Services aimed at restricting access to users who are at least 18 years of age and that the age verification mechanism includes a ‘cooling-off period’ aimed at preventing users—once they have ascertained that it is impossible to access the service by entering their real personal data—from immediately entering a different date of birth that would allow them to access the service. The Company also stated that a process was being developed to use language analysis to identify and prevent the use of the Services by persons under the age of 18.

In light of the above, the Garante alleged that Luka had possibly infringed Article 5(1)(c), 6; 7; 8; 24 and 25(1) of the Regulation for failing to put in place appropriate systems to verify the age of individuals as at 2 February 2023.

4. EXISTENCE OF EU JURISDICTION AND COMPETENCE OF THE GARANTE

As a preliminary observation, the Garante considers it appropriate to address the issues relating to the applicability of European data protection legislation to the service offered by Luka and to its own competence, also taking into account the objections raised by the Company in its reply dated 8 May 2023 to the request for information sent by the Garante.

Article 3 of the Regulation governs the territorial scope of application of the legislation, establishing different criteria depending on whether or not the data controller is established in the European Union. 

In the first case (Article 3(1), known as the establishment criterion), the Regulation applies regardless of whether the processing is carried out in the Union or not, and competence is determined in accordance with the one-stop-shop mechanism, pursuant to Article 56 of the Regulation.

In the second case (Article 3(2), known as the targeting criterion), the Regulation applies to the processing of personal data of data subjects who are in the Union insofar as the processing activities relate to: i) the provision of goods or services to data subjects in the Union (Article 3(2)(a) of the Regulation); ii) the monitoring of the behaviour of data subjects located in the Union insofar as such behaviour takes place in the same Union (Article 3(2)(b) of the Regulation).

The Company stated in the above-mentioned letter that it has a single establishment in the European Union in the Netherlands, reporting that it has ‘a group of employees located in the Netherlands, including a number of decision-makers involved in cross-border data processing for the development of LLM and of the product’ and that ‘the Company's employees located in the Netherlands are involved in decisions concerning the processing of personal data by the Company and the operation of LLM globally, including decisions concerning the minimum portion of users located in Italy’. The existence of a Dutch establishment in the European Union would entail the application of the one-stop-shop mechanism and the competence of the Dutch data protection authority as lead supervisory authority in cooperation with the authorities concerned.

However, this statement is not supported by any evidence. In fact, both in the privacy policy published on Replika's website as at 2 February 2023 (version updated on 5 July 2022) and in subsequent versions thereof (including the current version updated on 23 February 2024), there is no mention of an establishment of the company in the Netherlands; likewise, no mention can be found in the Terms of Service (neither in the version updated on 14 September 2022 nor in the current version, updated on 7 February 2023), which, on the contrary, states that Luka is ‘a software company who designed and built Replika, incorporated in Delaware, and operating in San Francisco, CA’.

Furthermore, the statements in the letter of 8 May 2023 are extremely vague, as they do not even indicate the name and registered office of the company allegedly established in the European Union (thus making it impossible to carry out any checks in cooperation with the Dutch supervisory authority pursuant to Article 61 of the Regulation) and are not supported by any document evidence (e.g. the Dutch company's articles of association or chamber of commerce registration).

As things stand, therefore, the Garante considers that no evidence has been provided to effectively demonstrate the existence of an establishment of the Company in the European Union and, as a result, the applicability of the establishment criterion under Article 3(1) of the Regulation and of the one-stop-shop mechanism with the competence of the Dutch data protection authority.

In the present case, the existence of EU jurisdiction and the competence of the Garante must be ascertained on the basis of the targeting criterion set out in Article 3(2), of the Regulation: more specifically, it must therefore be ascertained, as a preliminary matter, whether the Replika service can be considered as offered to data subjects located in the European Union for the purposes of the applicability of point (a) of the aforementioned Article 3 of the Regulation.

In this regard, reference is made to the ‘Guidelines 3/2018 on territorial scope’, adopted by the European Data Protection Board (EDPB) on 12 November 2019, which provide that the ‘controller [...] demonstrates its intention to offer goods or services to a data subject located in the Union’ (see paragraph 2(a) of the aforementioned Guidelines) and the case law of the Court of Justice of the European Union (judgment Pammer/Reederei Karl Schlüter GmbH & Co and Hotel Alpenhof/Heller – joined cases C-585/08 and C-144/09), which may be taken into account in order to determine whether a commercial activity carried out by an entity is directed to a Member State, among which the fact that the European Union is mentioned in connection with the goods or services offered, the international nature of the activity or the launch of advertising and marketing campaigns aimed at the public of an EU country.

In the present case, the evidence that the Replika service was offered to data subjects located in the European Union and, in particular, in Italy as at 2 February 2023, is clear from the Company's initial reply to the order for temporary limitation issued by the Garante in its urgent measure  No. 39/2023, which states (see letter dated 3 March 2023, p. 1) that ‘the Company promptly complied with the request for temporary limitation of processing for users established in Italy, immediately blocking access to both the app and the service website from Italy’.

Having demonstrated the territorial applicability of the Regulation and the competence of the Garante in the manner and within the terms set out above, the following remarks are made.

The processing of personal data carried out by Luka qualifies as cross-border processing of personal data within the meaning of Article 4(1)(23) of the Regulation, as it is likely to affect data subjects in more than one Member State.

For this type of processing, where the controller has identified a single or main establishment in the European Union, as already explained, the cooperation mechanism described in Articles 60 et seq. of the Regulation applies, and the competence to exercise the tasks and powers referred to in Articles 57 and 58 of the Regulation lies, pursuant to Article 56(1) of the Regulation, with the lead supervisory authority, i.e. the supervisory authority of the Member State in which the single or main establishment is located.

If, on the contrary, as in the present case, the data controller does not have an establishment in the European territory, the data controller shall ‘liaise with the supervisory authorities of each Member State in which it operates through the designated representative’ (see paragraph 3.3. of the ‘Guidelines on the Lead Supervisory Authority’ adopted by the Article 29 Working Party on 13 December 2016, revised on 5 April 2017 and endorsed by the EDPB on 25 May 2018).

In fact, where a controller does not have an establishment in the European Union (or, more precisely, in the EEA), the special rule in Article 56 does not apply and the general rule set out in Article 55(1) of the Regulation applies, according to which ‘each supervisory authority shall be competent for the performance of the tasks assigned to and the exercise of the powers conferred on it in accordance with this Regulation on the territory of its own Member State’.

In the present case, as mentioned above, Luka is a company based in the United States of America and has not demonstrated that it has an establishment in the territory of the European Union. Therefore, the Garante (Italian Data Protection Authority) is competent to evaluate, as regards its own territory, the compliance with the Regulation of the processing of personal data carried out by the Company and to exercise the powers conferred on it by Article 58 of the Regulation.

5. FINDINGS OF INFRINGEMENT

5.1 ARTICLES 5(1)(A) AND 6 OF THE REGULATION

The Garante notified Luka of the infringement of Articles 5(1)(a) and 6 of the Regulation for failing, as at the date of 2 February 2023, to identify the legal bases for the different processing operations carried out through the Replika service which was provided and made available to the public in Italy on that date.

Article 5(1) of the Regulation provides that ‘Personal data shall be: a)processed lawfully, fairly and in a transparent manner in relation to the data subject (‘lawfulness, fairness and transparency’); b) collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes; further processing for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes shall, in accordance with Article 89(1), not be considered to be incompatible with the initial purposes (‘purpose limitation’); c) adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’); d) accurate and, where necessary, kept up to date; every reasonable step must be taken to ensure that personal data that are inaccurate, having regard to the purposes for which they are processed, are erased or rectified without delay (‘accuracy’); e) kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed; personal data may be stored for longer periods insofar as the personal data will be processed solely for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes in accordance with Article 89(1) subject to implementation of the appropriate technical and organisational measures required by this Regulation in order to safeguard the rights and freedoms of the data subject (‘storage limitation); f) processed in a manner that ensures appropriate security of the personal data, including protection against unauthorised or unlawful processing and against accidental loss, destruction or damage, using appropriate technical or organisational measures (‘integrity and confidentiality’)’. Paragraph 2 of the same Article provides that ‘The controller shall be responsible for, and be able to demonstrate compliance with, paragraph 1 (‘accountability’)’.

Recital 39 clarifies that ’Any processing of personal data should be lawful and fair. It should be transparent to natural persons that personal data concerning them are collected, used, consulted or otherwise processed and to what extent the personal data are or will be processed. The principle of transparency requires that any information and communication relating to the processing of those personal data be easily accessible and easy to understand, and that clear and plain language be used. That principle concerns, in particular, information to the data subjects on the identity of the controller and the purposes of the processing and further information to ensure fair and transparent processing in respect of the natural persons concerned and their right to obtain confirmation and communication of personal data concerning them which are being processed’.

Article 6 of the Regulation sets out the conditions for lawful processing by listing six possible legal bases (consent, contract, legal obligation, vital interest, public interest and legitimate interest) on which the data controller must rely to lawfully process personal data necessary for carrying out its activities. As clarified by the EDPB ‘The legal basis must be identified at the outset of processing, and information given to data subjects in line with Articles 13 and 14 must specify the legal basis. (see Guidelines 2/2019 on the processing of personal data under Article 6(1)(b) GDPR in the context of the provision of online services to data subjects)’.

The Company did not submit any defence statements or documents, pursuant to Article 166(5) of the Code, following the Garante’s notice of infringement and initiation of proceedings, and thus did not provide any counter-arguments regarding the alleged infringement related to the failure to indicate the legal basis for each of the processing activities carried out by Luka within the scope of the Replika service.

In the present case, the documentation reviewed during the investigation—particularly the text of the privacy policy published on the date of the adoption of the Garante's urgent measure, as last updated on 5 July 2022—shows that the Company failed to identify, in a granular manner, the legal basis for the different processing operations carried out by the Company within the Replika service, including the processing of data used for the development of the LLM.

The only references provided in the introductory section of the privacy policy are as follows:
‘We care about the protection and confidentiality of your data. We therefore only process your data to the extent that:

•    It is necessary to provide the Replika services you are requesting,

•    You have given your consent to the processing, or

•    We are otherwise authorized to do so under the data protection laws’.

The legal bases referred to therein—namely, the performance of a contract (Art. 6(1)(b) of the Regulation), the consent of the data subjects (Art. 6(1)(a) of the Regulation), and a legal authorisation (although the Regulation in fact requires a legal obligation, not merely an authorisation, as a legal basis)—are cited only implicitly and generically. They are not referred to specific processing operations (the so-called principle of granularity), thereby making it impossible to identify and assess their appropriateness.

Finally, neither the privacy policy nor the documents on file contain any reference to the legal basis for the processing of personal data for the purpose of developing the LLM that powered the chatbot until 2 February 2023.

Specifically, while the evidence provided by Luka is relevant, it is not conclusive. In particular, the DPIA and the privacy policy submitted on 8 May 2023 do not overcome the concerns raised by the Garante regarding the principle of lawfulness and the identification of a valid legal basis for the processing, as required respectively under Articles 5(1)(a) and 6 of the Regulation, since:

-    The privacy policy, including the later version updated on 22 March 2023, does not explicitly mention the purpose of ‘Model Development’ nor its legal basis in the table set out in paragraph 2;

-    The DPIA, while distinguishing between the two processing purposes of ‘Chatbot Interaction’ and ‘Model Development’ (par. I) and analysing their respective legal bases (par. II), does not provide a clearly identified date and therefore does not demonstrate that the identification of the lawfulness conditions under Article 6 of the Regulation occurred prior to 2 February 2023. Moreover, the DPIA refers to legitimate interest as the legal basis for processing for the purposes of ‘Model Development’ without providing any arguments relating to the so-called ‘three-step test’ required in the legitimate interest assessment. Finally, it is pointed out that the DPIA, while being an excellent accountability tool, is not the document chosen by regulators for informing data subjects about processing activities; such information must instead be provided in the privacy policy.

With reference to Article 5(1)(a) of the Regulation, attention is drawn to the principle expressed by the EDPB in its binding decision 1/2021 on transparency—also applicable to the principle of lawfulness—according to which the principles set out in Article 5 of the Regulation must be considered as a general concept which is then implemented in various provisions and specific obligations (in the case of lawfulness, in Articles 6, 7, 8, 9 and 10 of the Regulation). Therefore, according to the EDPB, it is necessary to distinguish the specific obligations arising from a principle (in this case, Article 6 of the Regulation) from the principle itself as set out in Article 5 of the Regulation, since the principle cannot be circumscribed to the specific obligation, although the specific obligation is a concretisation of the principle.

The principle of lawfulness is indeed an overarching principle that reinforces other principles (such as fairness and accountability). This is confirmed by Article 83(5) of the Regulation which allows for separate sanctions for infringing the lawfulness obligations independently of any breach of the principle itself. In this specific case, the Garante considers that there has also been an infringement of the principle of lawfulness referred to in Article 5(1)(a) of the Regulation, taking into account the gravity (lack of a clear and granular identification of the legal bases underpinning the different processing operations), the nature (this is an essential element of data processing) and the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the single specific infringement of the obligation referred to in Article 6 of the Regulation.

Based on the foregoing, the Garante considers that Luka failed, as at 2 February 2023, to identify the legal bases applicable to the different processing operations carried out through the Replika service, provided and made available to the public in Italy on that date, in breach of Articles 5(1)(a) and 6 of the Regulation.

With regard to the substantive analysis and evaluation of the legal bases under Article 6(1)(b) and (f) of the Regulation—allegedly relied upon for the use of the chatbot and the post-training of the LLM underlying the Replika service—and, more generally, the legal bases applicable throughout the entire lifecycle of the generative AI system developed by the Company, the Garante reserves the right to initiate a separate and autonomous investigation.

5.2 ARTICLES 5(1)(A), 12 AND 13 OF THE REGULATION

The Garante notified Luka of the infringement of Articles 5(1)(a), 12 and 13 of the Regulation for having provided, as at the date of 2 February 2023, a privacy policy concerning the Replika service which did not comply with the obligations and general principles on transparency established under the Regulation. 

Article 5(1)(a) of the Regulation requires that personal data be processed lawfully, fairly, and in a transparent manner in relation to the data subject (principles of lawfulness, fairness and transparency).

Article 12 of the Regulation lays down rules regarding transparency and the modalities for the exercise of rights, while Article 13 specifies the information that a data controller must provide when personal data are collected from the data subject.

On the subject of transparency, Recital 58 of the Regulation requires that any information addressed to the public or the data subject be concise, easily accessible and easy to understand, and that clear and plain language be used, and, with reference to the specific protection of children, it provides that ‘where processing is addressed to a child, should be in such a clear and plain language that the child can easily understand’.

On the issue of transparency, the Committee's guidance is also relevant, particularly Guidelines 2/2019 on the processing of personal data under Article 6(1)(b) of the GDPR in the context of the provision of online services to data subjects, where it is provided that the legal basis for the processing must not only be identified at the outset of processing, but also explicitly specified in the ‘information given to data subjects in line with Articles 13 and 14’; the Committee's Guidelines 1/2022 on data subject rights – Right of access, are also applicable. Paragraph 142 of these Guidelines affirms that ‘a controller that offers a service in a country should also offer answers in the language that is understood by the data subjects in that country’.

Finally, the Guidelines adopted by the Article 29 Working Party on 11 April 2018, clarified that ‘The concept of transparency in the GDPR is user-centric rather than legalistic and is realised by way of specific practical requirements on data controllers and processors in a number of articles. The practical (information) requirements are outlined in Articles 12 - 14 of the GDPR. (…) The transparency requirements in the GDPR apply irrespective of the legal basis for processing and throughout the life cycle of processing. This is clear from Article 12 which provides that transparency applies at the following stages of the data processing cycle:  i) before or at the start of the data processing cycle, i.e. when the personal data is being collected either from the data subject or otherwise obtained; ii) throughout the whole processing period, i.e. when communicating with data subjects about their rights; and iii) at specific points while processing is ongoing, for example when data breaches occur or in the case of material changes to the processing’.

The Company did not submit any defence statements or documents, pursuant to Article 166(5) of the Code, following the Garante’s notice of infringement and initiation of proceedings, and thus did not provide any counter-arguments in relation to the alleged infringement of the transparency obligations and general principles of transparency as required by the Regulation.

The Garante's investigation, as already noted above, focused on the privacy policy adopted and published by Luka on 2 February 2023, that is to say the version updated on 5 July 2022.
First of all, from a formal point of view, the investigation established that, as at the date of 2 February 2023, the privacy policy was available only in English, not considering the language of the country in which the service was offered, namely Italian. 

From a substantive point of view, it is noted that as at the date of 2 February 2023, the privacy policy failed to comply with the principles of fairness and transparency as it was incomplete and inaccurate.

In particular, with regard to the accuracy of the information provided to data subjects, it was found that the privacy policy:

-    did not specify in a granular manner the legal basis for each processing operation carried out, nor the type of data processed;

-    did not distinguish the purposes of the two distinct types of processing activities, namely the processing of data through ‘Chatbot Interaction’, intended to allow users to register for the service and interact with the platform, and the processing of data for ‘Model Development’, aimed at improving the security and performance of the Large Language Model (LLM) underlying the service offered (‘Model Development’);

-    did not clearly state that the service was intended exclusively for users aged 18 and above, although it encouraged users under the age of 13 not to use the service. In particular, paragraph 8 of the aforementioned privacy policy stated: ‘We do not knowingly collect Personal Data from children under the age of 13. If you are under the age of 13, please do not submit any Personal Data through the Services. We encourage parents and legal guardians to monitor their children’s Internet usage and to help enforce our Privacy Policy by instructing their children never to provide Personal Data on the Services without their permission. If you have reason to believe that a child under the age of 13 has provided Personal Data to us through the Services, please contact us, and we will endeavour to delete that information from our databases’. Although this information made it clear that the service was not intended for persons under the age of 13 (‘If you are under the age of 13, please do not submit any Personal Data through the Service’), it did not clearly specify that access to the chatbot was restricted only to users aged 18 and over, and that users between the ages of 13 and 18 were excluded. This latter circumstance was clarified by the Company only at a later stage;

-    did not provide any precise indication as to the storage period of the personal data or the criteria used to determine that period;

-    did not clarify whether personal data were transferred outside the EEA, nor did it specify, where such transfers occurred, the legal basis for processing or the adequacy measures adopted under Chapter V of the Regulation. More specifically, the information provided by the Company (namely the section of the privacy policy reading: ‘By using our services or providing us with any information, you consent to this transfer, processing, and storage of your information in the U.S.A., a jurisdiction in which the privacy laws may not be as comprehensive as those in the country where you reside or are a citizen’) appeared likely to mislead data subjects as to the transfer of their personal data to the USA. The absence of any transfer of personal data to third countries was confirmed by the Company itself in its letter of 8 May 2023 (Reg. No. 74173/23) concerning the applicability of the criterion of establishment in the European Union. It is therefore noted that the controller's own statements confirm the presence of misleading information;

-    Section 6, ‘Your data protection rights’, while not expressly referring to Article 22 of the Regulation, provided specific information on that right, thus giving users the unfounded belief that their personal data were subject to automated decision-making. The absence of an automated processing within the meaning of Article 22 of the Regulation was confirmed by the data controller in its reply letter (Reg. No. 74173/23), where it claimed that ‘although the chatbot relies on automated processes to generate responses, the Services do not make decisions based on profiling which produce legal effects on data subjects or similarly affect them within the meaning of Article 22 of the Regulation’. This again confirms, through the controller's own statements, the presence of misleading information.

With reference to Article 5(1)(a) of the Regulation, reference is made to the same binding decision of the EDPB mentioned in the previous paragraph (binding decision 1/2021), according to which transparency is to be regarded as a general concept which is then concretised in various provisions and specific obligations (e.g. Articles 12, 13, 14, 25 and 35). It is therefore necessary to distinguish the specific obligations arising from the principle of transparency (set out in Articles 12-14 of the Regulation) from the principle expressed in Article 5 of the Regulation, since although these obligations are a concretisation of the general principle, the latter has a broader scope.

The principle of transparency, in fact, is an overarching principle that reinforces other principles (such as fairness and accountability). This interpretation is confirmed by the fact that Article 83(5) of the Regulation allows for distinct administrative fines for infringing transparency obligations independently of any breach of the principle itself. In other words, the transparency obligations do not define the entire scope of the principle of transparency, it follows that a breach of the transparency obligations laid down in Articles 12 to 14 of the Regulation may also constitute a violation of the principle of transparency where such a breach is marked by elements of gravity and systematicity.

In the present case, the Garante considers that there has also been an infringement of the principle of transparency laid down in Article 5(1)(a) of the Regulation, taking into account the gravity (failure to provide information to the data subjects on the legal bases underlying the different personal data processing operations), the nature (lack of clear information on the essential elements of the processing, such as the legal basis, purpose, storage principle, transfer outside the EU) and the impact (this is a new type of processing connected to an innovative technology such as generative artificial intelligence) of the single specific infringements of the obligations under Articles 12 and 13 of the Regulation.

For the above reasons, the Garante considers that Luka infringed, as at the date of 2 February 2023, Articles 5(1)(a), 12 and 13 of the Regulation.

For the sake of completeness, it should be noted that subsequent technical investigations revealed that the data controller updated the privacy policy for the Replika service again on 23 February 2024. In this latest version, certain inaccuracies previously identified were rectified. Notably, the privacy policy in force at the date of adoption of this decision now granularly mentions the legal basis for each processing activity carried out by the controller and the type of data processed; it expressly clarifies that the service is exclusively intended for users over the age of 18, and contains no reference, not even implicit, to automated decision-making within the meaning of Article 22 of the Regulation. Nonetheless, the information provided under Articles 12 and 13 of the Regulation remains available only in English, does not include specific information on the personal data storage period or the criteria used to determine such a period, and may still mislead data subjects as to the possible transfer of their personal data to the USA.

5.3 ARTICLES.  5(1)(C), 6, 7, 8, 24 AND 25(1) OF THE REGULATION

The Authority notified Luka of the infringement of Articles 5(1)(c); 6; 7; 8; 24 and 25(1) of the Regulation for failing to set up users’ age verification systems as at the date of 2 February 2023. 
Pursuant to Article 5(1)(c) of the Regulation: ‘Personal data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed’.

Pursuant to Article 24(1) of the Regulation: ‘Taking into account the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for the rights and freedoms of natural persons, the controller shall implement appropriate technical and organisational measures to ensure and to be able to demonstrate that processing is performed in accordance with this Regulation. Those measures shall be reviewed and updated where necessary’.

Pursuant to Article 25(1) of the Regulation, the controller shall implement those measures ‘Taking into account the state of the art, the cost of implementation and the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for rights and freedoms of natural persons posed by the processing, the controller shall, both at the time of the determination of the means for processing and at the time of the processing itself ‘.

In the Guidelines 4/2019 on Article 25 of the Regulation, the EPDB clarified that ‘The core of the provision is to ensure appropriate and effective data protection both by design and by default, which means that controllers should be able to demonstrate that they have the appropriate measures and safeguards in the processing to ensure that the data protection principles and the rights and freedoms of data subjects are effective’ and called on data controllers to take into account also the obligations to provide specific protection to children under 18 and other vulnerable groups with a privacy-oriented approach in the data processing design process and default settings.

In the same Guidelines, the EDPB also emphasised that: ‘In line with Article 25(1) the controller shall implement appropriate technical and organisational measures which are designed to implement the data protection principles and to integrate the necessary safeguards into the processing in order to meet the requirements and protect the rights and freedoms of data subjects. Both appropriate measures and necessary safeguards are meant to serve the same purpose of protecting the rights of data subjects and ensuring that the protection of their personal data is built into the processing. Technical and organizational measures and necessary safeguards can be understood in a broad sense as any method or means that a controller may employ in the processing. Being appropriate means that the measures and necessary safeguards should be suited to achieve the intended purpose, i.e. they must implement the data protection principles effectively’.

The Company did not submit any defence statements or documents pursuant to Article 166(5) of the Code, following the Garante’s notice of infringement and initiation of proceedings, and thus did not provide any counter-arguments in relation to the alleged infringement of failing to set up users’ age verification systems. 

In the light of the above-mentioned rules and guidelines, the Authority notes that the data controller is required to implement appropriate technical and organisational measures to ensure, and be able to demonstrate, that processing is carried out in accordance with the Regulation and to process only personal data that are adequate, relevant and limited to what is necessary for the purposes for which they are processed. 

However, the preliminary investigation revealed that the Company failed to adopt measures to ensure specific protection of personal data processed through the Replika service in relation to children under the age of 18. In particular, the absence of age verification procedures, as well as the lack of mechanism to block or restrict access following declarations by users indicating that they are under 18, showed that the data controller did not assess, ex ante, the risks likely to arise from minors registering for and using the service. As a result, on the one hand, the controller did not take any measures to prevent, minimise or mitigate such risks, and, on the other, it processed more data than those necessary for the intended purposes of the processing (i.e. offering the service to users over the age of 18).

The investigation showed that, as at 2 February 2023, the Company had not implemented any age verification mechanisms, either at the time of registration to the Replika service or during its use, despite excluding minors from potential users.

In particular, the following shortcomings were identified:

-    absence of an age verification procedure (the system only required name, email address and gender) with the consequent risk of exposing minors to answers inappropriate for their level of development and self-awareness, including sexually explicit content;

-    lack of banning or blocking mechanisms, even when users declared or otherwise made it clear that they were underage, as well as the provision by the chatbot of responses that were clearly incompatible with the level of protection that should be guaranteed to children and, more generally, to all vulnerable individuals.

Until 2 February 2023, therefore, the Company had not adopted any technical and organisational measures to ensure compliance with the general principles of the Regulation or to safeguard the rights and freedoms of minors, thereby exposing them to the significant personal risks that the legislation in question is intended to limit, including the risk of receiving responses inappropriate to their level of psychophysical development and self-awareness.

Luka implemented age verification mechanisms only after receiving the request from the Garante in the context of the temporary limitation measure, adopted as a matter of urgency on 2 February 2023. In particular, during the discussions that followed the adoption of the aforesaid decision and with specific reference to the age gate issue, the Company explained that it had introduced an age gate across all registration pages of the Services restricting access to users over the age of 18 and which includes a ‘cooling-off period’ aimed at preventing users who are initially denied access based on their real personal data from immediately reattempting registration using a different date of birth. 

The Company also stated its intention to develop a process based on language analysis to detect and prevent use of the Services by individuals under the age of 18.

Prior to the Garante's intervention, therefore, all users—including minors—could register to and use the Replika service without being asked to undergo any age verification. As already clarified in the notice of infringement, in the Garante's view, the lack of a common standard capable of guaranteeing, with absolute certainty, the effectiveness of age verification systems, and the ongoing debate at European level on this issue, are not sufficient to exempt the data controller from its obligations, in particular the obligation to verify the user's actual capacity to enter into a contract, which is essential for its validity.

It follows from the foregoing that, as at 2 February 2023, the Company had not implemented, in accordance with Article 24 of the Regulation, the necessary measures to ensure that the processing of personal data at the time of registration for the Replika service complied with Articles 5(1)(c), 24, and 25 of the Regulation. In particular, the Company failed to implement technical and organisational measures ‘which are designed to implement data-protection principles, such as data minimisation, in an effective manner and to integrate the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of data subjects’. As a result, personal data were processed in excess of what was necessary for the purposes of a service which, according to the declarations of the data controller and the documents on file, was intended to be offered only to users over the age of 18 years.

With specific reference to the infringement of Article 5(1)(c) of the Regulation, it should be noted that, in this case, the adoption of appropriate technical and organisational measures—by design—intended ‘to implement data-protection principles, such as data minimisation’, not only constitutes a core requirement under Article 25(1) of the Regulation, but also represents an additional element substantiating the infringement of the minimisation principle itself, in line with Recital 78 of the Regulation.

More specifically, the Company's failure to adopt appropriate measures to safeguard access to and use of the Replika service resulted not only in the systematic processing by Luka of personal data exceeding what was necessary for the intended purpose of the processing (i.e. providing the service to users over the age of 18), but also that such processing involved data concerning vulnerable individuals (children, even potentially under the age of 13) who, because of this shortcoming, combined with the innovative technology underlying the service and the highly sensitive nature of the interactions generated by the chatbot, were exposed to a particularly high risk. 

The news reports that prompted the Garante to initiate its investigation—together with documented cases of self-harm associated with the use of the chatbot reported in foreign media and brought to the attention of the judicial authorities—support the Garante's allegations. On the basis of the principles expressed by the oft-referred to EDPB binding decision 1/2021, the gravity and impact of the infringements require due consideration, leading to the conclusion that both the infringement of the principle laid down in Article 5(1) of the Regulation and the specific infringement of the obligations under Articles 24 and 25(1) of the Regulation are substantiated.

The Garante, on the contrary, does not consider there to be sufficient grounds to establish an infringement—pursuant to Article 166(5) of the Code—of the provisions regarding the consent of minors, specifically the requirement of a positive act on the part of the child in relation to information society services, as set out in Articles 6, 7 and 8 of the Regulation. In particular, it should be noted that, as emerged during the preliminary investigation—contrary to what was erroneously stated in the version of the privacy policy dated 2 February 2023 (see § 5.2)—the Replika service was not, and is not, offered to minors. Consequently, the data controller was not required to comply with the obligation to identify a legal basis for processing operations which were presumed not to be carried out. 

Based on the foregoing, the Garante considers that, as at 2 February 2023, Luka infringed Articles 5(1)(c), 24 and 25(1) of the Regulation.

For the sake of completeness, it should be noted that, on the date of adoption of this decision, further technical assessments have revealed continuing deficiencies in the age verification system currently implemented by the controller; in particular, it was found that:

- after the user profile is created, it is possible to change the date of birth in the ‘My Profile’ section without this being followed by any verification by the data controller. As a result, children who initially provided a false date of birth to register could subsequently enter their real age and still retain access to the service; 

- the cooling-off period (24 hours) does not apply when the creation of the profile occurs while browsing in incognito mode; in fact, it appears that following an initial failed age check, users may still successfully complete the registration process by entering a different (event fictious) email address;

- no language analysis mechanisms are in place to systematically prompt age confirmation through the age gate process, when users indicate that they are under 18 years of age—except in limited cases where they provide specific inputs (i.e. they unambiguously declare that they are under 18). In such cases, the application prompts users to confirm that they are over 18.

The technical investigation also showed that while users are given the possibility of flagging certain conversations as inappropriate, it is not possible to identify the subsequent actions resulting from such reports.

6. CONCLUSIONS

Based on the above considerations, the Garante confirms the existence of the majority of the infringements alleged and notified in the notice of initiation of proceedings, as detailed below. It also declares the unlawfulness of the personal data processing carried out by the Company, in breach of Articles 5(1)(a) (in relation to both the principles of lawfulness and transparency) 5(1)(c), 6, 12, 13, 24 and 25(1) of the Regulation.

Having established the aforementioned infringements of the Regulation, the Garante shall adopt consequent corrective measures pursuant to Article 58(2) of the Regulation, specifically an order to bring processing operations into compliance under Article 58(2)(d) of the Regulation, and the imposition, under Article 58(2)(i) of the Regulation, of an administrative fine pursuant to Article 83(3) and (5) of the same Regulation. 

Furthermore, given the high sensitivity of the personal data involved, the Garante considers it appropriate to apply the ancillary penalty of publishing this decision on its website, as provided for by Article 166(7) of the Code and Article 16 of the Garante's Regulation No. 1/2019.

As previously noted, the Garante reserves the right to initiate a separate and autonomous investigation to assess the lawfulness of the processing operations carried out by the Company with a particular focus on the legal bases applicable throughout the entire lifecycle of the generative AI system underlying the Replika service.

7.  CORRECTIVE MEASURES PURSUANT TO ARTICLE 58(2)(D) OF THE REGULATION

Pursuant to Article 58(2) of the Regulation, the Garante is granted a series of corrective powers—both of a prescriptive and sanctioning nature—to be exercised when unlawful processing of personal data is found.

Such powers include, pursuant to Article 58(2)(d) of the Regulation, the power ‘to order the controller […] to bring processing operations into compliance with the provisions of this Regulation, where appropriate, in a specified manner and within a specified period’.

From the findings and considerations set out in the preceding paragraphs, it emerges that, as at 2 February 2023, Luka infringed Articles 5(1)(a) (in relation to both the principle of lawfulness and the principle of transparency), 5(1)(c), 6, 12 and 13, as well as Articles 24 and 25(1) of the Regulation. However, following the urgent measure of the Garante, the Company implemented some measures to remedy the identified shortcomings and, subsequently, took additional measures in relation to the infringements notified in the notice of initiation of proceedings, which brought processing into compliance with data protection legislation.

Specifically, Luka remedied the infringement of Articles 5(1)(a) and 6 of the Regulation by amending the privacy policy (see latest version dated 23 February 2024), detailing the legal bases for the different processing operations carried out through the Replika service.

Following the amendments to the privacy policy as described above, it is considered that, at this stage, there are no grounds for adopting further corrective measures under Article 58(2) of the Regulation.

On the other hand, with regard to the infringement of Articles 5(1)(a), 12 and 13 of the Regulation, concerning information obligations, and of Articles 24 and 25(1) and 5(1)(c) of the Regulation, concerning the age verification system, certain aspects remain non-compliant with the Regulation, which the Garante considers must be addressed by specific corrective measures.

In particular, as regards the information obligations, the Garante found that, as of today, Luka’s privacy policy (latest version dated 23 February 2024) is still non-compliant with data protection legislation insofar as: i) it is available only in English; ii) it does not specify the storage periods of personal data or the criteria used to determine such periods; iii) it may still mislead data subjects as to the possible transfer of their personal data to the USA.

Therefore, pursuant to Article 58(2)(d) of the Regulation, the controller is ordered to bring the privacy policy into compliance with Articles 5(1)(a), 12 and 13 of the Regulation by remedying the above shortcomings.

Furthermore, regarding the age verification system, the Garante found that, at the date of adoption of this decision, the age verification system used by the controller does not comply with the principle of data minimisation and with the principles of privacy by design and by default, in that:

- after the user profile is created, users can change their date of birth in the ‘My Profile’ section without this being followed by any verification by the controller. As a result, children who initially provided a false date of birth to register could subsequently enter their real age and still retain access to the service;

- the 24-hour cooling-off period does not apply when the creation of the profile occurs while browsing in incognito mode; in fact, it appears that following an initial failed age check, users may still successfully complete the registration process by entering a different (even fictious) email address;

- the controller has not implemented any language analysis mechanisms prompting users to reconfirm their age through the age gate process when there are clear indications that the user is under the age of 18, except in limited cases where the user provides specific inputs (i.e. they unambiguously declare that they are under 18). Only in such cases, does the application prompt users to confirm that they are over 18.

However, the Garante positively noted the implementation by Luka of a function allowing users to flag certain conversations as inappropriate, although it was not possible to determine the effect of such reports.

In view of the above, pursuant to Article 58(2)(d) of the Regulation, the controller is ordered to bring the age verification system into compliance with Articles 5(1)(c), 24 and 25(1) of the Regulation by remedying the shortcomings identified above.

8. INJUNCTION ORDER FOR THE APPLICATION OF THE ADMINISTRATIVE FINE AND ANCILLARY PENALTIES

The Garante, pursuant to Articles 58(2)(i) and 83 of the Regulation and Article 166 of the Code,  shall have the power to impose an administrative fine pursuant to Article 83, in addition to, or instead of corrective measures referred to in the same paragraph.

In determining the amount of the administrative fine, the Garante shall take into account the principles and interpretation provided by the EDPB in their Guidelines 4/2022 on the calculation of administrative fines under the GDPR, version 2.1, adopted on 24 May 2023.

Based on the arguments set out above, the Garante found that the following provisions of the Regulation have been infringed: Articles 5(1)(a) and 6; Articles 5(1)(a), 12 and 13; Articles 5(1)(c), 24 and 25(1) of the Regulation.

In the present case, it should first be noted that the Company engaged in a number of conducts leading to multiple infringements, as specifically outlined and substantiated in the preceding paragraphs. The infringements relating to the legal basis (Articles 5(2) and 6 of the Regulation), transparency (Articles 5(1)(a), 12 and 13), and the age gate (Articles 24 and 25(1)) may, under the principle of unity of action, be considered together under Article 83(3) of the Regulation, which provides that, if a controller or processor infringes several provisions of the Regulation for the same or linked processing operations, the total amount of the administrative fine shall not exceed the amount specified for the gravest infringement. Specifically, in relation to such infringements, it may be assumed that the processing operations are linked, as defined in paragraph 28 of the above-mentioned guidelines (a unitary conduct consists of several parts that are carried out by a unitary will and are contextually, spatially and temporally related in such a close way that, from an objective standpoint, they would be considered as one coherent action). Among the aforementioned infringements, the most serious is considered to be the infringement of transparency obligations, given that both Article 5(1)(a) (Principle of transparency) and Articles 12 and 13 (Rights of the data subject) are subject to the administrative fines laid down in Article 83(5) up to a maximum amount of €20 million or, in the case of an undertaking, up to 4% of the total worldwide annual turnover of the preceding financial year, whichever is higher.

Pursuant to Article 83(1) of the Regulation, the administrative fine shall be effective, proportionate and dissuasive in each individual case. According to the aforementioned guidelines, the EDPB has clarified that the calculation of administrative fines should commence from a harmonised starting point which forms the beginning for further calculation, in which all circumstances of the case are taken into account and weighed (see par. 46). The harmonised starting point should consider three elements: 1) the categorisation of the infringements by nature under Articles 83(4)–(6) of the Regulation; 2) the seriousness of the infringement; 3) the turnover of the undertaking (see par. 48).
As regards the first aspect, two infringements are found, in abstract terms, of a more serious nature (Article 83(5) of the Regulation) and one of a less serious nature (Article 83(4) of the Regulation). The first two concern the infringement of the legal basis and transparency, while the third concerns the infringement of Article 25 of the Regulation.

Regarding concrete seriousness, the elements to be taken into account are: a) the nature, gravity and duration of the infringement (Article 83(2)(a) of the Regulation); b) the intentional or negligent character of the infringement (Article 83(2)(b) of the Regulation) ; c) the categories of personal data affected (Article 83(2)(g) of the Regulation).

In this case, regarding the three infringements linked under the principle of unity of action (legal basis, transparency, and data protection by design and by default), the level of seriousness of the infringements must be considered high, given that: i) the nature of the infringements concerns two fundamental principles (accountability and transparency), namely, on the one hand, the controller’s failure to demonstrate that it had identified the legal bases for processing prior to the commencement of such processing, and, on the other hand, the failure to provide appropriate information to the data subject, particularly regarding the purposes of the two distinct types of processing (‘Chatbot Interaction’ and ‘Model Development’), and the involvement of data such as minors’ data, in excess of what was necessary to achieve the service provision purpose; ii) the nature of the processing involves significantly high risks as it is linked to an innovative, disruptive and rapidly evolving technology; iii) the processing has a cross-border nature and global scope, with effects that are practically uncontrollable by data subjects; iv) the purpose of the processing falls within the core business of the Company; v) the number of Italian data subjects involved cannot be precisely quantified, but in general terms it can reasonably be presumed to be very high, as information available on Google’s App Store (Google Play) shows that the application has exceeded 10 million downloads (suggesting a similar figure may exist, though not verifiable, for downloads via Apple Store), while academic sources (Shikhar Ghosh, Replika: Embodying AI, Harvard Business School, Faculty & Research) indicate that the Company had already reached 10 million users by January 2022; vi) the nature of the data has also involved special category data—considering the nature of the chatbot itself (which is still presented as ‘an AI companion always ready to chat when you need an empathetic friend’)— and, in the absence of age verification mechanisms and data filtering systems, personal information relating to underage users. The duration of the infringement is significant, given that the app was released to the public in November 2017; indeed, the fact that the chatbot’s success materialised later in time does not counterbalance the judgement of high seriousness, since the end of the infringement was triggered by and coincided with the Garante’s urgent measure. 

All the infringements must be regarded as being unintentional. As stated by the Article 29 Working Party in the Guidelines on the application and setting of administrative fines for the purposes of Regulation (EU) 2016/679, adopted on 3 October 2017 and endorsed by the EDPB on 25 May 2018 (WP 253 guidelines), ‘intent’ includes both knowledge and wilfulness to commit a violation, whereas ‘unintentional’ means that there was no intention to cause the infringement, although there was a breach of the duty of care. The Court of Justice of the European Union (CJEU), in a recent ruling (Judgment C-807/21 of 5 December 2023), established that it is the responsibility of the supervisory authority to determine whether an infringement has been committed intentionally or negligently by the controller, as only an unlawful infringement constitutes a condition for an administrative fine to be imposed. In this regard, it should be noted that while the CJEU stated in the aforementioned judgment that Article 83 of the Regulation does not allow an administrative fine to be imposed without it being established that the infringement was committed intentionally or negligently by the controller (see par. 75), it also upheld the fundamental principle ‘ignorantia legis non excusat’, stating that ‘a controller can be penalised for conduct falling within the scope of the GDPR where that controller could not be unaware of the infringing nature of its conduct, whether or not it is aware that it is infringing the provisions of the GDPR’ (see par. 76). This principle had already been established by the CJEU in another case (Judgment C-601/16 of 25 March 2021, paragraphs 97 and 98) where it affirmed that ‘an undertaking may be punished for conduct falling within the scope of Article 101(1) TFEU where that undertaking could not have been unaware of the anticompetitive nature of its conduct, whether or not it was aware that it was infringing the competition rules of the Treaty (see, to that effect, judgment of 18 June 2013, Schenker & Co. and Others, C 681/11, EU:C:2013:404, paragraph 37). It follows that the fact that that undertaking has characterised wrongly in law its conduct upon which the finding of the infringement is based cannot have the effect of exempting it from imposition of a fine in so far as it could not be unaware of the anticompetitive nature of that conduct’ (Judgment of 18 June 2013, Schenker & Co. and Others, C 681/11, EU:C:2013:404, paragraph 38). In the present case, it is considered that Luka could not, at the time its service was made available (also) to users located within the European Union, and specifically in Italy, avoid a duty to be aware of and to apply the Regulation, which, as is well known, safeguards a fundamental right provided for and protected by Article 8 of the Charter of Fundamental Rights of the European Union. In light of the specific circumstances of the case, the context in which the controller operates, and the disruptive and rapidly evolving technology characterising its activities, it is considered that the failure to ensure compliance of the personal data processing with EU data protection law is indicative of the negligence underlying the concept of fault and demonstrates the existence of such a subjective element on the part of the Company. Furthermore, this fault must be regarded as serious, precisely due to the scale and innovative nature of the service offered, which entails large-scale processing of personal data at a global level. 
In order to determine the amount of the administrative fine, the aggravating factors set out in Article 83(2)(d) and (f) of the Regulation are also applicable. 

As regards the first aspect, the degree of responsibility of the controller must be considered high due to the failure, at the time of the service launch, to adopt appropriate technical and organisational measures to mitigate the risks to the rights and freedoms of data subjects and enable them to exercise the rights laid down in Chapter III of the Regulation. As for the second circumstance, regarding the degree of cooperation, it should be noted that although the Company responded to the request for information, it did not submit any defence statements in reply to the notice of infringement notified pursuant to Article 166 of the Code, thereby demonstrating limited cooperation with the Garante.

In order to determine the amount of the administrative fine, the following measures implemented by the controller to remedy the infringement and mitigate its possible adverse effects (Article 83(2)(f) of the Regulation), are considered as mitigating factors. Namely:

-    the updates to the privacy policy, both immediately following Decision No. 39/2023 and subsequently, particularly with reference to the latest version dated 23 February 2024, as described in the previous paragraph, although such updates are considered not exhaustive;

-    the implementation of age-gating mechanisms as described in the previous paragraph, although not exhaustive. 

Based on the above elements, assessed in their entirety, and in the absence of data regarding the total worldwide annual turnover of the Company for the preceding financial year, it is hereby decided that, pursuant to Article 83(3) of the Regulation, the total amount of the administrative fine is set at EUR 5,000,000.00 (five million), equal to half of the maximum amount provided for in Article 83(5) of the Regulation. This amount has been determined as follows:

•    pursuant to Article 83(3) of the Regulation, considering the conduct as a single instance due to the interrelated nature of the processing activities for the reasons stated above, the administrative fine for the most serious infringement—namely of Articles 5(1)(a), 12, and 13 of the Regulation—is set at EUR 3,000,000.00;

•    the fine is increased by EUR 1,000,000.00 for the infringement of Articles 5(1)(a) and 6 of the Regulation;

•    the fine is further increased by EUR 1,000,000.00 for the infringement of Articles 5(1)(c) and 25(1) of the Regulation.

This administrative fine is considered to be effective, proportionate and dissuasive, pursuant to Article 83(1) of the Regulation.

Taking into account the particular sensitivity of the data processed, it is considered that the ancillary penalty of publication of this decision on the Garante's website shall apply, as provided for by Article 166(7) of the Code and Article 16 of the Regulation of the Garante’s Regulation No. 1/2019; this in light of the nature and gravity of the established infringements, in particular considering that they involve large-scale processing operations affecting a high number of data subjects, the data protection risks associated with the provision  of a service based on innovative and complex technology in the absence of appropriate safeguards. It is also considered that there is a general interest in the topic of generative artificial intelligence that requires the widest possible awareness of the Garante’s position on the matter.

Finally, it is considered that the conditions set out in Article 17 of the Garante’s Regulation No. 1/2019 concerning internal procedures with external relevance aimed at performing the tasks and exercising the powers entrusted to the Garante, are met for recording the infringements identified herein in the Garante’s Internal Register, as provided for by Article 57(1)(u) of the Regulation.

BASED ON THE FOREGOING, THE GARANTE

pursuant to Article 57(1)(f) of the Regulation, declares unlawful the processing activities described and carried out by Luka Inc., based in 490 Post St Suite 526, San Francisco, California, United States of America, as set out in the reasoning, for infringing Articles 5(1)(a) (with regard to both the principles of lawfulness and transparency), 6, 12, 13, 5(1)(c), 24, and 25(1) of the Regulation and, consequently:

a)    pursuant to Article 58(2)(d) of the Regulation, orders the Company, within thirty days of notification of this decision, to bring its processing activities into compliance with the provisions of the Regulation, in particular by aligning its privacy policy with Articles 5(1)(a), 12, and 13 of the Regulation, and aligning its age verification system with Articles 5(1)(c), 24, and 25 of the Regulation, remedying the shortcomings identified in paragraphs 5 and 6 of this decision, respectively;

b)    pursuant to Article 157 of the Code, orders the Company to inform the Authority, within sixty days of notification of this decision, of the initiatives undertaken to implement the corrective measure referred to in the preceding point; failure to comply with the provisions set out in this point may result in the imposition of the administrative fine provided for in Article 83(5) of the Regulation

ORDERS

Luka Inc., based in 490 Post St Suite 526, San Francisco, California, United States of America, to pay the sum of EUR 5,000,000.00 (five million) as an administrative fine for infringing Articles 5(1)(a) and 6; Articles 5(1)(a), 12, 13, 5(1)(c), 24 and 25(1) of the Regulation, stating that the infringing party, pursuant to Article 166(8) of the Code, has the right to settle the dispute by paying, within sixty, an amount equal to half the imposed fine.

REQUIRES

a)    the aforesaid Company, in the event of failure to settle the dispute pursuant to Article 166(8) of the Code, to pay the sum of EUR 5,000,000.00 (five million), according to the modalities indicated in the annex, within sixty days of notification of this decision, under penalty of the adoption of the consequent executive actions pursuant to Article 27 of Law No. 689/1981.

PROVIDES

a)    that this decision be published, pursuant to Article 154-bis of the Code and Article 37 of the Garante’s Regulation No. 1/2019;

b)    that the application of the ancillary penalty of the publication of this injunction order on the Garante’s website, as provided for by Article 166(7) of the Code and Article 16 of the Garante’s Regulation No. 1/2019, be applied;

c)    that this decision be recorded in the Garante’s Internal Register—as laid down in Article 57(1)(u) of the Regulation and Article 17 of the Garante’s Regulation No. 1/2019 concerning internal procedures with external relevance, aimed at performing the tasks and exercising the powers entrusted to the Garante—regarding infringements and measures adopted in compliance with Article 58(2) of the Regulation.

The Garante reserves the right to investigate and assess in a separate and autonomous proceeding, the aspects concerning the lawfulness of the processing operations carried out by Luka Inc., with specific reference to the legal bases for processing applicable throughout the entire lifecycle of the generative AI system underlying the Replika service.

Under Article 78 of the Regulation, Article 152 of the Code and Article 10 of Legislative Decree No. 150/2011, this decision may be challenged before the ordinary judicial authority, by lodging an appeal with the ordinary court of the controller’s place of residence, within thirty days from the date the decision was notified, or within sixty days if the appellant resides abroad.

Rome, 10 April 2025             

 

THE PRESIDENT
Stanzione

THE RAPPORTEUR
Scorza

THE ACTING SECTRETARY GENERAL
Filippi