Mass Profile Deletion

25 Oct 2026 » Platform

In my series on Deletion in the Adobe Experience Platform (AEP), I explained how to delete data stored in AEP, without any particular use case in mind. This post will address one of the most common use cases: deleting a significant amount of profiles, both from the Real-Time Customer Profile (RTCP) and the Data Lake. I will go step by step, so expect a larger-than-usual post.

Why Bulk Deletion

It is not uncommon to find a situation when your profile count exceeds your expectations. There are two typical situations that lead to this case:

  • Due to a mistake in your ETLs or export jobs, you have found that you have ingested your entire customer database instead of only the subset that you intended to send to AEP.
  • You have never performed a cleanup of profiles (lost customers, legal restrictions…) and now you have too many unaddressable profiles.

None of the methods that I have described so far help, at least as I have explained them. Either they are too slow or too radical. There is a process to reduce the profile count to the right number.

Warnings

Before you attempt anything, even logging into AEP:

  • Read this whole post in detail.
  • Understand each of the steps completely.
  • Remember that data will irrevocably be deleted.

Preparation

Before starting the process, some decisions and preparations need to be made.

Dataset Evaluation

The first step in the preparation phase is to determine which datasets contain the profile fragments that need to be trimmed. These are usually datasets that are sourced from your internal customer databases. The rest of the post will refer to a single dataset, but there may be more. You will have to apply the same actions to all datasets.

Backups

The process described in this post requires the deletion of a dataset. Before starting the process, you need to decide whether you want to keep all historical data in this dataset. In general, I would expect that you do not need a backup, as the data comes from your own systems. However, you may need to keep the data in the Data Lake to run historical reports.

Lower Environment

This process is going to have a significant impact on the data. There are multiple steps, which means that the process can break in many places. It is strongly recommended that you first test it in a lower environment.

Scripts

I also recommend that you write some scripts to automate the steps. The process I am going to describe assumes that you are going to do all the steps manually. However, most, if not all, of the steps can be automated using scripts and the AEP API. I recommend using Python and AEPP.

These scripts will also make the process faster and less error-prone.

ETL Full Export

I assume that your ETL that sends data to AEP sends only deltas. However, the process described below requires a full export of all active profiles. You will need to develop an ETL to send a full export, which should be configured to run on demand, and not on a schedule.

ETL Delta Export

If the reason that brought you to this post is that you are sending all customer profiles to AEP and you only want a subset, you will need to update your existing ETL to send deltas to AEP containing only the subset of profiles that you want in AEP. Now is the time to work on this development.

Current Campaigns

Since data is going to be heavily manipulated, it may affect currently running marketing campaigns. You need to review all campaigns and decide whether they will be affected and whether they should be stopped during the process. For example, Adobe Journey Optimizer journeys may need to be stopped if profiles to be deleted are in journeys.

Dataset Consumption

You should conduct an audit to surface all parties and entities relying on the dataset that is going to be deleted. Some examples of places where this dataset may be used, and may be affected, are:

The output of this audit will also determine the places that need to be adjusted to point to the new dataset that will be created later in the process.

Day 1

The goal of day 1 is to create a new dataset with only the desired profiles.

Step 1.1

Capture the number of profiles in the sandbox:

  • Total number of profiles as shown in the reports
  • Profiles that are meant to be deleted, probably through a query

Step 1.2

Pause the ETL that generates the delta files you are currently sending to AEP, which feed the dataset to be deleted.

Step 1.3

Delete the files generated by previous executions of the ETL, which are stored in the cloud storage used by the source connector.

Step 1.4

Create a new dataset using the same schema as the current dataset, and configure it:

  • Enable for profile
  • Apply any data governance labels

Step 1.5

Configure a new source connection for the new dataset, cloning the mappings from the previous source connection. In step 1.3 you should have deleted all files in the source folder. If you have decided to keep those files, you must turn off the backfill option on the source connection.

Step 1.6

Run the ETL described in ETL full export to generate the full export of all profiles that need to stay in AEP. Ingest the ETL output files containing the good profiles into the new dataset. Check the AEP monitoring log for any rejected records.

Step 1.7

Wait until all profiles have been ingested into the Unified Profile Store. The standard recommendation for this step is to wait 24 hours, but closer monitoring may shorten this step.

Day 2

At this moment in time, you have two datasets: one with all profiles, and another one with only the profiles that should stay in AEP. If the dataset evaluation came up with more than one dataset, you should have by now multiple pairs of datasets, one for each dataset to be deleted.

To avoid any data gaps, all steps described for this day should be taken before the end of the day. This is why I suggested the creation of one or multiple scripts.

Step 2.1

Perform the following data validations:

  • Capture the total number of profiles in the sandbox.
  • The number of profiles should not have changed.
  • Sample some profiles and check they have the expected attributes.
  • Check for any other discrepancy.

Step 2.2

Decide between Go and No Go based on the results of step 2.1:

  • If it is a No Go:
    1. Delete the newly created dataset.
    2. Evaluate what went wrong.
    3. Fix the issues.
    4. Go back to step 1.4.
  • If it is a Go, proceed to the next step.

Step 2.3

If you have decided to create a backup:

  1. Create a backup dataset. Note that this is a new dataset, different from the original one and the new one created in step 1.4. Never enable this backup dataset for profile.
  2. Copy all data from the original dataset to the backup dataset using Data Distiller.

Step 2.4

This is the moment where the output of the dataset consumption preparation step is used. In summary, all references to the original datasets must be changed to the new dataset.

This is also the last step when you may stop any campaigns that may negatively be affected by the deletion of profiles.

Step 2.5

This is the most critical step in the process: delete the original dataset, the dataset with all profiles. There is no undelete or restore. Let me repeat it: if you delete the wrong dataset, there is no way back, not even if you open a P1 ticket with client care.

This step will clean up the Data Lake and the identity store. Only good profiles will remain in the profile store from the new dataset. This step will take time: expect multiple hours or a couple of days.

Do not wait until the dataset is completely deleted to move to the next step.

Step 2.6

Enable the ETL that generates the delta files. If you have configured step 1.5 correctly, these delta files will now feed the new dataset. You can also reactivate all stopped campaigns.

Subsequent Days

If all has gone well, all profiles that were supposed to be deleted should not be in the profile store anymore. It is recommended to keep track of the profile count. It may also be a good idea to run test scripts for a few days, to confirm that the data is correct.

 

Photo by Erik Mclean on Unsplash

Related Posts