In my previous post on Data Distiller Optimization, I finished by mentioning the SNAPSHOT clause. In this post, I will explain in more detail how to use this Data Distiller feature. This will be a deep technical post.
Additional Context
Before going into the particulars of this post, I want to clarify a few important details:
- I will assume that you have already read the previous post. If you have not, I highly recommend that you do so first to get the full context.
- This post will be largely based on the official documentation page for Incremental load in Query Service. The information in the Experience League is sometimes difficult to follow, so my goal here is to make it more digestible. For example, I do not find any obvious connection between this help page and the documentation of the
SNAPSHOT clause.
- Throughout this post, I will use the word “table”. However, remember that there are no such database tables in the Adobe Experience Platform (AEP). The underlying data is stored in a dataset, as usual.
- The
SNAPSHOT clause and, therefore, this post only applies to scheduled queries.
Auxiliary Table Creation
The first step is to create an auxiliary table to keep track of the snapshot IDs. We will use the same table for all queries.
DROP TABLE IF EXISTS checkpoint_log;
CREATE TABLE checkpoint_log AS
SELECT
cast(NULL AS string) process_name,
cast(NULL AS string) process_status,
cast(NULL AS string) last_snapshot_id,
cast(NULL AS TIMESTAMP) process_timestamp
WHERE false;
Be very careful with this query. If you accidentally run it a second time in production, the first command DROP TABLE IF EXISTS checkpoint_log; will delete the table and, with it, all existing data.
Initialization
The query that I will show in the next section expects an initial value in the table. For every such query, you need to insert a row in the table:
INSERT INTO checkpoint_log
SELECT
'DIM_TABLE_ABC' process_name,
'SUCCESSFUL' process_status,
cast(NULL AS string) last_snapshot_id,
CURRENT_TIMESTAMP process_timestamp;
You must replace DIM_TABLE_ABC with a unique name for your query. There is no need to use the name of the dataset you are querying, you can use any other unique text for your query, as long as you use it consistently. In fact, if you are going to run multiple queries against the same dataset, I do not recommend that you use the dataset name, but a different name for each query.
As with the table creation, you should not run this query a second time for the same process_name in production. It will add a new entry with the current timestamp and all previous entries will be ignored. The effective consequence is that the next run of the main query will start from the beginning of time.
Incremental Processing
Now it is time to run your scheduled query. This is how it will look like:
$$ BEGIN
SET resolve_fallback_snapshot_on_failure=true;
SET @from_snapshot_id =
SELECT coalesce(last_snapshot_id, 'HEAD')
FROM checkpoint_log a
JOIN (
SELECT MAX(process_timestamp)process_timestamp
FROM checkpoint_log
WHERE process_name = 'DIM_TABLE_ABC' AND process_status = 'SUCCESSFUL'
) b
ON a.process_timestamp = b.process_timestamp;
SET @to_snapshot_id = SELECT snapshot_id FROM (SELECT history_meta('DIM_TABLE_ABC')) WHERE is_current = true;
SET @last_updated_timestamp = SELECT CURRENT_TIMESTAMP;
CREATE TABLE DIM_TABLE_ABC_Incremental AS
SELECT * FROM DIM_TABLE_ABC SNAPSHOT BETWEEN @from_snapshot_id AND @to_snapshot_id;
INSERT INTO checkpoint_log
SELECT
'DIM_TABLE_ABC' process_name,
'SUCCESSFUL' process_status,
cast(@to_snapshot_id AS string) last_snapshot_id,
cast(@last_updated_timestamp AS TIMESTAMP) process_timestamp;
EXCEPTION
WHEN OTHERS THEN
SELECT 'ERROR';
END
$$;
Let’s analyze this code. There is a lot going on in just 30 lines.
Anonymous block
You need to insert your query in an anonymous block using the keywords BEGIN, EXCEPTION and END to be able to exit gracefully in case of errors.
Expired snapshots
The documentation mentions that there could be problems with expired snapshots. I do not exactly know what they mean by this concept, and it is not clearly explained either.
In any case, the proposed solution is to add the line SET resolve_fallback_snapshot_on_failure=true; to the query. I have added it just in case, but I am not sure it is always needed.
Retrieve the last snapshot ID
Next, you need to get the last snapshot ID from the auxiliary table checkpoint_log.
SET @from_snapshot_id =
SELECT coalesce(last_snapshot_id, 'HEAD')
FROM checkpoint_log a
JOIN (
SELECT MAX(process_timestamp) process_timestamp
FROM checkpoint_log
WHERE process_name = 'DIM_TABLE_ABC' AND process_status = 'SUCCESSFUL'
) b
ON a.process_timestamp = b.process_timestamp;
You need to replace DIM_TABLE_ABC with the unique name for your query that you decided when you initialized the table.
Now you have the variable @from_snapshot_id set to the snapshot ID of the last execution of the query. In the first run, it will be set to NULL thanks to the initialization.
Retrieve the current snapshot ID
Using the function history_meta, you need to get the last snapshot ID of the dataset DIM_TABLE_ABC. In this case, you have to replace this dataset name with your real dataset name, not the process_name value.
It took me a bit of time to understand why we need this value. I initially thought that we could just use 'TAIL' but then I realized that there could be race conditions between the query execution and new batches: if a new batch arrives while the query is running, there could be a case where the new data would be skipped.
SET @to_snapshot_id = SELECT snapshot_id FROM (SELECT history_meta('DIM_TABLE_ABC')) WHERE is_current = true;
Get the current timestamp
To sort the entries in checkpoint_log the query relies on the timestamp of the execution of the query, so we need to retrieve it too.
SET @last_updated_timestamp = SELECT CURRENT_TIMESTAMP;
Your query
Now is when you finally run your complex query, probably dozens of lines long. The example above shows a very simple query as a placeholder:
CREATE TABLE DIM_TABLE_ABC_Incremental AS
SELECT * FROM DIM_TABLE_ABC SNAPSHOT BETWEEN @from_snapshot_id AND @to_snapshot_id;
This can be any query. The fact that the tables are called DIM_TABLE_ABC and DIM_TABLE_ABC_Incremental is irrelevant and it has no connection with the value you have inserted into the auxiliary table. The only important relation here is that the process_name value in checkpoint_log and your query is 1:1. For every query you schedule, you need to come up with a new process_name.
The truly important part of the query is SNAPSHOT BETWEEN @from_snapshot_id AND @to_snapshot_id, which you will need to embed in your query in the right place. This tells the Query Service engine to only process the dataset entries between these two markers.
If you have to combine two or more datasets that need to also use snapshot IDs, you will have to replicate the code to manage them, using different variable names and process_name values for each dataset. I am not implying that you have to use it for all your datasets in a query. If you are combining a large dataset with a small one, you may not need to use the SNAPSHOT clause with the small dataset.
In general, this central body of the query will be developed in isolation before adding the rest of the surrounding code:
- Start by developing and testing your query without the
SNAPSHOT clause.
- Once your base query works, add the
SNAPSHOT clause.
- Add some manual entries to
checkpoint_log and test the query with the SNAPSHOT clause.
- Add the rest of the code around your main query and test it.
- Schedule your query and check for a few days that it works.
Finally, you are ready to go to production:
- Manually insert the initialization entry.
- Copy the query from development or staging to production.
- Schedule the query.
It goes without saying that you should monitor it for a few days.
Update the auxiliary table
With the query successfully executed, you should insert the last snapshot ID into the auxiliary table:
INSERT INTO checkpoint_log
SELECT
'DIM_TABLE_ABC' process_name,
'SUCCESSFUL' process_status,
cast(@to_snapshot_id AS string) last_snapshot_id,
cast(@last_updated_timestamp AS TIMESTAMP) process_timestamp;
The value of @to_snapshot_id in the current execution will become the value of @from_snapshot_id in the next execution.
As I also explained in my previous post, there is no UPDATE operation with datasets. This is why you have to add the value to the end of the dataset (INSERT) and add a timestamp.
Finalization
I am not intimately familiar with exception handling in SQL. My understanding is that this code just returns 'ERROR' if any query failed.
EXCEPTION
WHEN OTHERS THEN
SELECT 'ERROR';
END
Photo by Creative Minds Factory on Unsplash
]]>
