Bonus Lab: QLDB replay

Reconstruct the Aurora Serverless database to a specific point in time relative to QLDB

In this bonus lab, we will see how to replay QLDB to stream events that will stop at a desired time.

Due to the lab only running for roughly 30 minutes, we will replay all events that happened from the first 5 minutes of our simulation test.

Ensure that no new records are being processed in the Aurora Serverless database before starting this section.

New Schema

We already have an Aurora Serverless database up and running. A new schema will need to be added to the Aurora Database. To save time, this will be completed by a CloudFormation template.

CloudFormation Template

A cloudformation script will be used to provision the infrastructure for this lab. Use the quicklink below to start the cloudformation template in the AWS console.

The following resources will be created via the cloudformation template.

Region Launch CloudFormation Template
US East (Virginia) Launch Stack in us-east-1
US East (Ohio) Launch Stack in us-east-2
US West (Oregon) Launch Stack in us-west-2
Europe (Frankfurt) Launch Stack in eu-central-1

or, download the file to your local workstation and create a CloudFormation stack by uploading the template.

You will see the Quick create stack page as shown below. In the Stack name block, leave the Stack name as streaming-lab-bonus.

In the Parameters block, insert the AuroraEndpoint, all subnets for SubnetName, and the SecurityGroupName.

In the Other parameters section. Insert the values for QLDBLedgerName and the QLDBStreamRoleName (do not include the region)

For the StreamStartTime enter the time that was captured at the end of the load simulation cloudformation step. Add 5 minutes to the StreamEndTime.

Under Capabilities, click the check box for “I acknowledge that AWS CloudFormation might create IAM resources with custom names”.

Click on Create stack.

Using the RDS Query Console

Finally, we are ready to query QLDB documents from the Aurora Serverless databases. Head back over to the RDS service page.

From the AWS console, start typing Aurora or RDS and click on RDS when the service appears.

Now, click on the Query Editor that is on the left Amazon RDS menu.

There should be no need to sign in back into the database. If credentials are needed, follow the same steps outlined in the earlier sections.

Now, lets make sure that new tables were created in the Aurora databases to support the new QLDB stream.

In the Query Editor enter the following SQL command to list the tables. Ensure to insert the database name.

show TABLES FROM <Lowercase QLDB Ledger Name>;

As shown below, 4 more tables have been created with the name bonus appended to each table.

Now, look at the schema for one of the bonus tables and compare to the existing tables. Insert database name.

DESCRIBE <Lowercase QLDB Ledger Name>.person;
DESCRIBE <Lowercase QLDB Ledger Name>.personbonus;

Recall that that the schema for the person table has only docid as the primary key.

Notice that the schema for personbonus has DocId and DocV as a composite primary key.

Execute the below query to get a sample of the data in the bonus lab. Insert database name.

SELECT * FROM <Lowercase QLDB Ledger Name>.personbonus ORDER BY DocId LIMIT 10;

Notice that the document Ids will have all revisions now in the Aurora Database. Additionally, only data that covered a 5 minute period was streamed by QLDB. Not only are you now able to see the data in the Aurora Databases as a different projection or view, but the QLDB data has been replayed to a specific and finite period in time.

Reliability, Scalability, Evolvability

Having the ability to replay a database increases reliability for the overall architecture. When architectural components are failing and the downstream system begins to work with an inconsistent state. Processes can be terminated and replayed with no need to worry about data consistency or corruption from the source database.

Replaying a database can increase scalability as running multiple projections or views which allows for horizontal scaling. This can be done to a single database (as demonstrated in this lab), or to replicas of a database, or to multiple different database engines.

Reliability and scalability are common to AWS fully managed services. With QLDB having the option to roll-back or replay state, we now have Evolvability. Not only can we scale to our changing business needs, but we can adapt and evolve in the way business or customers need us to. Jeff Bezos talked about one-way doors and two-way doors in his 1997 letter to shareholders and that one-way doors are consequential and irreversible. Two-way doors are reversible and if the wrong decision was made for a business or its customers, simply reverse the decision and try again. Reversible decisions make sense in business and with QLDB, think of a replay or roll-back as adding a two-way door into the architecture.

Congratulations! This bonus lab is complete and head over to the clean up section.