To start off the lab, an Amazon RDS database will be created. For the RDS engine type, Amazon Aurora Serverless will be selected with the MySQL compatibility edition. Serverless edition is helpful in this lab as it comes with an integrated query editor that can be accessed directly from the console.
Creating the target database is the first step in this lab as it will take around 10 minutes to provision the required infrastructure.
If Amazon Aurora serverless is being evaluated for a production workload. Ensure that it is being used for the appropriate use case and thoroughly tested at the expected production peak capacity. Aurora Serverless use cases can be found at the following link. aws.amazon.com/rds/aurora/serverless
A cloudformation script will be used to provision the Aurora Serverless cluster. Use the quicklink below to start the cloudformation template in the AWS console. Ensure that the correct region is used.
The following resources will be created via the cloudformation template.
|Region||Launch CloudFormation Template|
|US East (Virginia)||Launch Stack in us-east-1|
|US East (Ohio)||Launch Stack in us-east-2|
|US West (Oregon)||Launch Stack in us-west-2|
|Europe (Frankfurt)||Launch Stack in eu-central-1|
or, download the file to your local workstation and create a CloudFormation stack by uploading the template.
You will see the Quick create stack page as shown below. In the Stack name block, leave the Stack name as
Leave the Parameters block as-is. The default username is
username and the default password is
Keep QLDBStreamRoleName as
QLDBStreamRole. (This role will be used in a future step)
If a custom username and password are used. Then ensure that it is used throughout the lab and know that a minor code change will be required in a future step.
Under Capabilities, click the check box for “I acknowledge that AWS CloudFormation might create IAM resources with custom names” and click on
Now that the resources are being created. Let’s take a look at a few important configurations from the CloudFormation template.
First, in the RDSCluster section, take a look at the EngineVersion, ScalingConfiguration, and then special attention to the EnableHttpEndpoint field. Setting the EnableHttpEndpoint field to
true will enable the AWS console built in query editor to be leveraged.
RDSCluster: Properties: DBClusterParameterGroupName: Ref: RDSDBClusterParameterGroup Engine: aurora-mysql EngineMode: serverless EngineVersion: "5.7.mysql_aurora.2.07.1" ScalingConfiguration: AutoPause: false MinCapacity: 16 MaxCapacity: 32 EnableHttpEndpoint: true MasterUserPassword: Ref: DBPassword MasterUsername: Ref: DBUsername Type: "AWS::RDS::DBCluster"
Now take a look at the RDSDBClusterParameterGroup section. The value for tx_isolation is set to
SERIALIZABLE. In any streaming application, correctly configuring the target database is extremely important.
In this lab, QLDB will have high frequency updates that are intended to end up “out of order” in the Kinesis stream. Using a serializable transaction level will ensure that we can do the appropriate logic against the database and have a QLDB Ledger and target database that are completely synced. Using an incorrect transaction isolation level can cause issues like dirty or skew reads and the result will be a source and target database that are out of sync.
RDSDBClusterParameterGroup: Properties: Description: "CloudFormation Sample Aurora Cluster Parameter Group" Family: "aurora-mysql5.7" Parameters: time_zone: US/Eastern tx_isolation: SERIALIZABLE Type: "AWS::RDS::DBClusterParameterGroup"
There is no need to wait for the template to complete as the build process takes about 10 minutes. Head on over to the next section.