In this lab, you will learn how to use Amazon QLDB streaming in a simulated real world example.
This lab guides you through steps and fundamental concepts to implement a QLDB streaming scenario. To simulate the real world example, continuous data will be loaded into a QLDB ledger via INSERTS, UPDATES, and DELETES. The objective of this lab is to replicate the QLDB user view to a downstream MySQL database. Out of order processing, duplicate messages, stream / processing failures, and other concepts are discussed in this lab. The theme in this scenario will continue with the department of motor vehicles (DMV) application example.
This lab focuses on the architectural concepts of inserting data records from Kinesis streams into a target database at a simulated production scale. Language specific development concepts will not be covered in depth for this lab.
To complete this lab, you must have access to an AWS account and sufficient privileges to administer QLDB ledgers. See Accessing Amazon QLDB for more info.
This lab is assuming that you have completed the QLDB fundamentals lab. If you haven’t, head over to the lab and go through it by clicking here. It will help you understand why streaming QLDB to another purpose built database can be a useful architectural decision.
This lab uses python as the developing language. However, there is no developing experience needed to run this lab. The architectural patterns referenced in this lab can be done with any of the supporting QLDB supported development languages.
Kineses is the underlying service that is used by QLDB streams.
Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.
Building within the the limits of Kineses is extremely important in having a successful QLDB streams implementation.
Spend some time to get comfortable with the Kineses developer guide of save the link for future reference. docs.aws.amazon.com/streams/latest/dev/introduction.html
In this workshop, Amazon Aurora will be used as the target database from QLDB streams.
Amazon Aurora is a MySQL and PostgreSQL-compatible relational database built for the cloud, that combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases.
Amazon Aurora is up to five times faster than standard MySQL databases and three times faster than standard PostgreSQL databases. It provides the security, availability, and reliability of commercial databases at 1/10th the cost. Amazon Aurora is fully managed by Amazon Relational Database Service (RDS), which automates time-consuming administration tasks like hardware provisioning, database setup, patching, and backups.
Amazon Aurora features a distributed, fault-tolerant, self-healing storage system that auto-scales up to 64TB per database instance. It delivers high performance and availability with up to 15 low-latency read replicas, point-in-time recovery, continuous backup to Amazon S3, and replication across three Availability Zones (AZs).
Amazon Aurora will make a great fit for QLDB in low or high TPS applications.
When adding an additional purpose built database to an existing architecture. It is important to consider the characteristics and limitations of every database.
Spend some time to get comfortable with the Amazon Aurora developer guide of save the link for future reference. docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html
If using RDS in a micro-service type architecture, take a look at managing connections with Amazon RDS Proxy - docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy.html