DevOps Event Driven Architecture — Part 3

Overview 🔍

With our foundational AWS resources provisioned in Lab 2 (DynamoDB, Kinesis Stream, S3), Lab 3 focuses on building the pipeline that captures the "Like" event and processes it. We'll create the API endpoint, the Lambda functions to handle ingestion and updates, and connect everything via the Kinesis stream. We will also set up the archival path using Kinesis Firehose.

Purpose 🎯

The purpose of this lab is to:

• Build the event ingestion and processing flow of our EDA.
• Implement serverless functions (AWS Lambda) for specific tasks:
  • IngestLikeLambda: Receives the request from the API, validates it, and publishes the event to Kinesis.
  • UpdateCountLambda: Triggered by Kinesis, processes like events, and updates the count in DynamoDB.
• Configure API Gateway to provide an HTTP endpoint (POST /likes) that triggers the IngestLikeLambda.
• Connect the Kinesis Data Stream to trigger the UpdateCountLambda using an Event Source Mapping.
• Set up Kinesis Data Firehose to read from the Kinesis Stream and automatically archive batched data to the S3 bucket.
• Define necessary IAM Roles and Policies granting permissions for services to interact.
• Deploy the entire ingestion stack using Terraform.
• Obtain the API endpoint URL for the POST /likes request.

Lab Environment 🛠️

You'll continue using:

• AWS Account, Terraform CLI, AWS CLI.
• Code editor and Git.
• Potentially tools specific to your Lambda runtime (e.g., Node.js/npm or Python/pip) to package dependencies if any.

What We'll Accomplish ✅ In this lab, you will:

• Develop the Python code for IngestLikeLambda and UpdateCountLambda.
• Define the Lambda function resources in Terraform, including code packaging and IAM role references .
• Define the Kinesis Data Stream Event Source Mapping to trigger UpdateCountLambda.
• Define the Kinesis Data Firehose delivery stream (Kinesis Stream -> S3).
• Create the API Gateway REST API resources (/likes resource, POST method, Lambda integration).
• Define the API Gateway deployment and stage.
• Define Terraform outputs for the API Gateway invoke URL.
• Run terraform apply to deploy the Lambda functions, API Gateway, and integrations.
• Retrieve the POST /likes API endpoint URL from Terraform outputs 👀.
• Integration Step: Update your UI application's configuration to use the new endpoint URL.
• Push your updated Terraform code and Lambda code to your GitHub repository.
• Test the end-to-end flow by clicking "Like" in the UI and verifying results.

Bringing the Architecture to Life 🚀

This lab is where the core EDA pattern emerges. The IngestLikeLambda simply accepts the event and quickly publishes it to Kinesis, decoupling the initial request from the actual database update. Kinesis acts as a durable buffer. The UpdateCountLambda consumes events independently, ensuring that even if the update process is slow or fails temporarily, the "Like" is captured and can be retried. Firehose provides a seamless, managed way to archive the raw data without extra coding.

You now have a functioning, asynchronous pipeline to handle movie likes!