In this blog post we will focus on creating the Lambdas that comprise the execution part of our application landscape. Our Lambdas will read from S3, transform data, and store this into the RDS instance we created in part 3 of our blog series.
By the way, if you are curious about reducing VPC costs with Lambda, read more about reducing costs in VPC.
Recap
In Part 1, 2 and 3 of this blog series, we described how to create a custom VPC including security groups and subnets, with S3 buckets, and an RDS database. These first steps represent our infrastructure that is the foundation for our new architectural setup with AWS CDK using TypeScript. If you are just beginning to use AWS CDK and want to know how to get started, we recommend you start reading Part 1. This blog post is part four of our six-part blog series on AWS CDK:
- How to create a custom VPC
- How to create S3 buckets
- How to create an RS instance
- How to create Lambdas
- How to create a step function
- AWS CDK Part 6: Lessons learned
From this point onward, we assume you have completed everything discussed in the previous parts, that everything is compiling and you successfully deployed the VPC, S3 bucket and RDS instance to your AWS account.
The Lambda for our Serverless application
Now that the infrastructure is set up, we will start with the application side of our project. We will add our Lambdas to our stack.
#!/usr/bin/env node
import 'source-map-support/register';
import cdk = require('@aws-cdk/core');
import {VpcStack} from "../lib/vpc-stack";
import {S3Stack} from "../lib/s3-stack";
import {ApplicationStack} from "../lib/application-stack";
import {RDSStack} from "../lib/rds-stack";
const app = new cdk.App();
const vpcStack = new VpcStack(app, 'VpcStack');
new S3Stack(app, 'S3Stack', {
vpc: vpcStack.vpc,
subnetName: vpcStack._subnetName
});
const rdsStack = new RDSStack(app, 'RDSStack', {
vpc: vpcStack.vpc
});
new ApplicationStack(app, 'ApplicationStack', {
vpc: vpcStack.vpc,
inboundDbAccessSecurityGroup: rdsStack.mySQLRDSInstance.connections.securityGroups[0].securityGroupId,
rdsEndpoint: rdsStack.mySQLRDSInstance.dbInstanceEndpointAddress,
rdsDbUser: rdsStack.dbUser,
rdsDb: rdsStack.dbSchema,
rdsPort: rdsStack.dbPort,
});
app.synth();
Now we create the file ./lib/application-stack.ts containing the following code:
//lambda-stack.ts
import cdk = require('@aws-cdk/core');
import s3 = require('@aws-cdk/aws-s3');
import lambda = require('@aws-cdk/aws-lambda');
import {Duration, StackProps} from '@aws-cdk/core';
import {LambdaDestination} from '@aws-cdk/aws-s3-notifications';
import {Bucket} from "@aws-cdk/aws-s3";
export class LambdaStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props: ApplicationStackProps) {
super(scope, id, props);
// create a lambda
// get the bucket
// add an s3 notification to the lambda
}
// helper functions
}
We will create a simple Lambda which will be triggered upon an S3 file upload. The first Lambda will be running in a default non-VPC-bound environment.
//create a lambda
const stepFunctionTrigger = new lambda.Function(this, 'stepFunctionTrigger', {
functionName: 'stepFunctionTrigger', //overwrites the default generated one
runtime: lambda.Runtime.NODEJS_10_X,
handler: 'triggers/stepFunction.trigger', // links to a file inside the code artifact below
code: lambda.Code.fromAsset('../lambdas/deployment'),
});
In this example, we can see some recognisable features of a Lambda function such as runtime, handler, and code. All of these are mandatory when setting up your Lambda in the console (or any other framework for that matter). Common properties, like memory or timeout, can all be added additionally if needed.
The S3 Bucket
As mentioned before, we would like this Lambda to be triggered upon the firing of an S3 event. At the time of writing, however, many versions we tried of AWS CDK are buggy when it comes to programatically adding an S3 event trigger.
Until now we just scripted our infrastructure top down. Now we will start to encapsulate some CDK execution in functions as well to demonstrate the handiness when using programmatically defined infrastructure. Using functions improves visibility and enables re-use.
//get the bucket
const s3LambdaTriggeringBucket = this.bucket();
// add an s3 notification to the lambda
this.addBucketNotificationToLambda(stepFunctionTrigger, s3LambdaTriggeringBucket);
Unfortunately we need to perform a CDK hack / workaround so CDK will generate the correct CloudFormation scripts to actually hook up the S3 events to the Lambda.
These functions are implemented as follows.
private bucket() {
return new s3.Bucket(this, 'cdk-file-sync-bucket', {
versioned: false,
bucketName: 'cdk-file-sync-bucket',
encryption: s3.BucketEncryption.KMS_MANAGED,
publicReadAccess: false});
}
private addBucketNotificationToLambda(lambda: lambda.IFunction, s3Bucket: Bucket) {
s3Bucket.addObjectCreatedNotification(new LambdaDestination(lambda), {suffix: '.json'});
// Code below is a hack too actually make the addObjectCreatedNotification call work. From: https://github.com/aws/aws-cdk/issues/3318#issuecomment-532275430
const logicalId = 'BucketNotificationsHandler050a0587b7544547bf325f094a3db834';
const notificationsResourceHandler = this.node.findChild(logicalId);
const customNotificationsResource = s3Bucket.node
.findChild('Notifications')
.node.findChild('Resource');
customNotificationsResource.node.addDependency(notificationsResourceHandler.node.findChild('Role'));
}
Hopefully in time it will just be a call to addObjectCreatedNotification’ instead of hardwiring the complete notificationResourceHandler.
Packaging the Lambas
Next to the CDK infrastructure, we of course need our actual Lambdas which we want to deploy. We will add a directory containing the Lambda sources (written in TypeScript). We will keep this setup rather simple, to keep focus on the learnings about CDK. To get a full picture, this snippet shows our directory structure.
1// Directory structure for part 4: 2. 3├── bin 4│ └── part4.ts 5├── cdk.json 6├── lambdas 7│ ├── package.json 8│ ├── src 9│ │ └── triggers 10│ │ ├── stepFunction.ts 11│ │ └── vpcProcessing.ts 12│ └── tsconfig.json 13├── lib 14│ ├── application-stack.ts 15│ ├── s3-stack.ts 16│ └── vpc-stack.ts 17├── package-lock.json 18├── package.json 19└── tsconfig.json
There is a separate package.json inside the Lambdas folder which contains all necessary dependencies for our Lambdas and two commands needed to package the Lambda build-ts and package.cdk. We need to package these Lambdas before we are able to deploy them.
npm i && npm run build-ts && npm run package-cdk
The result of this is a new folder inside the Lambdas directory called ‘deployment’, which contains the production dependencies (node_modules) and the .js Lambdas needed in our deploy.
VPC-bound Lambdas
In our project, we need a handful of Lambdas to access our RDS instance from part 3. In order to do so, we need our Lambda stack to know about our VPC and assign our VPC to the Lambdas. We can achieve this by adding the following properties to our ApplicationStackProps:
interface ApplicationStackProps extends cdk.StackProps {
vpc: Vpc;
inboundDbAccessSecurityGroup: string
rdsEndpoint: string
rdsDbUser: string
rdsDb: string
rdsPort: string,
}
So we can add the Lambda, we need the Lambda to access the S3 bucket and we need to retrieve the password for RDS.
const s3AccessRole = new PolicyStatement({
effect: Effect.ALLOW,
actions: ['s3:*'],
resources: [s3LambdaTriggeringBucket.bucketArn+'/*']
});
const secret = Secret.fromSecretAttributes(this, rdsPassword, {
secretArn: 'arn:aws:secretsmanager:<region>:<organisationId>:secret:ImportedSecret-fPNc0O',
});
const rdsLambda = ApplicationStack.createVpcLambda(this, 'rdsLambda', 'triggers/vpcProcessing.trigger', props, secret, s3AccessRole);
static createVpcLambda(context: any, name: string, handler: string, props: ApplicationStackProps, secret: ISecret, policies: PolicyStatement): lambda.Function {
const newLambda = new lambda.Function(context, name, {
functionName: name,
runtime: lambda.Runtime.NODEJS_10_X,
handler: handler,
code: lambda.Code.fromAsset('../lambdas/deployment'),
timeout: Duration.seconds(350),
memorySize: 1024,
vpc: props.vpc,
vpcSubnets: {subnetType: SubnetType.ISOLATED},
securityGroup: SecurityGroup.fromSecurityGroupId(context, 'inboundDbAccessSecurityGroup' + name, props.inboundDbAccessSecurityGroup),
environment: {
USERNAME: props.rdsDbUser,
ENDPOINT: props.rdsEndpoint,
DATABASE: props.rdsDb,
PORT: props.rdsPort,
PASSWORD: secret.secretValue.toString()
}
});
newLambda.addToRolePolicy(policies)
return newLambda;
}
The most distinguishing features here are vpc, vpcSubnets, securityGroups attributes of the Lambda. Whenever you define a VPC, you need to assign a strategy for selecting subnets in which the Lambda should be available. For security groups, we dynamically add a label like this (‘inboundDbAccessSecurityGroup’ + name), because it needs to be unique and I like to keep things descriptive.
Some additional interesting Lambda properties are used in this example as well. We use the environment attribute, which accepts a map-like structure to add some CDK properties to the NodeJS execution environment parameters. Additionally we added memorySize and timeout to explicitly restrict the Lambda execution.
Meanwhile, let us check if our new setup actually compiles to an updated Cloudformation template. Run the following commands:
npm run build && cdk synth
The console output should log that the stack was synthesized successfully. At this point you could deploy your new stack. Yet, we will code a few more things before actually starting a deployment to AWS.
Final build & deploy
We are all set up and ready to deploy our new CDK stack to our AWS cloud. In Part 1 we have already set up our credentials, so this time we can build and deploy by simply running the following commands:
npm run build && cdk synth
After successfully having synthesized the Cloudformation template, you can comfortably check what changed by running the command:
cdk diff --profile sample
Finally, we deploy the changes made to the AWS cloud by running the command:
cdk deploy --profile sample
Upon signing in to AWS Cloudformation you should see the application stack being created. We are now a step closer to reaching our architecture. We have discussed how to add Lambdas with S3 triggers and set up Lambdas in a restricted VPC. Next up is finishing our setup with a step function orchestration to actually make everything work together.
Was this post helpful?
More articles
fromKevin van & Maik Kingma
31.12.2019 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog authors
Kevin van
Do you still have questions? Just send me a message.
Maik Kingma
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.