Python and CDK (Part 2): Taking control of Python dependencies in AWS Lambda

In Part 1 of this series, Developing AWS Lambda Functions with Python and CDK, we covered the initial setup of a CDK and Python project. We walked through the process of creating a basic Hello World* Lambda function, testing it with a unit test, defining the function within our CDK stack, deploying it to AWS, and conducting a test using the AWS CLI.

The Hello World Lambda function is a simple example that doesn't require any external dependencies. However, in real-world projects, it's highly likely that we'll have dependencies to manage. To handle our Python dependencies locally, we introduced poetry. However, these dependencies are not readily bundled and uploaded by CDK out of the box.

To address this, AWS introduced a concept called layers. By using layers, we can separate our function code from its dependencies. Essentially, layers are just ZIP files that contain the necessary dependencies. Keep in mind that a single Lambda function can include up to 5 layers. However, it's crucial to be mindful of the quota set by AWS, which limits the unzipped deployment package size to 250 MB. This size restriction includes both the lambda function itself and its associated layers.

With CDK, we can leverage the LayerVersion construct to create layers and associate them with Lambda functions. By default, we are required to pre-package the dependencies for the LayerVersion. Fortunately, there is a convenient construct available that streamlines the process of working with Python layers.

1npm install --save @aws-cdk/aws-lambda-python-alpha

The PythonLayerVersion scans for dependency definitions in the pyproject.toml and poetry.lock files. It exports the non-dev dependencies to a requirements.txt file and packages them into a ZIP file. Additionally, it includes all files in the entry path within the ZIP. However, this approach is not ideal due to the Lambda package size quota. Currently, the bundling parameters only offer an exclude option, which necessitates the explicit exclusion of each file and folder except our dependency definitions.

1import { PythonLayerVersion } from '@aws-cdk/aws-lambda-python-alpha';
2
3const depsLayer = new PythonLayerVersion(this, 'SampleAppDependencyLayer', {
4    layerVersionName: 'sample-app-dependency-layer',
5    entry: '.',
6    compatibleRuntimes: [Runtime.PYTHON_3_9],
7    bundling: {
8        assetExcludes: ['.*', '*.md', '*.json', '*.js', '*.py', 'bin/', 'lib/', 'src/', 'test/', 'cdk.out/', 'node_modules/'],
9    }
10});

Now, we assign the layer to our Lambda function. Once this is done, CDK will handle the automatic bundling and uploading of our Python dependencies during the Stack deployment.

1new Function(this, "HelloLambda", {
2    functionName: 'hello-lambda',
3    runtime: Runtime.PYTHON_3_9,
4    code: Code.fromAsset('src/hello/'),
5    handler: 'hello.handler',
6    layers: [depsLayer]
7});

What's coming up?

• Mocking AWS infrastructure, such a SQS or DynamoDB, for unit tests.
• Developing BDD acceptance tests using the behave framework.

Stay tuned!