Improving the Lambda developer experience

From a developer’s perspective, running Lambdas as a runtime to serve your main business logic is a breeze. If you are a dev and have embraced the operational side of things, you will have noticed it’s not an easy task. In general developing software is hard enough, and running it properly is quite an additional challenge. The serverless trend, and in particular Lambda functions as your main source of business-logic execution promise to keep devs focussed on creating (dev) stuff while running (ops) seamlessly. Personally, I have seen AWS Lambda living up to the promise of creating a nice ops experience (and Firebase functions as well). However, the developer experience takes some getting used to.

In this post I will share some of my lessons learned, and practices that improved my personal developer experience. They helped me a lot, and I hope they can do the same for you.

Emphasis on top and bottom of the testing pyramid

Yes, let’s start with the thing that needed the most adjustment. I am a big fan of automated validation of high coding standards. If we take the traditional testing pyramid, there are layers of tests which differ in running speed per test, degree of coverage and representativeness of a test result. Unit tests, integration tests, contract tests, UI tests, and end-to-end tests together make up an extensive array of proofs about the likelihood the software will deliver as promised. In conventional web software development (e.g. Java Spring projects), this works pretty well. After some years you will have some gut feeling about the right test mix.

For Lambda I have found that the stuff in the middle is not really offering me a lot of certainty. Each additional test should improve the likelihood of the software delivering the value in production. And tests come at a huge cost, so it should be worthwhile. Developing, maintaining, and running them will take a big chunk of your budget, so it should be a deliberate choice to do so.

In practice, your Lambda will likely use a lot of other AWS features, like SQS for queueing, S3 for flat file storage, Dynamo or RDS for storage, SNS for email etc. This makes integration testing really hard. Localstack is a popular local Docker-compatible replacement. You could create integration tests that run locally (or in a pipeline) without touching the cloud. However, I have found that in many of these cases, you are testing the wrong thing. When you are testing that your Lambda is writing properly to Dynamo, you are using a stunt-double from Localstack, you are still not testing the right API. You are testing Localstack integration, which is not that relevant. But these tests take lots of time to set up, lots of time to run, can be flaky due to the complexity of such a setup, and the results are not all that special. I found my favorite mix to be a huge emphasis on unit tests combined with some end-to-end coverage.

Emphasis on great quality source code (linting, typing, unit testing, mutation testing)

Having a complicated cloud for a runtime can make testing the execution tedious. So I would advise to invest heavily in unit tests. I use TypeScript for Lambdas running on Node in AWS, because it supports typing. Typed source code is a great communicator to your future self or your team members, but also provides lots of opportunity to be a bit rigid on linting. Proper linting makes for clean code and easier/faster code reviews/merges. In order to keep Lambdas testable, it’s advisable to delegate the processing to other classes – because you get it for free when you start with Lambda and loads of examples use this. Many people use the  handler.js (or ts) for their main business logic. However, testability can be poor. I am used to software with dependency injection which will steer you in a testable direction. I would advise to only bootstrap your classes in the handler and split the logic into testable classes.

For these classes, I aim for 100% test coverage if sensible. I use Jest for running the test, but any similar framework will do. 100% might seem like a bit much to most people, but if you separate your code properly, it really doesn’t take a lot of effort. When you are mocking, take enough time to create proper mocks which mimic the API results of your dependencies.

In order to test the strength of the unit test, I use mutation testing. I use the Stryker library, which is available for both JavaScript and TypeScript. It will present you a report of your test strength so you can assess the quality of the unit tests. I consider a mutation score of >70% a good mix of productivity and strength.

Making complex business software in a productive manner will probably force you to use third party libraries in your project. You naturally want to manage those properly. I found that using NPM for package management i.c.w. Yarn works really well. Commands like ‘yarn audit’ and ‘yarn outdated’ will keep your dependencies up to date.

Emphasis on end-to-end testing infrastructure for Lambdas

Many Lambdas interact with cloud services which can be hard to test. However, infrastructure automation has made it really easy to isolate your test runtime. I use Terraform for different Lambda projects, as it provides complete fine-grained control of (almost) all the services you will ever need in the cloud. Frameworks like ‘Serverless Framework’ or ‘SAM’ are too limited if you need to manipulate non-serverless resources. Terraform will provide you with lifecycle options to create new infrastructure, and to destroy what’s there in seconds. Also read my evaluation of Terraform, AWS CDK, Serverless Framework in this post.

I use the following setup. There will be one repository that hosts some general infrastructure configuration (like configuring billing, and creating a CICD account with proper access policies). Lambda gets Terraform scripts for actual acceptance and production deploys, but also for testing deploys. Terraform supports basic flow constructs like loops and conditions to generate a test context. Things like ‘random_string also help to keep tests isolated. At the end of your test suite, you can tear down the complete cloud setup using a Terraform destroy.

The right Lambda developer experience

So with the help of some infrastructure automation combined with high coding standards, developing business applications with Lambda became loads of fun and the most productive development experience in my career until now!