How development for AWS changes the local setup

When a new project is set up in the cloud these days, this usually means that AWS is used, that the backend is split into multiple services (e.g. microservices), the frontend and backend communicate using REST and multiple managed AWS services are used. Such a setup allows to easily leverage many services AWS provides and reduces the effort and time to implement the desired functionality. To fully use this potential, the development process has to be reviewed and adjusted.

In the old days, a single monolith was developed locally on the developer computer. Dependencies like database, message queue or SMTP server were mostly installed locally or, if this was not possible, mocked or a central installation was used. In most cases the developer could be certain that if the application runs on his computer it will also run on other environments (e.g. test stage or production environment). This approach is not feasible with how applications are developed these days. The differences stem from multiple sources:

• Splitting the application into multiple parts (e.g. microservices) creates a fully distributed application in which the glue which connects the parts is becoming more important. One side effect is, for example, that the glue also has to be tested. Another consequence of a distributed application is that the application becomes unwieldy in the way that multiple programs have to be running to experience the full application. The development team also has to think about distributed failures and aspects like resilience are getting important.
• Using cloud services introduces new APIs and a new way of thinking. So, for example, the database is not just there, but you also have to think about request capacities for your database access.
• Using AWS especially adds the burden of not being able to install the cloud solution on your local computer(s). Other solutions like Kubernetes allow you to do this. While not a real limitation these days, you lose the ability to develop offline. More important is that the development becomes more expensive since these AWS services also have to be paid for the developers, and this sometimes means that the cost will scale with the number of developers.

DevOps can solve the first two issues, but the last one could be described as vendor lock-in.

Let’s take a look at some scenarios and assess them. In the following, “stage” means a self-contained version of the software under development. This could be (for AWS): a set of EC2 instances, an ECS cluster, a unique URL, a set of DynamoDB tables identified by a prefix, a set of CodePipelines and other AWS services.

Use a shared dev stage

In a very easy setup, there is one stage for development. If you want to do a change, you commit and push your change, wait for the build and deployment and then test your change. While it’s easy to get this setup running, it has certain drawbacks:

• The long turnaround times since you have to wait for the build and deployment. This usually takes a few minutes.
• Debugging your services is cumbersome or impossible.
• Since AWS CodePipelines do not support branches, the commit has to be performed in the master. This may be O.K. if the team develops in the master using feature toggles. This, however, prevents other teams to use feature branches and perform code reviews using pull requests.
• Potential problems because of conflicts between developers which use the same code areas and the same stage.

Use personalized stages

One idea to reduce the drawbacks it to create a stage for each developer. This also means the code pipelines have to be duplicated and adjusted to listen to developer-specific branch names. This allows us to review the code with pull requests in addition to developing independently. This is especially important if there are incompatible code changes in the backend services or changes in the state itself. However, there are some existing drawbacks and a new one:

• Long turnaround times
• Debugging is hard or impossible
• Extra costs for each stage

Run application services locally

To improve the turnaround times and allow for debuging of the application services easily, it is possible to run the application services locally. This may mean that only the one service under development is started or that all services are running locally. While it’s easy to run the application services locally, it’s not that easy or even impossible to get the AWS services (i.e. SQS, S3) running on the local machine. So the AWS services of one stage have to be used. While this setup can be combined with one shared dev stage or with the personalized stages mentioned above, the shared dev stage has two disadvantages:

• If there are structural changes to the AWS services or the data stored / transferred extra effort is required to ensure that other developers using the same stage are unaffected. It may not be worth the effort and a personalized stage is the better solution.
• There may be too little control over the execution: if during the test one application service adds a message to a queue (e.g. SQS) and another locally running application service is the consumer of this queue, it is not guaranteed that the locally running consumer will get this message.

Another aspect is how these application services are reached:

• Accessed by the frontend or other application services: This is usually through a load balancer. In such a case a locally running load balancer replacement is required. We have successfully used nginx for this with a script which converts AWS load balancer rules into a nginx config file.
• Activated by AWS services: An example here is SQS consumption. This is a problem with a shared stage but works for personalized stages.

In general this approach is possible and is recommended if the conditions are right (low frequency of structure changes, either no AWS service activation or personalized stages are used).

Running as much as possible local

It is possible to run more, in addition to the load balancer, AWS services locally. This allows to further reduce the dependency for AWS services during development and reduce the costs. Besides DynamoDB which is available for local installation the other AWS services are not available for local installation. One can view a replacement software as locally running AWS service or one can also see it as a mock for the real AWS service. This also depends on how sophisticated the service implementation has to be to successfully run the application on the local computer. In general it also means that the fact that the application works locally doesn’t mean necessarily that it also works on a full AWS stage.

There are replacements for multiple AWS services:

• moto : Originally a mocking library but also allows to start as a separate server.
• localstack : “A fully functional local AWS cloud stack. Develop and test your cloud apps offline!”
• Eucalyptus (doc): complete private cloud implementation which claims to be API compatible with AWS.

Out of these three Eucalyptus looks not active maintained. It also looks like the owner of the backing company changed recently. LocalStack uses moto but aims to provide a more complete solution. There is also the beginning of a commercial company backing LocalStack.

For specific services (an older list ):

• For Kinesis:
  • kinesalite : “An implementation of Amazon’s Kinesis built on LevelDB”
• For S3:
  • minio : “Minio is an open source object storage server with Amazon S3 compatible API. “ It supports multiple underlying storage providers.
  • ceph : “Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability.” It also offers an S3 interface.
  • Fake S3 : “A lightweight server clone of Amazon S3 that simulates most of the commands supported by S3 with minimal dependencies”
• SQS:
  • elasticmq : “Message queueing system with an actor-based Scala and Amazon SQS-compatible interfaces”
  • goaws : “AWS (SQS/SNS) Clone for Development testing”
• Lambda and API gateway:
  • serverless-offline : “Emulate AWS λ and API Gateway locally when developing your Serverless project”
  • emulambda: “Python emulator for AWS Lambda”

Conclusion

There are surprisingly few experience reports about local AWS development. LocalStack, on the other hand, was developed by Atlassian and while there is no official press release it is reasonable to assume that they felt a problem and developed a solution for it.

If money is not a concern, the recommendation is to provide a personalized stage for each developer and run the application services locally. In such a case a stripped down stage without the active application services is enough.

If the number of developers is larger or a stage consists of many costly elements, the recommendation is to try to use replacements which run locally, for example LocalStack. If there is a setup which avoids AWS services, you may also want to consider using this setup in CI/CD. Still plan for the case that the real AWS services may behave differently.