As I was learning the AWS Serverless paradigm I had some difficulties figuring out how to effectively develop, test and debug Lambda applications locally and deploy to AWS without issues. I was using the SAM (Serverless Application Model) template which made life pretty good when putting together a moderately complex API.

For most of my API’s I have a set of basic requirements:

  1. Ability to develop a full Lambda API locally (that works with API Gateway using the proxy integration).
  2. Share common code between functions.
  3. Run local tests across the board with and without using SAM scripts.
  4. Run the API locally and navigate as needed.

I was able to achieve items #1, #3 and #4 to some degree of success but #2 stumped me for a while.

Shared Code

This is a very common requirements and frankly if you’re not sharing common code within a Serverless application / api then you probably need to review.

I landed on a Services Pattern where each lambda function handles a few tasks related to a single data model (e.g. user, account, company etc.). A handler function determines which action to take based on the method and path in the event source object and makes the necessary call.

A job that every handler object (and frankly, every lambda function) needs is the ability to send back a response in the format that API Gateway is looking for. Having one way of doing this ensures consistency and a great candidate for a common function. The challenge is how to actually share the code efficiently (without copying into each Lambda function).

My Solution

I’m sure there are many different ways to implement this but my focus was having a solution that works not only for a single api but any/all I develop. Some code is just useful across the board and I didn’t just want to find a solution that works for one application but would then require me to copy shared code over again. The solution I found was actually quite simple (once I wrapped my head around layers). It boils down to the following steps:

1. Common Code Package

I decided to create a separate python package containing all the useful classes and functions that Serverless functions might find useful. Over time I can update it and deploy as needed.

I setup a new Python project and added all the common classes/functions to it. Also added the necessary files needed to make it a package that can be installed into other programs and housed it in a private GitHub repository. You can find a public version of it here.

2. Create Lambda Layer

Lambda Layers are really quite simple and pretty powerful. The trick was figuring out how to make them work locally and at the same time work within the SAM workflow. Turns out it was pretty easy 🙂

I created a separate post on creating a Lambda Layer so you can read more detail there. The punchline is that I installed the common code package into a Lambda Layer that can be shared. You can include any other common packages in one or more layers but just know a Lambda function can only reference up to 5 layers.

3. Add Layer Reference in template.yaml

In order to let SAM know to use a layer simply add a line to your template.yaml file pointing to the arn of the layer (in AWS).

I like to add it to the globals section so I know that every function I create will have it available. If you’re only referencing common code (or other package) in one or two functions you should remove it from globals and just add it to the relevant functions directly.

For adding to globals, the template.yaml looks something like this:

Globals:
    Function:
        Runtime: python3.8
        Timeout: 8
        Layers:
            - arn:aws:lambda:us-west-2:123456789:layer:common_python_lambda_functions:1

When you run testing via SAM scripts or run the api locally it’ll be pulling from the layer.

4. Install Package(s) in VirtualEnv

A layer is all well and good but when developing locally (outside of the SAM templates) it won’t know anything about the layer. Your IDE will also complain that it can’t find the package. The simple solution is to just install the same packages found in the layer into your virtual environment. That way you have everything you need to develop efficiently.

In the project where I’m developing the Serverless API install the common function package into the virtual environment so it’s available while coding. This is only used for local development but a key step if you’re just running local tests outside of the SAM build/test infrastructure (which I like to do).

$ pip install git+https://github.com/depster26/public-lambda-helpers

In Conclusion

As mentioned at the beginning, I’m not sure if this is the best approach but it’s one that solves my development criteria:

  • for larger applications, I want a single spot for common code rather than copy into each Lambda function.
  • using a Layer keeps the deployment packages smaller and focused on the task at hand
  • can still develop locally and test parts of the system without relying on SAM scripts. Still need both, but for quick unit tests it streamlines the process. Well, for me anyway!

Sound off below if you think I’m crazy and/or have better solutions for this.

Leave a comment