Create AWS Lambda Layers using Cloud 9

A quick and easy solution to add libraries to your Lambda functions

Published in

AWS Tip

5 min readSep 19, 2022

Layers add external packages to your Lambdas (film photography by me)

You can use AWS Lambdas with different runtime environments but the one that we’ll be looking at today is Python. At the time this article came out, the latest Python version that is supported was Python 3.9 so that is the one which is used here.

Why use Lambda Layers?

AWS Lambda functions are great to run server-less code for which you only pay for the compute time. This is ideal for running short programs that are run sporadically and don’t require a dedicated instance. However, the light-weight nature of Lambdas also means that they come with the bare minimum environment and therefore don’t have popular third-party libraries out-of-the-box (for instance Pandas).

Furthermore, the inline code editor in the AWS web console for Lambda is also limited to packages under 3MB so if you include any library larger than that (for comparison, Pandas is more than 30MB if you include its dependencies), you will see the message below making it more painful if you want to test a change quickly.

Message of the code editor when your lambda package is above 3MB

Lambda Layers enable you to overcome these limitations by creating separate “packages” that contain the libraries your Lambda need. This decouples the packaging of the code of your Lambda from its library dependencies–whether they are third-party or your own custom libraries. For example if you are using a CI/CD automated pipeline, changes made to your Lambda’s code don’t need to trigger the re-packaging of all external libraries (which can be time-consuming), but only your Lambda function script.

Layers are also useful when you have many Lambdas that have the same dependencies since a Layer can be shared between many Lambdas. You can even have your own custom modules in a Layer.

Creating a Layer

Packaging dependencies into a Layer can be done in several ways. If you are using Linux, chances are you can compress your libraries directly and upload the .zip archive to the AWS platform. If your Lambda is part of a CI/CD pipeline, then you will probably want to package your Lambda Layer using Docker. Unfortunately, if you are developing on Mac or Windows, you cannot simply compress your libraries and upload the when creating the layer. If you tried, you would probably get a message similar to this when running a Lambda that imports a module included in your layer:

Runtime.ImportModuleError: Unable to import module 'lambda_function': Unable to import required dependencies: numpy

That is because Lambda runs in a Linux environment and therefore the binaries are not compatible with Mac and Windows library binaries. The easiest solution? AWS Cloud 9.

Cloud 9 is an AWS service that gives you access to a built-in terminal with AWS CLI already preconfigured. It also runs on Amazon Linux — the exact same environment that Lambda runs on so you can be sure you won’t have compatibility issues.

Step 1: Create a Cloud 9 Environment

Click on “Create environment” button in the Cloud 9 page of the AWS web console.
Follow the steps to create the instance, and keep the settings to the default values (be sure that the platform is set at Amazon Linux 2).

Example of Cloud 9 environment creation settings

Once you finish the creation steps, you should be redirected to your environment page. It may take a couple of minutes to load. We are only going to use the terminal for our purposes.

Step 2: Install the right Python version

Type the command python -V in the terminal to see which Python version is preinstalled with your environment.

In our case, the instance has python 3.7.10, but since we want to create a Layer for a Lambda running on Python 3.9, we will install another Python version to make sure that we don’t have any compatibility issues.
Fetch the Python version you want using the following command (a list of the Python releases available can be found here):

wget https://www.python.org/ftp/python/3.9.10/Python-3.9.10.tgz

Extract the file.

tar xvf Python-3.9.10.tgz

Build and install an alternative Python version (these commands can take a few minutes to run).

cd Python-*/
./configure --enable-optimizations
sudo make altinstall
cd ..

Now if you run the command python3.9 -V you should have the correct version installed.

Step 3: Package your libraries

Now that we have Python all set up, we can start packaging the libraries we want. We’ll use the python build-in venv module to create a lightweight python environment with only the libraries we want to include in our package.

Create a new folder.

mkdir packaging
cd packaging

Create a venv environment.

python3.9 -m venv layer_package

Activate your venv.

source layer_package/bin/activate

At this point if you try python -V you’ll see that the Python version we installed is the default one.
You can now install all the libraries you want your Lambda Layer to have.

pip install pandas pyyaml fuzzywuzzy

If you have a requirements file, you could also upload that file (File > Upload Local Files…) to your Cloud 9 environment and use pip install -r requirements.txt.
Deactivate your venv.

deactivate

Compress the libraries into a file archive.

mkdir python
cp -r layer_package/lib/python3.9/site-packages/* python/
zip -r lambda_layer.zip python

You can upload your own module (File > Upload Local Files…) and add it to the archive.

zip -gr lambda_layer.zip ../custom_module/

Step 4: Create your new Lambda Layer

aws lambda publish-layer-version --layer-name my_lambda_layer --zip-file fileb://lambda_layer.zip --compatible-runtimes python3.9

Step 5: Teardown

If you tested that everything works with your Layer and don’t think that you are going to need to make a new one soon, shutdown your Cloud 9 environment to avoid unnecessary costs.