AWS Secrets Caching

On the 18th of October, AWS released a new Lambda extension to simplify and cache secrets and parameters: https://aws.amazon.com/about-aws/whats-new/2022/10/aws-parameters-secrets-lambda-extension/

This extension helps reduce the number of API calls a function needs to make to Secrets Manager and allows values to be cached locally for concurrent executions. Not only does this get around API rate limiting issues, but it can also drive down costs per lambda execution.

It works by hosting a local HTTP server within a Lambda Layer where your functions can make GET requests to retrieve secrets rather than relying on the AWS SDK.

We’re going to run through how to get up and running with this new extension using the Serverless Framework and NodeJS.

First, we will start by creating a new serverless project using the basic starter template:

[MacBook-Pro-3:serverless-extensions paulflowers$ serverless

Creating a new serverless project

? What do you want to make? AWS - Node.is - Starter 
[? What do you want to call this project? secrets-extension

~ Project successfully created in secrets-extension folder

[? Do you want to login/register to Serverless Dashboard? No

[? Do you want to deploy now? No

What next?
Run these commands in the project directory:
yamlhhhhhhh
serverless deploy    Deploy changes 
serverless info      View deployed endpoints and resources             
serverless invoke    Invoke deployed functions 
serverless --help    Discover more commands

When we open this in VScode we will have a directory structure like this:

We will start by adding the AWS Secrets extension as a new layer in ‘serverless.yml’

service: secrets-extension

frameworkVersion: '3'

provider:
  name: aws 
  region: eu-west-1 
  runtime: nodeis14eX

functions:
  hello:
    handler: handler.hello 
    layers:
      -arn:aws:lambda:eu-west-1:015030872274:layer:AWS-Parameters-and-Secrets-Lambda-Extension:2

I am deploying to eu-west-1 so I have used the ARN that corresponds to this region, you can find the relevant ARN for your deployment here https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html#retrieving-secrets_lambda_ARNs

We also need to configure the Lambda IAM role to allow access to secrets manager, I am going to set this quite loose, but you should be more selective in a production environment and scope the role accordingly.

Modify the ‘providers’ section like so:

provider:
  name: aws 
  region: eu-west-1 
  runtime: nodeistAx 
  iam:
    role:
      statements:
        # Allow functions to access all secrets - not for production use
        - Effect: Allow
        Action: 'seccetsmanager:Get*
        Resource: I * I
        - Effect: Allow
        Action: 'secretsmanager :List*'
        Resource:
        - Effect: Allow
        Action: 'secretsmanager :DescribeSecret
        Resource: I * I

We now have a Lambda function defined with Serverless, have attached the Secrets Manager extension and configured the required IAM permissions to fulfill its duties.

Next, we will create a Secret which we will use to test the extension using the AWS CLI:

aws secretsmanager create-secret --name my-secret --secret-string "Thor Love & Thunder wasn't actually that bad"

And now we will configure our Lambda function to grab the secret using the extensions web service and output it to the console.

Update handler.js as follows:

'use strict'

exports.hello = function (event, context, callback) {
  const https = require('http')
    const options = {
      hostname: 'localhost' 
      port: 2773, 
      path: '/secretsmanager/get?secretId-my-secret" 
      method: 'GET'
      headers: {
        'X-Aws-Parameters-Secrets-Token': process. env.AWS_SESSION_TOKEN
      },
  }
  const req = https. request (options, res =› {
    res.on ('data', d => {
      const mySecret = JSON.parse (d) . SecretString;
      console.log (mySecret)
      return d;
    })
  })
  reg.on('error', error => {
      console.error (error)
  })
  reg.end ( )
};

The header:

'X-Aws-Parameters-Secrets-Token': process.env.AWS_ SESSION_ TOKEN

is used to pass the session token from the Lambda functions execution role into the extension so it can authenticate with Secrets Manager.

Now deploy the function with:

[MacBook-Pro-3:secrets-extension paulflowers$ serverless deploy

Deploying secrets-extension to stage dev (eu-west-1)

Service deployed to stack secrets-extension-dev (36s)

functions:
  hello: secrets-extension-dev-hello (1.7 kB)

Finally, we can invoke the function and see the output of the cached secret:

aws lambda invoke  - -function-name "secrets-extension-dev-hello"   --log-type Tail out - -query
"LogResult" - -output text | base64 - - decode

START RequestId: Oaa674ba-9d21-47c8-932d-720a6506ab Version: $LATEST
[AWS Parameters and Secrets Lambda Extension] 2022/10/19 15:55:43 INFO ready to serve traffic
2022-10-19T15:55:43.9727 0aa674ba-9d21-47c8-932d-7e20fa6506ab
INFO  Thor Love & Thunder wasn't actually that bad
END RequestId: Oaa674ba-9d21-47c8-932d-7e20a6506ab
REPORT RequestId: 0aa674ba-9d21-47c8-932d-7e20fa6506ab
Duration: 9.05 ms Billed Duration: 10 ms
Memory Size: 1024 MB
Max Memory Used: 77 MB

And there you have it.