In Depth Guide to Serverless APIs with AWS Lambda and AWS API Gateway (Part 2)
TL;DR The repository with an example project can be found on GitHub
This is the last article in a two-part series about building a serverless API with AWS technology.
In the first part, we learned about authentication, request bodies, status codes, CORS and response headers. We set up an AWS SAM project that connected API-Gateway, Lambda, and Cognito so users could sign up and in.
In this article, we will talk about third-party integration, data storage, and retrieval and “calling Lambda functions from Lambda functions” (we will not actually do this, but learn about a method to achieve something similar.
Adding the Image upload
We will start with the image upload. It works as following:
- Request a pre-signed S3 URL via our API
- Upload an image directly to S3 via the pre-signed URL
So we need two new SAM/CloudFormation resources. A Lambda function that generates the pre-signed URL and an S3 bucket for our images.
Let’s update the SAM template:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "A example REST API build with serverless technology"
Globals:
Function:
Runtime: nodejs8.10
Handler: index.handler
Timeout: 30
Tags:
Application: Serverless API
Resources:
ServerlessApi:
Type: AWS::Serverless::Api
Properties:
StageName: Prod
Cors: "'*'"
Auth:
DefaultAuthorizer: CognitoAuthorizer
Authorizers:
CognitoAuthorizer:
UserPoolArn: !GetAtt UserPool.Arn
GatewayResponses:
UNAUTHORIZED:
StatusCode: 401
ResponseParameters:
Headers:
Access-Control-Expose-Headers: "'WWW-Authenticate'"
Access-Control-Allow-Origin: "'*'"
WWW-Authenticate: >-
'Bearer realm="admin"'
# ============================== Auth ==============================
AuthFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/auth/
Environment:
Variables:
USER_POOL_ID: !Ref UserPool
USER_POOL_CLIENT_ID: !Ref UserPoolClient
Events:
Signup:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signup
Method: POST
Auth:
Authorizer: NONE
Signin:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signin
Method: POST
Auth:
Authorizer: NONE
PreSignupFunction:
Type: AWS::Serverless::Function
Properties:
InlineCode: |
exports.handler = async event => {
event.response = { autoConfirmUser: true };
return event;
};
UserPool:
Type: AWS::Cognito::UserPool
Properties:
UserPoolName: ApiUserPool
LambdaConfig:
PreSignUp: !GetAtt PreSignupFunction.Arn
Policies:
PasswordPolicy:
MinimumLength: 6
UserPoolClient:
Type: AWS::Cognito::UserPoolClient
Properties:
UserPoolId: !Ref UserPool
ClientName: ApiUserPoolClient
GenerateSecret: no
LambdaCognitoUserPoolExecutionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt PreSignupFunction.Arn
Principal: cognito-idp.amazonaws.com
SourceArn: !Sub 'arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}'
# ============================== Images ==============================
ImageBucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: PublicRead
CorsConfiguration:
CorsRules:
- AllowedHeaders:
- "*"
AllowedMethods:
- HEAD
- GET
- PUT
- POST
AllowedOrigins:
- "*"
ImageBucketPublicReadPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref ImageBucket
PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Principal: "*"
Resource: !Join ["", ["arn:aws:s3:::", !Ref "ImageBucket", "/*" ]]
ImageFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/images/
Policies:
- AmazonS3FullAccess
Environment:
Variables:
IMAGE_BUCKET_NAME: !Ref ImageBucket
Events:
CreateImage:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /images
Method: POST
Outputs:
ApiUrl:
Description: The target URL of the created API
Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
Export:
Name: ApiUrl
Okay, we ended up with three new resources. We also needed a BucketPolicy that allows public reads on our new image bucket.
The ImagesFunction has an API event so we can handle POST requests with it. The function gets an S3 access policy and an environment variable so it knows the ImageBucket.
We need to create a new file for the function code functions/images/index.js.
const AWS = require("aws-sdk");
exports.handler = async event => {
const userName = event.requestContext.authorizer.claims["cognito:username"];
const fileName = "" + Math.random() + Date.now() + "+" + userName;
const { url, fields } = await createPresignedUploadCredentials(fileName);
return {
statusCode: 201,
body: JSON.stringify({
formConfig: {
uploadUrl: url,
formFields: fields
}
}),
headers: { "Access-Control-Allow-Origin": "*" }
};
};
const s3Client = new AWS.S3();
const createPresignedUploadCredentials = fileName => {
const params = {
Bucket: process.env.IMAGE_BUCKET_NAME,
Fields: { Key: fileName }
};
return new Promise((resolve, reject) =>
s3Client.createPresignedPost(params, (error, result) =>
error ? reject(error) : resolve(result)
)
);
};
So, what happens in this function?
First, we extract the userName from the event object. API-Gateway and Cognito control access to our function, so we can be sure that a user exists in the event object when the function is called.
Next, we create a unique fileName based on a random number, the current time-stamp, and the username.
The createPresignedUploadCredentials helper function will create the pre-signed S3 URL. It will return an object with url and fields attributes.
Our API client has to send a POST request to the url and include all the fields and the file in its body.
Integrate the Image Recognition
Now, we need to integrate the third-party image recognition service.
When an image is uploaded S3 will fire an upload event that will be handled by a lambda function.
This brings us to one of the questions from the beginning of the first article.
How to call a Lambda from a Lambda?
Short answer: You don’t.
Why? While you could call a Lambda directly from a Lambda via the AWS-SDK, it brings a problem. We would have to implement all the things that need to happen if something goes wrong, like retries and such. Also, in some cases, the calling Lambda would have to wait for the called Lambda to finish and we would have to pay for the waiting time too.
So what are the alternatives?
Lambda is an event-based system, our functions can be triggered by different event sources. The main course of action is to use another service to do so.
In our case, we want to call a Lambda when the file upload finished, so we have to use an S3 event as a source. But there are other event sources as well.
- Step Functions lets us coordinate Lambda functions with state machines
- SQS is a queue where we can push our results to so they can be picked up by other Lambdas
- SNS is a service that allows us to fan-out one Lambda result to many other Lambdas in parallel.
We add a new Lambda function to our SAM template that will be called by S3.
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "An example REST API build with serverless technology"
Globals:
Function:
Runtime: nodejs8.10
Handler: index.handler
Timeout: 30
Tags:
Application: Serverless API
Resources:
ServerlessApi:
Type: AWS::Serverless::Api
Properties:
StageName: Prod
Cors: "'*'"
Auth:
DefaultAuthorizer: CognitoAuthorizer
Authorizers:
CognitoAuthorizer:
UserPoolArn: !GetAtt UserPool.Arn
GatewayResponses:
UNAUTHORIZED:
StatusCode: 401
ResponseParameters:
Headers:
Access-Control-Expose-Headers: "'WWW-Authenticate'"
Access-Control-Allow-Origin: "'*'"
WWW-Authenticate: >-
'Bearer realm="admin"'
# ============================== Auth ==============================
AuthFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/auth/
Environment:
Variables:
USER_POOL_ID: !Ref UserPool
USER_POOL_CLIENT_ID: !Ref UserPoolClient
Events:
Signup:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signup
Method: POST
Auth:
Authorizer: NONE
Signin:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signin
Method: POST
Auth:
Authorizer: NONE
PreSignupFunction:
Type: AWS::Serverless::Function
Properties:
InlineCode: |
exports.handler = async event => {
event.response = { autoConfirmUser: true };
return event;
};
UserPool:
Type: AWS::Cognito::UserPool
Properties:
UserPoolName: ApiUserPool
LambdaConfig:
PreSignUp: !GetAtt PreSignupFunction.Arn
Policies:
PasswordPolicy:
MinimumLength: 6
UserPoolClient:
Type: AWS::Cognito::UserPoolClient
Properties:
UserPoolId: !Ref UserPool
ClientName: ApiUserPoolClient
GenerateSecret: no
LambdaCognitoUserPoolExecutionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt PreSignupFunction.Arn
Principal: cognito-idp.amazonaws.com
SourceArn: !Sub 'arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}'
# ============================== Images ==============================
ImageBucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: PublicRead
CorsConfiguration:
CorsRules:
- AllowedHeaders:
- "*"
AllowedMethods:
- HEAD
- GET
- PUT
- POST
AllowedOrigins:
- "*"
ImageBucketPublicReadPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref ImageBucket
PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Principal: "*"
Resource: !Join ["", ["arn:aws:s3:::", !Ref "ImageBucket", "/*" ]]
ImageFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/images/
Policies:
- AmazonS3FullAccess
Environment:
Variables:
IMAGE_BUCKET_NAME: !Ref ImageBucket
Events:
CreateImage:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /images
Method: POST
TagsFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/tags/
Environment:
Variables:
PARAMETER_STORE_CLARIFAI_API_KEY: /serverless-api/CLARIFAI_API_KEY_ENC
Policies:
- AmazonS3ReadOnlyAccess # Managed policy
- Statement: # Inline policy document
- Action: [ 'ssm:GetParameter' ]
Effect: Allow
Resource: '*'
Events:
ExtractTags:
Type: S3
Properties:
Bucket: !Ref ImageBucket
Events: s3:ObjectCreated:*
Outputs:
ApiUrl:
Description: The target URL of the created API
Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
Export:
Name: ApiUrl
We add a new serverless function resource that has an S3 event that is called when a new S3 object was created in our ImageBucket.
The need to call a third-party API, namely the Clarifai API, in our Lambda function brings us to the next question.
How store Credentials for third-party APIs?
There are many ways to do so.
One way is to encrypt the credentials with KMS. We encrypt it via CLI, add the encrypted key to the template.yaml as an environment variable and then, inside the Lambda we use the AWS-SDK to decrypt the credentials before we use them.
Another way is to use the Parameter Store of the AWS Systems Manager. This service allows storing encrypted strings that can be retrieved via the AWS-SDK inside a Lambda. We just need to provide our Lambda with the name that defines where the credentials are stored.
In this example, we are going to use the Parameter Store.
If you haven’t created a Clarifai account, now is the time. They will supply you with an API key we next need to store into the Parameter Store.
aws ssm put-parameter \
--name "/serverless-api/CLARIFAI_API_KEY" \
--type "SecureString" \
--value "<CLARIFAI_API_KEY>"
This command will put the key in the Parameter Store and encrypt it.
Next, we need to tell our Lambda function about the name via an environment variable and give it permission to call getParameter via the AWS-SDK.
Environment:
Variables:
PARAMETER_STORE_CLARIFAI_API_KEY: /serverless-api/CLARIFAI_API_KEY_ENC
Policies:
- Statement:
- Action: [ 'ssm:GetParameter' ]
Effect: Allow
Resource: '*'
Let’s look at the JavaScript side of things, for this, we create a new file functions/tags/index.js.
const AWS = require("aws-sdk");
const Clarifai = require("clarifai");
exports.handler = async event => {
const record = event.Records[0];
const bucketName = record.s3.bucket.name;
const fileName = record.s3.object.key;
const tags = await predict(`https://${bucketName}.s3.amazonaws.com/${fileName}`);
await storeTagsSomewhere({ fileName, tags });
};
const ssm = new AWS.SSM();
const predict = async imageUrl => {
const result = await ssm.getParameter({
Name: process.env.PARAMETER_STORE_CLARIFAI_API_KEY,
WithDecryption: true
}).promise();
const clarifaiApp = new Clarifai.App({
apiKey: result.Parameter.Value
});
const model = await clarifaiApp.models.initModel({
version: "aa7f35c01e0642fda5cf400f543e7c40",
id: Clarifai.GENERAL_MODEL
});
const clarifaiResult = await model.predict(imageUrl);
const tags = clarifaiResult.outputs[0].data.concepts
.filter(concept => concept.value > 0.9)
.map(concept => concept.name);
return tags;
};
The handler is called with an S3 object creation event. There will be only one record, but we could also tell S3 to batch records together.
Then we create an URL for the newly created image and give it to the predict function.
The predict function uses our PARAMETER_STORE_CLARIFAI_API_KEY environment variable to get the name of the parameter inside the Parameter Store. This allows us to change the target parameter without a change to the Lambda code.
We get the API key decrypted and can do the calls to the third-party API. Then we store the tags somewhere.
Listing and deleting tagged Images
Now that we can upload images and they get automatically tagged, the next step would be to list all the images, filter them by tags and delete them if not needed anymore.
Let’s update the SAM template!
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "An example REST API build with serverless technology"
Globals:
Function:
Runtime: nodejs8.10
Handler: index.handler
Timeout: 30
Tags:
Application: Serverless API
Resources:
ServerlessApi:
Type: AWS::Serverless::Api
Properties:
StageName: Prod
Cors: "'*'"
Auth:
DefaultAuthorizer: CognitoAuthorizer
Authorizers:
CognitoAuthorizer:
UserPoolArn: !GetAtt UserPool.Arn
GatewayResponses:
UNAUTHORIZED:
StatusCode: 401
ResponseParameters:
Headers:
Access-Control-Expose-Headers: "'WWW-Authenticate'"
Access-Control-Allow-Origin: "'*'"
WWW-Authenticate: >-
'Bearer realm="admin"'
# ============================== Auth ==============================
AuthFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/auth/
Environment:
Variables:
USER_POOL_ID: !Ref UserPool
USER_POOL_CLIENT_ID: !Ref UserPoolClient
Events:
Signup:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signup
Method: POST
Auth:
Authorizer: NONE
Signin:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signin
Method: POST
Auth:
Authorizer: NONE
PreSignupFunction:
Type: AWS::Serverless::Function
Properties:
InlineCode: |
exports.handler = async event => {
event.response = { autoConfirmUser: true };
return event;
};
UserPool:
Type: AWS::Cognito::UserPool
Properties:
UserPoolName: ApiUserPool
LambdaConfig:
PreSignUp: !GetAtt PreSignupFunction.Arn
Policies:
PasswordPolicy:
MinimumLength: 6
UserPoolClient:
Type: AWS::Cognito::UserPoolClient
Properties:
UserPoolId: !Ref UserPool
ClientName: ApiUserPoolClient
GenerateSecret: no
LambdaCognitoUserPoolExecutionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt PreSignupFunction.Arn
Principal: cognito-idp.amazonaws.com
SourceArn: !Sub 'arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}'
# ============================== Images ==============================
ImageBucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: PublicRead
CorsConfiguration:
CorsRules:
- AllowedHeaders:
- "*"
AllowedMethods:
- HEAD
- GET
- PUT
- POST
AllowedOrigins:
- "*"
ImageBucketPublicReadPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref ImageBucket
PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Principal: "*"
Resource: !Join ["", ["arn:aws:s3:::", !Ref "ImageBucket", "/*" ]]
ImageFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/images/
Policies:
- AmazonS3FullAccess
Environment:
Variables:
IMAGE_BUCKET_NAME: !Ref ImageBucket
Events:
ListImages:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /images
Method: GET
DeleteImage:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /images/{imageId}
Method: DELETE
CreateImage:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /images
Method: POST
TagsFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/tags/
Environment:
Variables:
PARAMETER_STORE_CLARIFAI_API_KEY: /serverless-api/CLARIFAI_API_KEY_ENC
Policies:
- AmazonS3ReadOnlyAccess # Managed policy
- Statement: # Inline policy document
- Action: [ 'ssm:GetParameter' ]
Effect: Allow
Resource: '*'
Events:
ExtractTags:
Type: S3
Properties:
Bucket: !Ref ImageBucket
Events: s3:ObjectCreated:*
Outputs:
ApiUrl:
Description: The target URL of the created API
Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
Export:
Name: ApiUrl
We add some new API events to our ImagesFunction and now need to update the JavaScript for it too.
const AWS = require("aws-sdk");
exports.handler = async event => {
switch (event.httpMethod.toLowerCase()) {
case "post":
return createImage(event);
case "delete":
return deleteImage(event);
default:
return listImages(event);
}
};
const createImage = async event => {
const userName = extractUserName(event);
const fileName = "" + Math.random() + Date.now() + "+" + userName;
const { url, fields } = await createPresignedUploadCredentials(fileName);
return response({
formConfig: {
uploadUrl: url,
formFields: fields
}
}, 201);
};
const deleteImage = async event => {
const { imageId } = event.pathParameters;
await deleteImageSomewhere(imageId);
return response({ message: "Deleted image: " + imageId });
};
// Called with API-GW event
const listImages = async event => {
const { tags } = event.queryStringParameters;
const userName = extractUserName(event);
const images = await loadImagesFromSomewhere(tags.split(","), userName);
return response({ images });
};
// ============================== HELPERS ==============================
const extractUserName = event => event.requestContext.authorizer.claims["cognito:username"];
const response = (data, statusCode = 200) => ({
statusCode,
body: JSON.stringify(data),
headers: { "Access-Control-Allow-Origin": "*" }
});
const s3Client = new AWS.S3();
const createPresignedUploadCredentials = fileName => {
const params = {
Bucket: process.env.IMAGE_BUCKET_NAME,
Fields: { Key: fileName }
};
return new Promise((resolve, reject) =>
s3Client.createPresignedPost(params, (error, result) =>
error ? reject(error) : resolve(result)
)
);
};
The delete function is straight forward, but also answers one of our questions.
How use a Query String or Route Parameters?
const deleteImage = async event => {
const { imageId } = event.pathParameters;
...
};
The API-Gateway event object has a pathParameters attribute that holds all of the parameters we defined in our SAM template for the event. In our case imageId because we defined Path: /images/{imageId}.
A query string used in a similar way.
const listImages = async event => {
const { tags } = event.queryStringParameters;
...
};
The rest of the code isn’t too complicated. We will use the even data to load or delete the images.
The query string will be stored as an object inside the queryStringParameters attribute of our event object.
Conclusion
Sometimes it happens that new technology brings a paradigm shift. Serverless is one of these technologies.
The full power of it boils often down to using managed services whenever possible. Don’t roll your own crypto, authentication, storage, or compute. Use what your cloud-provider has already implemented.
The big problem here is often that there are many ways to do things. There isn’t only one right way. Should we use KMS directly? Should we let the Systems Manager handle things for us? Should we implement auth via Lambda? Should we use Cognito and be done with it?
I hope I could answer the questions that arise when using Lambda to some extent.
Get User-Centric API Monitoring for AWS Lambda
14 day free trial. No credit card required.
Learn More
Related Articles
API Analytics and Monitoring
Introducing AI Explain
We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...
API Analytics and Monitoring
Monitoring Cost and Consumption of AI APIs and Apps
Discover how Moesif can help manage and optimize your AI-driven APIs with powerful observability tools that track costs and usage efficiently.
API Development
Unlock Deep API Insights with Moesif’s HTTP Body Analytics
Learn how to use Moesif to analyze HTTP bodies.
API Analytics and Monitoring
How Moesif API Observability Compares to Treblle
Learn how Moesif’s API observability platform compares to Treblle.