February 28, 2022 API Strategy

How to Monitor API Usage and Performance with the Moesif Plugin for AWS API Gateway

API gateways provide a central point to govern and control access to your APIs, enabling customers and partners to quickly create new experiences. Amazon API Gateway has native support for a variety of compute resources like AWS Lambda or Amazon Elastic Compute Cloud (Amazon EC2).

API Observability

API observability can provide your business and engineering teams with deep insights into how your APIs are used. API observability is leveraged by a variety of teams including:

Product teams to understand API usage and business value
Engineering teams to monitor and troubleshoot API issues
Security teams to detect and protect from API threats

Moesif API Analytics is an API observability solution that you can leverage to better understand API usage. There is a native integration with Amazon API Gateway which makes deployment just a matter of a few clicks and does not require any code change or restarts. As the gateway to the rest of your infrastructure, API gateways are also the natural place to provide API observability to your various business and engineering teams.

Solution overview and use cases

This solution adds API analytics to your APIs hosted behind AWS API Gateway. It works by forwarding structured API access logs from your Amazon API Gateway instance to Moesif via an Amazon Kinesis Data Firehose. Deployment of the solution can be done in a few clicks using the included CloudFormation template and doesn’t require any downtime. Once done, you can achieve a few objectives:

Understanding customer API usage

A key objective for API analytics is understanding who is using your APIs and how they use them. By default, Moesif ties your API calls back to a user identifier through parsing the request context with $context.authorizer.principalId or $context.identity.cognitoIdentityId so you can understand user behavior.

A critical report is understanding which customers are using your APIs the most. APIs logs traditionally don’t include customer demographic information, but an API analytics system like Moesif can automatically join with other data sets containing customer attributes like user email or company domain. Users can be tracked with client integrations like moesif-browser-js or Segment using the same user id. Then we can bring up a usage report showing API traffic by company.

Troubleshoot API issues

With high-cardinality, high-dimension API observability, you can slice and dice your API logs by any number of fields including HTTP headers or response time. This makes it easy to quickly troubleshoot issues without manual log search. A core engineering metric for APIs is latency percentiles such as the 90th percentile. The best practice is to look at 90th percentile latency over the average. This practice helps uncover large variations in your latency that can be masked by low averages. Your API users are looking for consistently low latency not the lowest average as spikes can wreak havoc in their own services.

To do this, go to Events -> Time series and Select the P90 Latency Metric. It’s a good idea to also understand this broken down by route or service. To do so, add a group by “Request URI.” Moesif will automatically consolidate routes such that /items/1 and /items/2 will show up as /items/:id in the UI:

Find API security threats

As you expose more APIs to the internet used by customers, partners, and single page apps, your security risk goes up. Traditional mechanisms like browser fingerprinting and captchas don’t work so you need to leverage advanced user behavior analytics to find suspicious users.

A common API security threat is not limiting access to your proprietary data. A hacker can then download all this data via a pagination attack. A method to detect customers abusing your API this way is to look at the amount of data downloaded per customer. To create this metric, add a summation of response.headers.Content-Length and then group it by customer name:

How to set up

The integration works by adding an Amazon Kinesis Data Firehose which receives API Access logs from your Amazon API Gateway and sends them to Moesif. There are two types of logs from API Gateway logs: API Access Logs and CloudWatch Execution Logs. While execution logs are unstructured but human readable, API access logs are structured and more machine-parsable. In addition, the logs contain the user identity which makes them perfect for user behavior analytics tools like Moesif.

1. Launch CloudFormation Stack

Use the Amazon Web Services CloudFormation template from Moesif to automatically create a Kinesis Data Firehose and configure it to send API access logs to Moesif. To get started, click the launch stack button below.

This will open the Quick create stack within the AWS Console. You will need to enter your real Moesif Application Id under the Parameters section. Your Application Id can be found by signing into your Moesif account which can be done on AWS Marketplace.

2. Enable API Gateway access logging

You will need to enable API Access Logs in Amazon API Gateway and send it to the Kinesis Data Firehose from Step 1.

Go to your AWS API Gateway instance within the AWS Console.
Select Stages on the left menu and then select the Logs/Tracing tab
Toggle on Enable Access Logging.
Add your Firehose ARN from Step 1 under Access Log Destination ARN.

3. Add the JSON log format

Now that you enabled access logs, you need to add the below JSON log format so the output is compatible with Moesif. Moesif will safely ignore any extra keys.

{   "apiId": "$context.apiId",
    "requestId": "$context.requestId", 
    "requestTime": "$context.requestTime", 
    "protocol": "$context.protocol", 
    "httpMethod": "$context.httpMethod",
    "resourcePath": "$context.resourcePath", 
    "requestHostHeader": "$context.domainName",
    "requestUserAgentHeader": "$context.identity.userAgent",
    "ip": "$context.identity.sourceIp", 
    "status": "$context.status",
    "responseLength":"$context.responseLength", 
    "durationMs": "$context.responseLatency",
    "caller": "$context.identity.caller", 
    "user": "$context.identity.user",
    "principalId": "$context.authorizer.principalId",
    "cognitoIdentityId": "$context.identity.cognitoIdentityId",
    "userArn": "$context.identity.userArn",
    "apiKey": "$context.identity.apiKey"
}

4. Success!

With the API Gateway integration done, you should see your API logs show up in Moesif. Make a few calls against your API Gateway domain and see them show up in Moesif’s event log in real-time. You should see the status code, URL, and other HTTP parameters captured like the below screenshot:

Advanced User Behavior Analytics

You can leverage your integration beyond just looking at API calls in isolation and stitch your entire customer journey together. This approach makes it easier to see things like funnel reports on “Time to First Hello World” and “Time to Value.”

Track user actions in your UI such as “Signed In” or “Viewed Docs” and start tracking user actions in your UI like “Signed In” or “Viewed Docs”. This makes it easier to slice and dice API usage by customer traffics. In order to do so, add the moesif-browser-js to your UI and call the track method:

moesif.track('Clicked Sign Up', {
  button_label: 'Get Started',
  sign_up_method: 'Google SSO'
});

Once done, the first thing you should do is generate a funnel report. In the below report, we created a funnel analysis composing of three steps.

The first step is a customer signing into your web application (a user action).
The second step is a single payment transaction via the API. Thus moving from step 1 to step 2 shows the conversion rate of sign ups to the first API call.
The third step is over 100 payment transactions. For this example, we consider this the “aha” moment demonstrating customer value. Moving from step 2 to step 3 shows the drop off of customers who made API calls who actually got to see real value.

Conclusion

Having the right API observability solution can provide your team with the right visibility to make informed decisions. While you can roll your own API gateway, data processing pipeline, and a data warehouse, this can create a massive time sink for your engineering team. Using fully managed services like Amazon API Gateway and Moesif API Analytics can help you scale without being held back by high maintenance costs or outdated data infrastructure. To get started, you can sign up for a free Moesif account right on AWS Marketplace.

Amazon , API Gateways , API Management

Derric Gilling

Co-founder & CEO @Moesif. Previously Computer Architect @Intel. Studied @UMichigan.

API Strategy

Announcing General Availability of the Moesif Developer Portal

We’re thrilled to announce the General Availability of the Moesif Developer Portal, a major leap forward enabling you to deploy a seamless experience for dev...

December 06, 2024

API Analytics and Monitoring

Introducing AI Explain

We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...

November 14, 2024