March 04, 2022 API Strategy

How to Monitor API Usage and Performance with Tyk API Gateway on EC2 with Moesif

This article provides an introduction to API Observability and how it fits within the overall APIOps Cycles. Then, we will walk through an example of how to successfully deploy and leverage Tyk Gateway and Moesif API Observability on Amazon EC2.

What is API Observability?

API Observability is a key component to properly execute APIOps Cycles and ensure your building something of value for your API users. If you’re not familiar with APIOps Cycles, take a look at this guide which provides an agile framework to quickly build APIs that are business-oriented and serve customer needs.

Traditional monitoring focuses on tracking known unknowns. This means you already know what to measure like Request Per Second or Errors Per Second. While the metric value may be unknown beforehand, you already know what to measure or probe such as a counter to track requests into buckets. This makes it possible to report on the health of a system (like Red, Yellow, Green), but is a bad tool for troubleshooting engineering or business issues which usually require asking arbitrary questions.

API observability was born out of control systems theory to observe as much as possible the internal workings of a system from its outputs by inferring state and behavior. With a sophisticated analytics tool to analyze all this data, you are able to answer any arbitrary question around your API behavior, not just a few predefined metrics that are directly measurable.

API Observability solutions like Moesif are engineered to bridge this gap between monitoring specific predefined metrics vs general observability for the overall health of a product or business. For more info on API observability vs monitoring, see this article.

About Moesif and Tyk

An API gateway like Tyk enables you to have piece of mind scaling and protecting your API portfolio. Written in Go, Tyk’s architecture takes a modular plug-and-play approach enabling you to add the components that fit your requirements. Many of these plugins are provided by Tyk out of the box so you can get up and running with authentication, rate limiting, and cross origin sharing are already handled.

Tyk has plugins for a number of analytics backends whether it’s a DIY build on top of open-source technologies like ElasticSearch or a managed service like Moesif API Observability. Moesif is an observability platform designed specifically for API products and has a close integration with Tyk to collect additional context such as the authenticated user (AliasId or OAuthId) so you’re able to drill into customer usage metrics and understand where they drop off in adopting your APIs.

With theMoesif Tyk plugin, your API logs are sent to Moesif which are analyzed to provide analytics andreports on usage. Moesif also processes payload information so you’re able to understand utilization of specific JSON keys, etc. For more info on how Moesif and Tyk work together, see Tyk + Moesif: the perfect pairing.

Dynamic Sampling and Governance

The Moesif plugin for Tyk pump also supports Moesif’s dynamic sampling and governance features. This makes it easy to sample API traffic based on user behavior rules or regex rules in Moesif so you can save on your Moesif subscription costs. Moesif will still extrapolate the original metrics for accurate reporting. For more info on how this works, see the Moesif infrastructure guide.

How to set up Moesif and Tyk on AWS

How to install the Tyk Gateway on EC2

In this example, we install Tyk using the Marketplace AMI on EC2. Tyk has three options, depending on your requirements:

For this guide, we’ll leverage the high availability (2-node) option, but the process is similar if you want the 3+ Nodes. The PoC version is not recommended for production use.

You can also set up Tyk and Moesif on Amazon Elastic Container Service using the docker compose file available on GitHub if the AMI does not fit your requirements.

Go to the Tyk listing on AWS Marketplace and click the yellow subscribe button at the top right. This will launch the setup process for the AMI. You’ll have to wait a few minutes as Amazon processes your request. Once finished, click the yellow Continue to Configuration at the top right.

You’ll want to confirm the Delivery Method is “Tyk Pro HighAvailability API Management”, select your AWS Region and then click the Continue to Launch button at the top right. You’ll need to confirm everything and the select Launch at the bottom.
On the CloudFormation create stack panel, you’ll want to leave the pre populated settings as is and then select Next.

On the stack details panel, you’ll need to fill in all the parameters including the admin username and password for the various components, subnets, and SSH key. You may need to generate this beforehand if you haven’t done so already. For full details on how to fill this out, see the video on Tyk’s website.
Wait for the CloudFormation stack to be created. This might take some time.

Once finished, you should be able to curl the gateway.

curl http://TYKElasticLoadBalancerALB-1234567879.us-east-1.elb.amazonaws.com/hello

How to add Moesif plugin

Now that you have Tyk API Gateway running. You’ll need to enable the Moesif plugin in Tyk Pump and add your Moesif Application Id.

Tyk Pump is an asynchronous component that pushes your raw analytics data to Moesif for analysis. The plugin fully supports Moesif’s dynamic sampling features which makes it easy for you to save cost.

Sign up for a Free Moesif account. On the onboarding installation page, click Tyk Gateway to bring up the set up instructions and your application\_id.

Now that you have your Moesif application id, SSH into your EC2 instances that was created in the last step. The Tyk Pump should be already up and running.
In your pump.conf, you’ll need to add the Moesif Application Id and configuration you obtained in the last step such that your config looks like so:

"pumps": {
  "moesif": {
    "name": "moesif",
    "meta": {
      "application_id": "Your Moesif Application Id"
    }
  },
}

If you want to log HTTP headers and body, ensure detailed analytics recording is enabled in your tyk.conf file.
With the Tyk Pump configured with your correct Moesif account, you should now restart your Tyk Pump and Tyk Gateway instances.

sudo service tyk-gateway restart

sudo service tyk-pump restart

That’s it! Hit the AWS ELB a few times so Moesif can record some API traffic. You should see API logs show up in your Moesif dashboard.

Using Moesif and Tyk together

Now that you have Moesif and Tyk set up, let’s walk through creating some reports.

Monitoring API Performance

One of the first questions that come to mind is how are my various APIs performing. To bring up this report, you’ll want to log into Moesif, go to Time Series, and then select P90 Latency. This will bring up a metric on 90th percentile latency for all your APIs being tracked via Moesif.

A key component of an API observability system like Moesif is to pivot on any property regardless of cardinality. In this case, let’s add a Group By on Request.Route. This will break down our metric by the RESTful path. Since Moesif is designed for APIs, it’ll automatically consolidate URI identifiers like /items/1 and /items/2 to a more analytics friendly /items/:id. `

Understand Customer API Usage

A key step in the APIOps Cycles is measuring both business and technical KPIs to reach your goals. This means having a deep understanding of who is using your APIs and how they are used. The Moesif Tyk plugin will map a Tyk Token Aliases to a User Id in Moesif. This makes it easy to store user properties such as email, revenue, and other customer demographics.

Install Moesif User Tracking

To do this, install Moesif’s browser-js SDKon your website. Use the same application id for your Tyk Pump configuration:

const moesif = require('moesif-browser-js');

moesif.init({
  applicationId: 'Your Moesif Application Id'
  // add other option here.
});

Then, we can identify the user and save user properties once you have that info:

moesif.identifyUser('12345', {
  email: 'john@acmeinc.com',
  firstName: 'John',
  lastName: 'Doe',
  title: 'Software Engineer',
  salesInfo: {
      stage: 'Customer',
      lifetimeValue: 24000,
      accountOwner: 'mary@contoso.com',
  },
});

While optional, you can also track user actions like “Signed In” and “Viewed Docs” via the moesif.track() method to better understand your customer journey.

Create a usage report

Now that you have saved customer demographics, let’s pull up a report showing top customers by usage broken down by the user’s email. Go to the same Time Series view under Events within Moesif. Then select User.Email so we show the API traffic for each customer. Since we want to look at long-term trends, change the interval to 7-days so you have a report like below:

Concluding Thoughts

Having the right API analytics stack can provide your engineering and business teams with the right visibility to make informed decisions. Moesif and Tyk worked together to provide a turnkey solution for securing and publishing your APIs while gaining a full API observability solution with just a few config changes. A managed API analytics service like Moesif reduces your build and maintenance cost drastically over a custom solution so you can focus on building great APIs.

API Gateways , API Management , Tyk Pump , Tyk

Derric Gilling

Co-founder & CEO @Moesif. Previously Computer Architect @Intel. Studied @UMichigan.

API Strategy

Announcing General Availability of the Moesif Developer Portal

We’re thrilled to announce the General Availability of the Moesif Developer Portal, a major leap forward enabling you to deploy a seamless experience for dev...

December 06, 2024

API Analytics and Monitoring

Introducing AI Explain

We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...

November 14, 2024