October 19, 2022 API Analytics and Monitoring

Mastering API Analytics for API Programs: The Developer Funnel

You have an API program that developers are adopting, but unsure how much. How long does it take for a new integration to move to revenue generating? If you came from a web or mobile product management background, you may already be familiar with mobile product analytics to measure app engagement and retention. Growing APIs have similar KPIs to measure the success of your API program. This article will talk more about what you should be measuring and how to leverage that information.

The Developer Funnel

While traditional customer funnels will consist of just a marketing and sales funnel component. However, APIs as a product where customers and partners include developers have what is called the developer funnel or integration funnel. The developer funnel is after the marketing funnel and before the sales funnel and has three core stages:

Pre-integration stage

Developers enter the pre-integration stage after the marketing funnel. While there is a blur for what is considered marketing vs developer funnel, intent is the main factor. Are they just considering a solution through well-defined marketing assets like case studies, content, and demo accounts or committed to testing the solution themselves.

Once the evaluation or consideration phase is positive, there is intent to test. This causes the developer to start the installation which can be seen by creating a blank workspace, downloading a configuration file, or copying an API key.

This implies your product is freemium, has a free trial, or other.

Sandbox stage

A potential customer enters the sandbox stage as soon as a single action is taken on your API that gives the developer a warm feeling that they accomplished something and are able to verify this is a viable solution for them. The time to go from pre-integration stage to the sandbox stage is usually measured as Time to First Hello World (TTFHW). This is a critical KPI that every API product manager should track and reduce as much as possible.

Developers may not move into the sandbox stage for many reasons including:

Integration issues such as errors and bugs in SDKs or implementation

For example, if the developer received 500 errors from your API or the SDK didn’t make any calls to your API. The later is one of the hardest to measure since the developer completed the implementation but no actions were captured on your side.
No easy way to verify that the implementation is correct

If your product looks empty even after integration, a developer may assume it doesn’t work. This can happen for security and analytics companies where there are no meaningful insights until a large amount of data is processed.
An action was taken on API but unclear what the value is

The action should have a clear value and not just a message saying “You’re integrated.” For a communications API, that value may be sending a single SMS. For an analytics company, it may be capturing and displaying a single event.

A developer enters the sandbox stage when the integration is successful and there is a clear sign of value, thus creating your aha moment. These developers start to become an internal advocate for your solution.

Production stage

Once a developer sees value in a solution, the developer wants to push their implementation to production as soon as possible. However, the developer may not be able to push your API to production due to internal blockers, many outside the control of the individual developer.

Developers may not move into the production stage for many reasons including:

Project priorities

Customer bugs or other project features may take priority. Your communications API integration took a back seat just because the developer had a pile of Jira tickets assigned to him or her.
Legal and compliance

Especially in regulated industries such as financial and health, the developer may have to wait until necessary approvals from legal.
Functional and performance testing

The company may have internal policies for testing third-party services which need to be finished first.

Your goal is to empower the developer with the correct tools to eliminate these objections. Once a developer moves into the production stage, there is a measurable amount of traffic going to your API from that account. The time it takes to get to production level traffic can be called Time To First Working App (TTFWA) or Time To First Paid App (TTFPA).

Creating Funnel Goals

In order to have an accurate snapshot of your API business, each funnel boundary requires well-defined goals. Let’s use Algolia, a search API, as an example.

Pre-integration step goals

The first step for Algolia will be when a developer uses the API key copy feature. At this point, the developer has already signed up, and is going through the onboarding flow.

Sandbox step goals

Algolia has two core features around their API. A query or search API and an index API. Algolia’s success metric would be developers who were able to index at least one entity via the index API. If the developer called the query API when no data has been stored in Algolia, that is not considered a success as the query will be empty.

In order to keep their integration simple, they can either:

Drive users to index a new entity via POST /1/indexes/ endpoint
Use sample data to bootstrap a sample index that the developer can query

Let’s say we go with (1), then our funnel conversion point would be any developers who performed at least one POST /v1/indexes operation. Algolia may want a tighter conversion metric so they may track any developer who performed at least one POST /v1/indexes operation AND one POST /1/indexes/{indexName}/query operation WHERE response Content-Length > 0. This means the developer successfully indexed at least a single entity and was able to read it back.

Production step goals

Creating a metric for what is considered production-level traffic can be hard as it could be industry or project dependent. For Algolia, an e-commerce app has different levels of production use than an internal enterprise app with lots of dynamic content. For the enterprise app with dynamic content, each entity could be reindexed tens or hundreds of times a day programmatically with updated data. However, if the data is never queried, then the value of Algolia hasn’t been seen yet. So just looking for developers with more than 1,000 POST /1/indexes/ operations may not be the best value metric to track. A better value metric might be developers who have more than 1,000 POST /1/indexes/{indexName}/query operations WHERE response Content-Length > 0.

Note how we are focusing on the core value of Algolia which is search. We are less focused in auxiliary features like Algolia’s monitoring API,reporting API, etc, nor do we look at advanced features like synonyms or rules APIs.

Developer Cohort Segmentation

Now that we have an understanding of conversion rates for each stage of the developer funnel, it’s important to leverage API Analytics to understand how different cohorts of developers move through your developer funnel. Without it, you can have a false sense of success.

For example, you can have a high conversion rate from Pre-integration to Production stage, yet if the majority of developers are working on side projects without buying power, then you may need to reevaluate your targeting.

Building profiles of users and companies

In order to segment your funnel further, your API analytics solution should have a way to create User Profiles and Company Profiles. Recommended variables to track in user profiles include:

Email
Name
Job Title
Referring site

Recommended variables to track in companies include:

Company Revenue
Number of Employees
Industry
Alexa Rank
Plan MRR

Once you have this information, your developer funnel analytics become far more valuable. Instead of just tracking the high-level integration journey, you’re able to see who is integrating and who isn’t. For example, we can segment by Company Revenue to compare the conversion rate of users working on side projects vs. the conversion rate of engineers at Fortune 500 companies.

Tracking companies (i.e. accounts) separately from users ensures you’re able to create metrics on the number of developers or teams at an organization using your solution.

Understanding API design

Just like building profiles of users and companies, your API Analytics solution should understand the business functionality of your API. If your API is rest, then tracking just POST /items/1 and POST /items/2 would not be valuable. However, adding metadata associated with the API transaction can give you higher level trends on usage patterns. For example if your API is RESTful, your analytics solution should bucket the API transactions into a specific REST resource operation like POST /items/:id. This enables higher level charts such as below which shows the top companies using each REST API operation.

Defining API engagement

Before we dig into critical business metrics like acquisition channel performance and API growth, we need to define what is considered engagement with an API. After all, a developer just hitting your root endpoint or just probing for API health may not be considered real engagement. Continuing with the Algolia example, Algolia’s core business value prob is fast search. Thus, their criteria to define engagement on an API is an account performing at least one search POST /1/indexes/{indexName}/query in the last 28-days.

Measuring Developer Acquisition

Efficient new customer acquisition can make or break an API business. It’s imperative to consider the conversion rates of our three steps defined in our developer funnel when measuring the effectiveness of online marketing campaigns across traffic sources. API product marketing leaders should move beyond basic metrics like sign-ups and consider the developer integration funnel. This enables you to understand how far those sign-ups went through the integration steps.

For example, a channel like Facebook ads may give your B2B solution a large reach, but if a very small fraction of those sign ups end up going through the integration process due to poor targeting, your overall effectiveness of the channel is lowered as it’s unlikely for you to extract much value from someone who only signed up without doing anything else.

How to properly measure developer marketing channels

If we sync marketing data such as from Urchin Tracking Module (UTM) parameters from marketing automation and analytics tools like Hubspot or Amplitude, then we can leverage this in our cohort analysis. This enables us to take our earlier developer funnel and segment by the channel. Once we do that, we’re able to see the conversion rate for each step broken down by campaign.

What is Moesif? Moesif is the most advanced API analytics platform used by Thousands of platforms to understand what your most loyal customers are doing with your APIs, how they’re accessing them, and from where.

Measuring API Growth and Engagement

API MAU and API DAU

If you want to track growth of your API program, API Monthly Active Users (API MAU) and API Daily Active Users (API DAU) are two KPIs every data-driven product leader should be tracking.

This is similar to web and mobile MAU and can give a high-level overview of growth metrics and KPIs. However, instead of tracking distinct users performing UI interactions such as button clicks within your app, API MAU is tracking distinct users performing programmatic interactions with your API. Depending on your industry, a distinct user can be an individual user or developer or it can be distinct companies or accounts. API Analytics products like Moesif can track which is critical for account-level analytics.

Why 28-days?

Even though we say MAU, we should really be measuring activity in fixed 28-day windows, not the entire month. By using 28-days, we are able to ignore bias due to the number of days in a month not being consistent. In addition, 28-days ensures each time window starts on the same weekday. This guarantees we always have the same number of weekdays and weekends in each time window. Many companies, especially B2B, have different levels of usage on the weekends compared to weekdays. By keeping these numbers consistent, we can remove those biases.

28-day API usage/28-day active companies

You should also be tracking total API usage per active company for the same engagement definition. API usage is simply the total API usage in sub of API calls rather than counting distinct users. By dividing API usage by active companies, you can track how each account is growing with your API over time. This metric is a leading indicator for revenue expansion. If API Usage/Active companies is flat (or worse decreasing), your API is not growing with the accounts and may be hard to drive upsells through plan limits. Whereas an API program with a rapidly growing API usage/active companies will enjoy a growing average contract value and top-line revenue. Of course, this assumes your pricing is based on some usage metrics such as search operations or index operations in Algolia’s case.

Distinct API Operations/Application

If you have an API with hundreds of separate features and entities like GitHub or Intuit’s APIs, then measuring how many separate functions each app calls can help you understand the breadth of API usage. It’s helpful to plot this as a histogram to find the percentage of power users who are using your API fully compared to other developers who may be using only one or two endpoints. How you track distinct API operations will depend on the architecture of your API. For example, if your API is REST, you may have REST endpoints like:

GET /items
GET /items/:id
POST /items

GET /users
GET /users/:id
POST /users

GET /purchases
GET /purchases/:id
POST /purchases

...

whereas, if you’re API is GraphQL, you may want to track query and mutate operations like:

getItems
createItem
createItem

getUsers
getUser
createUser

getPurchases
getPurchase
createPurchase

Conclusion

There are many deeper API metrics that you should be tracking if you intend to build a growing API program. However, if you haven’t started taking a data-driven approach yet, these metrics are a good starting point. As you grow your API program, you’ll be tracking other KPIs that can show you the health of your API program. Tools like Moesif API Analytics can help you get started measuring these metrics with just a quick SDK installation.

Continue with Part 2 of this series: Mastering API Analytics - Cohort Retention Analytics

Understand Your API Adoption Funnel With Moesif 14 day free trial. No credit card required. Learn More

API Platform , API Product Management , Funnel Analysis , Metrics

Derric Gilling

Co-founder & CEO @Moesif. Previously Computer Architect @Intel. Studied @UMichigan.

API Strategy

Announcing General Availability of the Moesif Developer Portal

We’re thrilled to announce the General Availability of the Moesif Developer Portal, a major leap forward enabling you to deploy a seamless experience for dev...

December 06, 2024

API Analytics and Monitoring

Introducing AI Explain

We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...

November 14, 2024