Developer Experience: The Metrics That Matter Most
Developer experience. If you provide APIs or API-first products, you likely hear that term a lot. After all, you need developers for an API to succeed — and if they don’t have a great experience, they’ll move on. Software development requires tools that provide the most value for the least amount of upkeep. Understanding your key performance indicator for your API product or software quality metrics allows you to improve on your developer experience.
What Is Developer Experience?
Developer Experience (DevEx or DX) is an extension of user experience (UX) where the focus is on users impacted by the technical side of things — for example, tooling, languages, and workflows. But DevEx is far more than “UX for developers”: it means ensuring that developers can easily understand and leverage an API for their own applications and use cases. Great DevEx happens when you communicate with your developer users, understanding and meeting their needs directly by looking at and iterating off of your software metrics. If you can win over developers, you can build a large and thriving ecosystem around your software engineering products.
What Metrics Matter? It Depends on Your Role
Celebrated management consultant Peter Drucker may have said it best: “If you can’t measure it, you can’t manage it.”
Improving DevEx starts with the right metric analysis. You’ll want to measure the things that matter most to DevEx, avoiding vanity metrics and linking specific API transactions to business value. What matters most to you, however, depends on your role in improving DevEx. Finding quality metrics that are meaningful to your API product will aid your development teams in valuing your users in the software development process.
There may be three distinct roles in your organization focused on DevEx: developer experience manager, developer relations professional, and API product manager. The concerns of each of these roles overlap in project management, but they differ in focus. These distinct focuses will be represented in the metrics they care about most.
Developer Experience Manager
DevEx managers focus on the effectiveness of the APIs or API-first products developers use. They also look at ways to improve processes so that developers achieve success. The DevEx manager is responsible for running everything developers access and use — from the developer portal and documentation to code samples and SDKs.
DevEx managers make sure developers have a great experience with the tools and processes associated with the product. As such, the lean metrics they most care about include the following.
Time to User Activation
How long does it take a development team to start using the API once they complete the registration — hours, days, weeks? If a developer doesn’t integrate or activate promptly, it could indicate a problem. Lead time is a major determining factor in integration success. Perhaps your documentation is confusing, or you don’t have SDKs for popular languages. If an engineering team does not integrate or activate within a few days, a good DevEx manager will take notice. Aiding developer productivity starts with a solid initial integration.
Time to First Hello World
Time to First Hello World (TTFHW) measures how long it takes a new developer to get up and running, achieving the minimal level of value from your API. Every developer has different criteria for what constitutes that first success. It could be calling the API for the first time and getting a response. It could be creating a simple test app and completing a transaction that verifies the API will suit the developer’s needs. But in any case, the mean time a developer requires to achieve a modicum of success with your API is a key indication of whether they’ll continue using it and convert to a paying customer. Understanding the project metrics that have the most value will allow you to more appropriately triage issues and speed up your development process and deployment frequency.
Number of Devs Needing Support
How many developers have reached out to get support on an issue? Are many developers experiencing the same quality assurance problem? DevEx managers should log all integration issues internally and track how many developers need help improving a specific performance metric or productivity.
Time to Support Resolution
If you have a support system in place, how long does it take your agile teams to resolve technical support or code quality issues? Developers don’t want to wait to resolve their issues with your API or the integration. They want prompt answers to help them move forward and achieve their development goals.
Developer Relations Professional
Developer relations (DevRel) professionals aim to get developers to adopt an API or API platform and complete their goals with it. Typical DevRel roles include developer evangelist, developer advocate, and community manager.
DevRel professionals focus on building and maintaining relationships with developers and the developer communities served by the product or service. People in DevRel positions care about making connections with developers, listening to their thoughts, and figuring out how to address their needs. They also help developers understand how to use the API or product, providing demonstrations and leading educational events, like hackathons and meetups.
Two north star metrics for DevRel professionals are TTFHW and Weekly Active Tokens (WAT).
Weekly Active Tokens (WAT)
DevRel professionals want to see how much traction the API gets and whether the developer community is growing. To help figure this out, they can look at the weekly active tokens (WAT) for the API. Most APIs limit access to authenticated users, so they can track how many unique tokens access the API platform weekly. If they narrow this metric to Weekly Active Integrated Companies, the DevRel team can figure out where to invest more time and effort in developer outreach and marketing.
Other DevRel Metrics
While TTFHW and WAT are the two main DORA metrics DevRel professionals focus on, they may also be interested in measures such as
- How much coverage does the company have with docs, guides, and quickstart resources?
- How often and how long are devs interacting with reference documentation? Which parts?
- How satisfied are devs with the support they receive?
- How frequently are teams creating content to support devs?
- Are devs engaging with the instructional content the company offers?
API Product Managers
API product management is somewhat new, so the responsibilities of this role vary depending on the company and the API products it offers. In general, an API product manager understands the business justification for the API or API platform and the high-level objectives for the company’s API initiatives. They also need to balance the needs of internal stakeholders and product developers with requests from customers.
API product managers focus primarily on user growth, user retention, and usage patterns. They need to know which features and endpoints developers use the most (and the least) to prioritize development. They also need to understand the impact of API changes, like deprecated endpoints and new API versions, on customers.
API product managers look at a range of metrics, but the two most common are API Usage Growth and Unique API Consumers.
API Usage Growth
API product managers need to measure API adoption, and they do that by looking at API usage growth. They measure API usage over extended periods, like weeks or months, to discover key trends in growth.
Unique API Consumers
Sometimes an increase in API usage occurs because of a single customer account instead of unique user growth. API product managers should measure the number of monthly or daily unique consumers of their APIs. They should also monitor these users daily or monthly to see if they remain active. Looking at monthly or daily active users (MAU/DAU) can tell you if increased API usage comes from new customers, existing customers, or both.
Other metrics API product managers look at include
- When, how, and how long do developers use the API?
- Which API features are developers using or not using?
- What kind of feedback are specific product features getting?
- How does the money spent compare to overall product adoption?
Tracking DevEx Metrics
While each of your team’s DevEx professionals cares about different things, they all need a tool that will help them efficiently track and measure relevant metrics. Is your current role focused on DevEx? Then you need an effective API analytics tool that will track and measure the metrics you care about and the metrics that matter to other roles.
The tool you choose should provide visibility into all APIs — e.g., REST, RPC, SOAP, Hypermedia — and APIs running on query languages like GraphQL. It should include plug-ins and SDKs so you can integrate with popular servers without having to write a lot of custom code. Finally, it should include pre-built dashboards to track the metrics every team cares about — from engineering and security to product and customer success.
You don’t have to look far to find an analytics tool with these features — the Moesif API Experience Platform has them all. You can try out the platform for free for 14 days.
Build Better Relationships With Developers Using Moesif
14 day free trial. No credit card required.
Learn More
Related Articles
API Strategy
Announcing General Availability of the Moesif Developer Portal
We’re thrilled to announce the General Availability of the Moesif Developer Portal, a major leap forward enabling you to deploy a seamless experience for dev...
API Analytics and Monitoring
Introducing AI Explain
We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...
API Analytics and Monitoring
Monitoring Cost and Consumption of AI APIs and Apps
Discover how Moesif can help manage and optimize your AI-driven APIs with powerful observability tools that track costs and usage efficiently.
API Development
Unlock Deep API Insights with Moesif’s HTTP Body Analytics
Learn how to use Moesif to analyze HTTP bodies.
