How to add Moesif API Analytics and Monitoring to Kong Ingress Controller
Kong is a popular open-source API gateway to help manage your APIs. With Kong, you can handle authentication, rate limiting, data transformation, among other things from a centralized location even though you have multiple microservices. Kong is built on NGINX at it’s core, one of the most popular HTTP servers. Being open-source, Kong is very easy to deploy on-premises usually in just a few minutes without requiring the installation of many components other than a Postgres or Cassandra store.
Kong Ingress Controller implements authentication, transformations, and other functionalities across Kubernetes clusters with zero downtime. Kong Gateway connects Kubernetes clusters with services running across any environment or platform. By integrating with the Kubernetes Ingress Controller, Kong ties directly to the Kubernetes lifecycle. As applications are deployed and new services are created, Kong will automatically live configure itself to serve traffic to these services. Kong and the Kong Ingress Controller has a full management layer and API, live configuration of targets and upstreams, and a durable, scalable state storage using either Postgres or Cassandra that ensures every Kong instance is synced without delay or downtime.
Moesif provides a plugin for Kong that makes getting started with API observability in just a few minutes so you can stay focused on shipping features customers love rather than deal with the maintenance cost of building your own data infrastructure. With Moesif, you’re able to understand how your API is used and by what customers, identify which customers are running into integration issues, and monitor for endpoints that need optimization. In this article, we’ll talk about the insights Moesif gives you, and how to integrate it with Kong Ingress Controller. Then, we will talk about how to best leverage API observability using Moesif and Kong.
Overview
Moesif Kong plugin is an agent that collects metrics and sends to the Moesif collection network. This enables you to get a complete picture of your API usage even across different Kong instances and data center regions. Moesif provides deep insights for engineering teams to understand how their APIs are used and quickly troubleshoot complex issues. Because Moesif also tracks who is calling your API and how they are accessed, product-driven teams can understand the entire customer journey and where to invest more resources. With API observability, forward-thinking engineering leaders can empowers customer-facing teams with self-service analytics on activation funnels, retention reports, and more. Moesif also analyzes your API payloads for troubleshooting and business insights so you’re able to understand utilization of specific payload keys, etc.
Setting up the Kong Ingress Controller
PLEASE NOTE that you need to perform this step only if you don’t have ingress controller setup and running. If you’ve already have ingress controller running, refer to the step on how to load the moesif plugin with existing kong ingress controller running.
We’ll perform following steps to deploy the Ingress Controller and load the Moesif plugin.
Create a ConfigMap with the Moesif plugin code:
You’ll need to clone the kong-moesif-plugin and navigate to the kong/plugins directory to create a configMap using
kubectl create configmap kong-plugin-moesif --from-file=moesif -n kong
Please ensure that this is created in the same namespace as the one in which Kong is going to be installed.
Create namespace
If you’ve the namespace that you’d like to use is created, you’d skip this step. But please make sure to use the same naespace in which Kong is installed.
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"name": "kong",
"labels": {
"name": "kong"
}
}
}
Create the namespace using
kubectl apply -f namespace.json
Add Kong Chart
You’ll need to add the kong chart and update the repo via Helm.
# Add kong chart
helm repo add kong https://charts.konghq.com
# Update helm repo
helm repo update
Deploy the Kubernetes Ingress Controller & Load Moesif plugin:
With Helm, you could load the Moesif plugin by adding the following values to your values.yaml file:
# values.yaml
plugins:
configMaps:
- name: kong-plugin-moesif
pluginName: moesif
Setting up Kubernetes Ingress controller and loading the Moesif plugin using Helm chart is as simple as:
helm install kong kong/kong --namespace kong --set ingressController.installCRDs=false --values values.yaml
Set up an environment variable
You’ll set up an environment variable with the IP address at which Kong is accessible. This will be used to actually send requests into the Kubernetes cluster. You will create a tunnel for kong-proxy and get the url.
If you’re deploying Kong Ingress on Amazon Elastic Kubernetes Service (EKS), you’ll need to set an environment variable using
export PROXY_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].hostname}" service -n kong kong-proxy)
Please note that if your kong-proxy service is named differently, you’ve to change the command accordingly.
Please note that if you’re deploying Kong Ingress via a different method, please refer to the docs on how to set an environment variable.
You could echo the environment variable using -
echo $PROXY_IP
# http://192.168.99.100:32728
Testing connectivity to Kong
This assumes that PROXY_IP environment variable is set to contain the IP address or URL pointing to Kong. If you’ve not done so, please follow one of the deployment guides to configure this environment variable.
If everything is set up correctly, making a request to Kong should return back a HTTP 404 Not Found.
curl -i $PROXY_IP
HTTP/1.1 404 Not Found
Date: Fri, 21 Jun 2019 17:01:07 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
Server: kong/1.1.2
{"message":"no Route matched with those values"}
This is expected since Kong doesn’t know how to proxy the request yet. Please note that if you’re using minikube, make sure that the services of type LoadBalancer is exposed via the minikube tunnel command. It must be run in a separate terminal window to keep the LoadBalancer running. Ctrl-C in the terminal can be used to terminate the process at which time the network routes will be cleaned up.
Next, we’ll enable the Moesif plugin globally.
Load Moesif Plugin with existing Kong Ingress Controller running
PLEASE NOTE that you need to perform this step only if you haven’t performed the previous step. You should perform this step when you’ve Kong Ingress controller already running and wants to enable Moesif plugin.
This section assumes that you’ve kong ingress controller already running and want to load Moesif plugin. This also assumes that you’ve helm kong chart available, if not please add Helm kong chart.
Create a ConfigMap with the Moesif plugin code:
You’ll need to clone the kong-moesif-plugin and navigate to the kong/plugins directory to create a configMap using
kubectl create configmap kong-plugin-moesif --from-file=moesif -n kong
Please ensure that this is created in the same namespace as the one in which Kong is going to be installed.
Deploy the Kubernetes Ingress Controller & Load Moesif plugin:
With Helm, you could load the Moesif plugin by adding the following values to your values.yaml file:
# values.yaml
plugins:
configMaps:
- name: kong-plugin-moesif
pluginName: moesif
Rollout the deployment
You’ll need to patch the kong ingress controller deployment with the Moesif plugin and rollout the deployment.
helm upgrade kong kong/kong --namespace kong --values values.yaml
Next, we’ll enable the Moesif plugin globally.
Enable Moesif plugin globally
PLEASE NOTE that you need to perform this step, irrespective of which of the above step you performed. This is a critical piece which would enable Moesif plugin globally and start logging data to Moesif.
To set up KongClusterPlugin resource, you will need your Moesif Application Id. You can get one by signing up for a free Moesif account, then select Kong during the onboarding. Moesif recommends using of a single application Id for all Kong instances and data-center regions. This ensures you have a unified view of your API usage data regardless of physical topology. You can still break down by any number of attributes using Moesif’s high-cardinality, high-dimension analytics engine.
Moesif still recommends creating separate Application Ids for each environment such as Production, Staging, and Development to keep data isolated.
Create a global-plugin.yaml file
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: moesif
annotations:
kubernetes.io/ingress.class: kong
labels:
global: "true"
config:
application_id: Your Moesif Application Id
debug: false
plugin: moesif
and then apply the plugin globally.
kubectl apply -f global-plugin.yaml
Please note that setting the label global to “true” will apply the plugin globally in Kong, meaning it will be executed for every request that is proxied via Kong.
Please ensure that this is created in the same namespace as the one in which Kong is installed. If your namespace is different from kong, you should change this command accordingly.
See Moesif plugin in action
At this point, if you already have Kong Resources (services) configured, all the api calls proxied via Kong will get logged in Moesif. You don’t have to do the next steps as that is just an example to create service/resource for someone who wants to create sample service.
Set up an echo-service
Set up an echo-service application to demonstrate how to use the Kubernetes Ingress Controller:
kubectl apply -f https://bit.ly/echo-service -n kong
Please note that echo-service file is not being changed, it’s listening to port 80.
Please ensure that this is created in the same namespace as the one in which Kong is installed. If your namespace is different from kong, you should change this command accordingly.
Create a new Ingress resource which uses Moesif plugin
You can annotate an Ingress or Service resource to instruct Kong on when to execute the plugin. You could configure plugins globally, on Service resource, or for a specific consumer. This will create an ingress rule to proxy the echo-server created previously.
echo "
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo-example-com
annotations:
kubernetes.io/ingress.class: kong
spec:
rules:
- host: example.com
http:
paths:
- path: /bar
backend:
serviceName: echo
servicePort: 80
" | kubectl apply -f - -n kong
Please note that the servicePort is set to 80 because in the echo-service we created earlier, port 80 was exposed. If you’ve the service exposed at different port, please change servicePort accordingly.
Please ensure that this is created in the same namespace as the one in which Kong is installed. If your namespace is different from kong, you should change this command accordingly.
Send a Request
Once you have the Moesif plugin installed, your API traffic should start showing up in the live event stream within Moesif. You could send traffic to the resource configured using
curl -i -H "Host: example.com" $PROXY_IP/bar/sample
If everything is set up correctly, you’d see the 200 response. Below is what sample response would look like -
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Wed, 17 Nov 2021 21:31:04 GMT
Server: echoserver
X-Moesif-Transaction-Id: f8a8660e-1843-4844-89a7-67fbf2ac15ae
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 0
Via: kong/2.5.1
Hostname: echo-5fc5b5bc84-5s772
Pod Information:
node name: ip-192-148-47-105.us-west-2.compute.internal
pod name: echo-5fc5b5bc84-5s772
pod namespace: kong
pod IP: 192.118.36.8
Server values:
server_version=nginx: 1.12.2 - lua: 10010
Request Information:
client_address=192.368.25.12
method=GET
real path=/bar/sample
query=
request_version=1.1
request_scheme=http
request_uri=http://example.com:8080/bar/sample
Request Headers:
accept=*/*
connection=keep-alive
host=example.com
user-agent=curl/7.68.0
x-forwarded-for=192.178.50.41
x-forwarded-host=example.com
x-forwarded-path=/bar/sample
x-forwarded-port=80
x-forwarded-proto=http
x-moesif-transaction-id=f8a8660e-1843-4844-89a7-67fbf2ac15ae
x-real-ip=192.178.50.41
Request Body:
-no body in request-
How to use API observability
Engineering metrics
The first thing you’ll probably interested in is engineering metrics like how is my API performing. This type of metric can be pulled up by going to Events -> Time Series view within Moesif. Then you can select 90th percentile latency as the metric to plot. You can then group by the URI route to understand which endpoints have the worst performance. Here, you can filter your traffic by API attributes like route, verb, along with HTTP headers and body fields.
Handling sensitive data
If your application consists of sensitive data such as healthcare or financial data, you can become data compliant with one of two options:
1. Zero-knowledge security
Client-side encryption has the benefits of low-maintenance SaaS while still putting you in control of your data. Because you control and encrypt the data on-premises before being sent to Moesif, Moesif physically cannot access your data. This requires only running a small appliance within your infrastructure called secure proxy to handle encryption/decryption of your data on the fly.
2. Data masking
If you don’t want client-side encryption, you can also mask data directly using the plugin. This is handled via the config.request_body_masks and config.response_body_masks configurations option. For example, you could mask a field called password that might be present in the payload. Additionally if you want to remove logging request and response body all together, you could set the config.disable_capture_request_body or config.disable_capture_response_body configuration option to true.
Closing thoughts
In this way, Moesif plugin will capture API requests and responses and log to Moesif for deep API observability and real-time monitoring of your API traffic via Kong and Kong handling all the other services around the application.
Get Deep API Observability for Kong Ingress Controller for Kubernetes
14 day free trial. No credit card required.
Learn More
Related Articles
API Strategy
Announcing General Availability of the Moesif Developer Portal
We’re thrilled to announce the General Availability of the Moesif Developer Portal, a major leap forward enabling you to deploy a seamless experience for dev...
API Analytics and Monitoring
Introducing AI Explain
We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...
API Analytics and Monitoring
Monitoring Cost and Consumption of AI APIs and Apps
Discover how Moesif can help manage and optimize your AI-driven APIs with powerful observability tools that track costs and usage efficiently.
API Development
Unlock Deep API Insights with Moesif’s HTTP Body Analytics
Learn how to use Moesif to analyze HTTP bodies.