How we built a Node.js Middleware to Log HTTP API Requests and Responses
There are many different runtimes and eco-systems used to build APIs and here at Moesif we try to make integration with them as simple as possible. We build many libraries that help with this integration, one of them is the Moesif Node.js Middleware Library, or short moesif-nodejs. Middleware in Node.js is a function that has access to request and response objects and the next middleware function in the request-response cycle.
Node.js handles requests asynchronously which can sometimes lead to problems, especially when we want to debug our systems or log what they are doing. API logs capture the intricate details of requests and responses exchanged between applications and servers, making them essential for understanding and troubleshooting such asynchronous behavior.
In this article, we will go through the steps that went into building the moesif-nodejs library, the places where the relevant logging data is and how to hook into Node.js’ http module to handle the data-gathering at the right time in the pipeline.
Introduction to API Logging
API logging is the process of capturing and recording API interactions, including requests, responses, and errors. API logs provide valuable insights into API performance, security, and usage, enabling developers to troubleshoot issues, optimize performance, and ensure compliance with regulatory standards. In this section, we will explore the importance of API logging, its benefits, and the different types of API logs.
API logs are essential for monitoring and maintaining the health, security, and efficiency of APIs. They provide a comprehensive view of API interactions, allowing developers to identify trends and patterns in API usage, detect security threats, and optimize performance. API logs can be categorized into different types, including access logs, error logs, security logs, and performance logs.
Access logs record details about each API request, such as the HTTP method, URL, and response status code. Error logs capture information about any errors that occur during API calls, helping developers identify and fix issues. Security logs track failed authentication attempts and access control violations, providing insights into potential security threats. Performance logs measure the efficiency and response speed of APIs, recording metrics such as response time and throughput.
By leveraging API logs, developers can gain a deeper understanding of how their APIs are being used, identify and resolve issues more quickly, and ensure that their APIs are secure and performant.
For developers looking for an effortless way to monitor and analyze API traffic, integrating Moesif can significantly streamline this process. Moesif’s API analytics platform provides deep insights into API usage, identifying trends and performance bottlenecks while ensuring compliance with security policies. By leveraging tools like the Moesif Node.js Middleware, developers can log API requests and responses efficiently without modifying their core application logic.
Understanding Middleware in Node.js
In Node.js, middleware is a function that can access the request object (req), the Middleware functions can perform tasks such as authentication, caching, and compression, and can be used to handle HTTP requests and responses. In this section, we will explore the concept of middleware in Node.js, its benefits, and how to create and use middleware functions.
Middleware functions are essential in Node.js applications, as they enable developers to perform tasks that are common to multiple routes or endpoints. Middleware functions can be used to authenticate users, cache frequently accessed data, and compress responses to improve performance. Node.js provides several built-in middleware functions, including express.static, express.json, and express.urlencoded.
The express.static middleware function serves static files, such as images, CSS files, and JavaScript files, from a specified directory. The express.json middleware function parses incoming requests with JSON payloads, making it easier to work with JSON data. The express.urlencoded middleware function parses incoming requests with URL-encoded payloads, which is useful for handling form submissions.
By using middleware functions, developers can create modular and maintainable code, as each middleware function can be responsible for a specific task. This modularity makes it easier to manage and update the application, as changes to one middleware function do not affect others.
For instance, Moesif’s middleware for Node.js seamlessly integrates into Express applications, allowing developers to capture request-response data for logging, debugging, and analytics. This approach not only simplifies tracking API performance but also enables real-time monitoring, anomaly detection, and automated alerts—essential for ensuring API reliability.
Node.js’ HTTP Module
Node.js comes with an HTTP-server implementation out-of-the-box, while it isn’t used directly in most applications, it’s a good start to understand the basics of requests and responses.
In a middleware setup, the current middleware function plays a crucial role in managing the request-response cycle. If the current middleware function does not complete the cycle, it must invoke the next middleware function to prevent requests from being left hanging.
Designing the Middleware Function
Designing a middleware function involves several steps, including defining the function signature, accessing the request and response objects, and calling the next middleware function. In this section, we will explore the best practices for designing middleware functions, including how to handle errors, how to access and modify the request and response objects, and how to call the next middleware function.
When designing a middleware function, it is essential to consider the function signature, which includes the request object (req), the response object (res), and the next middleware function (next). The middleware function should access and modify the request and response objects as needed, and call the next middleware function to pass control to the next function in the chain.
To handle errors effectively, the middleware function should include error-handling logic. This can be done by checking for errors in the request and response objects and logging any errors that occur. Additionally, the middleware function should call the next middleware function with an error argument if an error is detected, allowing the error to be handled by error-handling middleware functions.
Accessing and modifying the request and response objects is a common task in middleware functions. For example, a middleware function might add a custom header to the response object or modify the request object to include additional data. It is important to ensure that any modifications to the request and response objects do not interfere with other middleware functions or the final response.
Calling the next middleware function is crucial for ensuring that the request-response cycle continues. The middleware function should call the next middleware function after completing its task, passing control to the next function in the chain. This allows multiple middleware functions to work together to handle a single request.
By following these best practices, developers can design effective and efficient middleware functions that enhance the functionality and maintainability of their Node.js applications.
Basic Logging of a GET Request
The idea behind logging is that we write some kind of data in some persistent data store so we can review it later. To keep things simple, we will first write to stdout with console.log() Sometimes logging to stdout isn’t an option and we have to send the logging data to some other place. Like when running in a serverless environment, where the functions have no persistent storage. Let’s try a simple server that does nothing but sending empty responses for every request to illustrate how to get request data that can be logged. const http = require(“http”);
const http = require("http");
const server = http.createServer((request, response) => {
console.log(request);
response.end();
});
server.listen(8888);
If we send a request to http://localhost:8888 we see a giant object being logged to stdout, it is full of implementation details and finding the important parts isn’t easy. These log details include various components such as timestamps, client information, request and response details, and security events, which are essential for troubleshooting issues, monitoring performance, and recognizing potential security threats.
Let’s look at the Node.js documentation for IncomingMessage, the class of which our requests is an object of.
What information can we find here?
headersandrawHeaders(for invalid/duplicate headers)httpVersionmethodurlsocket.remoteAddress(for the client IP)
This should be enough for GET requests since they usually don’t have a body. Let’s update our implementation.
const server = http.createServer((request, response) => {
const { rawHeaders, httpVersion, method, socket, url } = request;
const { remoteAddress, remoteFamily } = socket;
console.log(
JSON.stringify({
timestamp: Date.now(),
rawHeaders,
httpVersion,
method,
remoteAddress,
remoteFamily,
url
})
);
response.end();
});
The output should look something like this:
{
"timestamp": 1562331336922,
"rawHeaders": [
"cache-control",
"no-cache",
"Postman-Token",
"dcd81e98-4f98-42a3-9e13-10c8401892b3",
"User-Agent",
"PostmanRuntime/7.6.0",
"Accept",
"*/*",
"Host",
"localhost:8888",
"accept-encoding",
"gzip, deflate",
"Connection",
"keep alive"
],
"httpVersion": "1.1",
"method": "GET",
"remoteAddress": "::1",
"remoteFamily": "IPv6",
"url": "/"
}
We only use specific parts of the request for our logging. It uses JSON as format so it’s a structured logging approach and has a timestamp so know we not only know what was requested by whom but also when the request started. Structured logging uses a uniform format across all log entries to facilitate easier querying and analysis.
Logging Processing Time
If we wanted to add data about how long the request took to process, we need a way to check when it’s finished. Performance logs measure the efficiency and response speed of APIs, recording metrics such as response time and throughput, which can be useful for such analysis.
The request is done when we finished sending our response, so we have to check when a response.end() was called. In our example, this is rather simple, but sometimes these end-calls are done by other modules.
For this, we can look at the docs of the ServerResponse class. It mentions a finish event that is fired when all the sever finished sending it’s response. This doesn’t mean the client received everything, but it’s an indicator that our work is done.
Let’s update our code!
const server = http.createServer((request, response) => {
const requestStart = Date.now();
response.on("finish", () => {
const { rawHeaders, httpVersion, method, socket, url } = request;
const { remoteAddress, remoteFamily } = socket;
console.log(
JSON.stringify({
timestamp: Date.now(),
processingTime: Date.now() - requestStart,
rawHeaders,
httpVersion,
method,
remoteAddress,
remoteFamily,
url
})
);
});
process(request, response);
});
const process = (request, response) => {
setTimeout(() => {
response.end();
}, 100);
};
We passed the processing of our request to a separate function to simulate an other module that takes care of it. The processing takes place asynchronously, because of the setTimeout, so synchronous logging wouldn’t get the desired result, but the finish event takes care of this by firing after response.end() was called.
Logging the Body
The request body is still not logged, which means POST, PUT and PATCH requests aren’t 100% covered.
To get the body into the logs too, we need a way to extract it from our request object.
The IncomingMessage class implements the ReadableStream interface. It uses the events of that interface to signal when body data from the client arrives.
- The
dataevent is fired when the server received new chunks of data from the client - The
endevent is called when all data has been sent - The
errorevent is called when something goes wrong
Let’s update our code:
const server = http.createServer((request, response) => {
const requestStart = Date.now();
let errorMessage = null;
let body = [];
request.on("data", chunk => {
body.push(chunk);
});
request.on("end", () => {
body = Buffer.concat(body);
body = body.toString();
});
request.on("error", error => {
errorMessage = error.message;
});
response.on("finish", () => {
const { rawHeaders, httpVersion, method, socket, url } = request;
const { remoteAddress, remoteFamily } = socket;
console.log(
JSON.stringify({
timestamp: Date.now(),
processingTime: Date.now() - requestStart,
rawHeaders,
body,
errorMessage,
httpVersion,
method,
remoteAddress,
remoteFamily,
url
})
);
});
process(request, response);
});
This way we log an additional error message when something goes wrong and add the body content to the logging.
Caution: The body can be very big and/or binary, so validation checks are needed, otherwise the amount of data or the encoding can mess up our logs.
Logging Response Data
Now that we got the requests down, the next step is the logging of our responses.
We already listen to the finish event of our response, so we have a pretty safe way to get all the data. We just have to extract what the response object holds.
Let’s look at the docs for the ServerResponse class to find out what it offers us.
statusCodestatusMessagegetHeaders()
Let’s add it to our code.
const server = http.createServer((request, response) => {
const requestStart = Date.now();
let errorMessage = null;
let body = [];
request.on("data", chunk => {
body.push(chunk);
});
request.on("end", () => {
body = Buffer.concat(body).toString();
});
request.on("error", error => {
errorMessage = error.message;
});
response.on("finish", () => {
const { rawHeaders, httpVersion, method, socket, url } = request;
const { remoteAddress, remoteFamily } = socket;
const { statusCode, statusMessage } = response;
const headers = response.getHeaders();
console.log(
JSON.stringify({
timestamp: Date.now(),
processingTime: Date.now() - requestStart,
rawHeaders,
body,
errorMessage,
httpVersion,
method,
remoteAddress,
remoteFamily,
url,
response: {
statusCode,
statusMessage,
headers
}
})
);
});
process(request, response);
});
Handling Response Errors and Client Aborts
At the moment we only log when the response finish event is fired, this isn’t the case if something goes wrong in response or if the client aborts the request.
For these two cases, we need to create additional handlers.
const server = http.createServer((request, response) => {
const requestStart = Date.now();
let body = [];
let requestErrorMessage = null;
const getChunk = chunk => body.push(chunk);
const assembleBody = () => {
body = Buffer.concat(body).toString();
};
const getError = error => {
requestErrorMessage = error.message;
};
request.on("data", getChunk);
request.on("end", assembleBody);
request.on("error", getError);
const logClose = () => {
removeHandlers();
log(request, response, "Client aborted.");
};
const logError = error => {
removeHandlers();
log(request, response, error.message);
};
const logFinish = () => {
removeHandlers();
log(request, response, requestErrorMessage);
};
response.on("close", logClose);
response.on("error", logError);
response.on("finish", logFinish);
const removeHandlers = () => {
request.off("data", getChunk);
request.off("end", assembleBody);
request.off("error", getError);
response.off("close", logClose);
response.off("error", logError);
response.off("finish", logFinish);
};
process(request, response);
});
const log = (request, response, errorMessage) => {
const { rawHeaders, httpVersion, method, socket, url } = request;
const { remoteAddress, remoteFamily } = socket;
const { statusCode, statusMessage } = response;
const headers = response.getHeaders();
console.log(
JSON.stringify({
timestamp: Date.now(),
processingTime: Date.now() - requestStart,
rawHeaders,
body,
errorMessage,
httpVersion,
method,
remoteAddress,
remoteFamily,
url,
response: {
statusCode,
statusMessage,
headers
}
})
);
};
Now, we also log errors and aborts.
The logging handlers are also removed when the response finished and all the logging is moved to an extra function.
Logging to an external API
At the moment the script only writes its logs to the console and in many cases, this is enough because operating systems allow other programs to capture the stdout and do their thing with it, like writing into a file or sending it to a third-party API like Moesif, for example. API logs are vital tools for monitoring the health of APIs, optimizing performance, and ensuring compliance with regulatory standards.
In some environments, this isn’t possible, but since we gathered all information into one place, we can replace the call to console.log with a third-party function.
Let’s refactor the code so it resembles a library and logs to some external service.
const log = loggingLibrary({ apiKey: "XYZ" });
const server = http.createServer((request, response) => {
log(request, response);
process(request, response);
});
const loggingLibray = config => {
const loggingApiHeaders = {
Authorization: "Bearer " + config.apiKey
};
const log = (request, response, errorMessage, requestStart) => {
const { rawHeaders, httpVersion, method, socket, url } = request;
const { remoteAddress, remoteFamily } = socket;
const { statusCode, statusMessage } = response;
const responseHeaders = response.getHeaders();
http.request("https://example.org/logging-endpoint", {
headers: loggingApiHeaders,
body: JSON.stringify({
timestamp: requestStart,
processingTime: Date.now() - requestStart,
rawHeaders,
body,
errorMessage,
httpVersion,
method,
remoteAddress,
remoteFamily,
url,
response: {
statusCode,
statusMessage,
headers: responseHeaders
}
})
});
};
return (request, response) => {
const requestStart = Date.now();
// ========== REQUEST HANLDING ==========
let body = [];
let requestErrorMessage = null;
const getChunk = chunk => body.push(chunk);
const assembleBody = () => {
body = Buffer.concat(body).toString();
};
const getError = error => {
requestErrorMessage = error.message;
};
request.on("data", getChunk);
request.on("end", assembleBody);
request.on("error", getError);
// ========== RESPONSE HANLDING ==========
const logClose = () => {
removeHandlers();
log(request, response, "Client aborted.", requestStart);
};
const logError = error => {
removeHandlers();
log(request, response, error.message, requestStart);
};
const logFinish = () => {
removeHandlers();
log(request, response, requestErrorMessage, requestStart);
};
response.on("close", logClose);
response.on("error", logError);
response.on("finish", logFinish);
// ========== CLEANUP ==========
const removeHandlers = () => {
request.off("data", getChunk);
request.off("end", assembleBody);
request.off("error", getError);
response.off("close", logClose);
response.off("error", logError);
response.off("finish", logFinish);
};
};
};
With these changes, we can now use our logging implementation as we would use moesif-nodejs.
The loggingLibrary function takes an API-key as configuration and returns the actual logging-function that will send the log-data to a logging service via HTTP. The logging-function itself takes a request and response object.
Conclusion
With these changes, we can now use our logging implementation as we would use moesif-nodejs.
The loggingLibrary function takes an API-key as configuration and returns the actual logging-function that will send the log-data to a logging service via HTTP. The logging-function itself takes a requestresponse object.
Here are a few:
For developers who want deeper visibility into API traffic with minimal setup, Moesif offers a seamless solution. Unlike traditional logging libraries, Moesif provides real-time API analytics, retention tracking, and security monitoring, making it an essential tool for scaling and optimizing API-driven applications.
Monitor and Analyze API Traffic with Moesif
14 day free trial. No credit card required.
Try for Free
Related Articles
API Development
WSO2 Choreo + Moesif API Analytics: Optimize API Performance And Drive Adoption
Learn how you can optimize API performance and drive adoption in WSO2 Choreo with Moesif API analytics plugin.
Podcasts
APIs Over IPAs 15: Customer Observability with Mike Amundsen, Amundsen.com
Derric Gilling sits down with Mike Amundsen, a leader in the API ecosystem, to explore the role of observability in API design and customer experience.
API Development
Achieving API observability in Kong Konnect with Moesif API analytics plugin
Learn how you can achieve API observability and monitoring in Kong Konnect with Moesif API analytics plugin.
API Analytics and Monitoring
Monetizing Proprietary Data Through APIs: How to Unlock New Revenue in the AI World
Monetizing data can bring numerous benefits to an organization. By leveraging their existing data, companies can unlock new revenue streams and gain a compet...
