Building a modern/serverless application is the most talked-about trend in the cloud engineering space as it enhances business agility and lowers the total cost of ownership. It frees your developer from the heavy lifting of server management with scalability & high availability so that the team can focus on application development.

However, debugging is the most significant problem of serverless applications and existing solutions.

This blog talks about a use case where Motherson Technology Services helped a client debug and troubleshoot their serverless application.

Use Case Overview

One of clients built a serverless application on AWS using API Gateway, Lambda and DocumentDB as the backend for payment processing gateway (represented in the image below), which is critical and demands very low latency and offers high availability.

As a result, high availability was achieved using AWS Managed Services, including API Gateway, Lambda and DocumentDB; however, the latency and timeout errors were causing the client to have hard times.

These latency issues and timeouts were random and didn’t have any pattern. Therefore, the team started to analyze each layer of the serverless application to find the actual component facing the issue.

The Approach

It started with the analysis of the API Gateway logs. We examined the IntegrationLatency metrics to measure the responsiveness of the backend and Latency metrics to calculate the overall responsiveness of the API calls.

Subsequently, we studied the Lambda logs and turned on the X-Ray, a great tool to visualize the application parts, troubleshoot performance and inspect the Lambda cold start issues.

To handle the Lambda cold start issue, which occurs if there are no reusable containers to serve the request, AWS needs to provision a new container, download the code and dependencies from S3, and initialize by executing the code outside the handler function.

This cold start introduces a latency of about 1-4 seconds, which is highly critical for financial applications.

The team scheduled CloudWatch events to call each Lambda function every 5 minutes in order to keep the Lambda functions warm. In each Lambda function, a code existed that needed to be examined if it received a request from the CloudWatch event (passing a dummy parameter). After receiving the same, it returned the request without carrying out any processes.

Upon analyzing the Lambda X-Ray logs, we were able to identify that it was due to a cold start issue.

Findings

Based on the logs and error message research, the team determined the main reason for the ongoing issue- the MongoDB Node.js Driver. Therefore, we advised the client to upgrade the driver, following which the timeout issue was resolved.

Moreover, about the latency issues, we found problems in the client approach of keeping the Lambda containers too warm to handle the Lambda cold start issues. Resultingly, we suggested a new method of using Provisioned Concurrency and helped the client implement the same, which solved the latency trouble.

To learn more about Serverless Web Applications, request a demo or contact us here.