BigBear.ai
  • Home
  • Industries
    • Academia
    • Government
    • Healthcare
    • Manufacturing
  • Solutions
    • Cyber
    • Data Analytics
    • Enterprise Planning and Logistics
    • Intelligent Automation
    • Modeling Solutions
    • Professional Services
  • Company
    • About
    • Investor Relations
    • Partners
    • Team
  • Careers
    • Benefits
    • Culture
    • Explore Jobs
    • Military and Veterans
    • Applicant Login
    • Employee Login
  • Resources
    • Blog
    • Events
    • Newsroom
    • Resource Library
    • Online Store
  • Contact
Search

Home Technology Solution for Distributed Tracing with Spring Boot and ELK

Blog

Solution for Distributed Tracing with Spring Boot and ELK

Chung Ha
March 9, 2022
  • Share
  • Share

As we discussed in a previous blog post, the key characteristic of a microservice is a distributed system. As you design the microservice system, there are several challenges you need to overcome when you develop the system. Today, we will discuss one of the challenges – log data management.

 

Design Challenge

In a microservice architecture, the system will consist of multiple services running in parallel. In practical terms, this means that each will generate log files. Unfortunately, it’s hard and time-consuming to visit and review the individual log files for troubleshooting and understand precisely what happened. In addition to that, you need to identify a series of service calls made between microservices and tie them back to the initial HTTP request made by a client. Anyone who has tried to employ log data will have found that, if you manually comb through the log data spread, the sequence of the executions distributed among the microservices would be extraordinarily inefficient and painful.

 

Solution

We can employ two design patterns for this problem – Log Aggregation and Log Correlation. Log Aggregation is used to continuously log the service activities and store the logs into a single repository – supported by a search capability. Log Correlation – which can be called Distributed Tracing – helps to assign the initial service request, which is called by the external client. These service requests use a unique ID and pass the request along to the next service.

 

Log Correlation Implementation

Suppose you want to implement Log Correlation manually. In that case, you will need to decide where the unique ID –  also called a correlation ID – is generated – and assign it to the incoming requests. The best place for that is the API gateway – which we will cover in more detail in a future blog. But, first, we need to be aware that all incoming requests must feed through the API gateway since the API gateway acts as the front door for the microservice system. As the requests enter, they are intercepted by the log correlation filter and assigned with the correlation ID.

It is certainly possible to develop an  HTTP filter in the getaway for the log correlation, but I would like to show how you can leverage Spring Cloud Sleuth in this paper.   As shown in Figure 1, the Sleuth library needs to be added to the pom.xml in each service. It will download all libraries required by the Spring Cloud Sleuth. That’s all you need to do for the Sleuth.

Figure 1: Dependency for Sleuth added in pom.xml

 

Figure 2: HelloWorld service that calls GoodNight service

 

Figure 3: GoodNight service

 

Figure 4: Service Name, Trace ID, and Span ID are inserted by Sleuth as the logger prints out

 

For the implementation example, I prepared two services, HelloWorld and GoodNight, as shown in Figures 2 and 3. The Sleuth will inject the correlation ID and the span ID in every log when the services are called, as shown in Figure 4. The correlation ID will be the same as propagated to the next services, but the span ID will be different for each service call. You have the logical information to group the related log data, but you still have the logs spread over multiple files. Let’s discuss how we can implement log aggregation to collect all logs in one place.

 

Log Aggregation Implementation

To aggregate the logs, you will need to build a centralized logging architecture. There are multiple options available, but we will review the ELK (Elasticsearch, Logstash, Kibana) stack in this blog.

Below is the high-level description of each tool responsibility.

  • Logstash: Responsible to ingest the log data to Elasticsearch
  • Elasticsearch: Responsible to index the ingested data
  • Kibana: Responsible to retrieve the data from Elasticsearch

 

Figure 5: Flow of Log Aggregation using ELK Stack

 

You can run the ELK stack by downloading from the elastic website or leveraging Docker with the docker-compose configuration similar to Figure 6.

 

Figure 6: Docker Compose configuration to run ELK Stack containerized

 

Regardless of how you run the ELK stack, there are two key steps you need to ensure to make the log aggregation work correctly.

  • Set up a log appender in the service: This will determine where the logs will be sent to, such as files, tcp, stdin, etc. In my example, I used them to be written to files.
  • Define the input and output of Logstash: You need to configure Logstash for input and output in the config file. The input should be matched with the log appender, and the output should be for the Elasticsearch. The Logstash configuration file will be something like Figure 7.

 

Figure 7: Example of Logstash Configuration File

 

When the log aggregation works with the ELK stack correctly, you will be able to query the logs with the same correlation ID as shown in Figure 8.

 

Figure 8: Logs searched by Trace ID in Kibana

 

Conclusion

We discussed why distributed tracing is needed for the microservice and how Spring Boot and the ELK stack can be implemented as the solution for that. Even though the log correlation can be implemented manually, you will save time and effort by employing Spring Cloud Sleuth. The ELK stack can be the perfect open-source solution for log aggregation.

 

About the Author
Chung Ha  is Principal Solution Architect at BigBear.ai who has more than 20 years of IT experiences in designing and implementing innovative business solutions.  He has a wide range of skills and experiences in enterprise IT system solutioning for on-prem, cloud, and hybrid environment.  He is specialized in microservice system design including migrating monolithic applications to microservice architecture.
Posted in Technology.
BigBear.ai
  • Home
  • Industries
  • Solutions
  • Company
  • Careers
  • Blog
  • Investor Relations
  • Contact
  • Twitter
  • Facebook
  • Linkedin
  • Google My business for BigBear.ai
1-410-312-0885
[email protected]
  • Privacy Policy
  • Terms of Use
  • Accessibility
  • Site Map
© BigBear.ai 2023
We value your privacy
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Privacy Policy | Do not sell my personal information
AcceptCookie Settings
Manage Consent

Cookies Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
JSESSIONIDsessionThe JSESSIONID cookie is used by New Relic to store a session identifier so that New Relic can monitor session counts for an application.
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
CookieDurationDescription
__atuvc1 year 1 monthAddThis sets this cookie to ensure that the updated count is seen when one shares a page and returns to it, before the share count cache is updated.
__atuvs30 minutesAddThis sets this cookie to ensure that the updated count is seen when one shares a page and returns to it, before the share count cache is updated.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
CookieDurationDescription
_ga2 yearsThe _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_ga_NK4L4Q320Q2 yearsThis cookie is installed by Google Analytics.
_gat_gtag_UA_163894009_21 minuteSet by Google to distinguish users.
_gid1 dayInstalled by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
at-randneverAddThis sets this cookie to track page visits, sources of traffic and share counts.
CONSENT2 yearsYouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
uvc1 year 1 monthSet by addthis.com to determine the usage of addthis.com service.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
CookieDurationDescription
f5avraaaaaaaaaaaaaaaa_session_sessionbusinesswire.com cookie
loc1 year 1 monthAddThis sets this geolocation cookie to help understand the location of users who share the information.
VISITOR_INFO1_LIVE5 months 27 daysA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSCsessionYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.
yt-remote-connected-devicesneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt-remote-device-idneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
Save & Accept