Challenge Overview
In this challenge we are looking for a Proof of Concept solution to demonstrate using filters in the Logstash pipeline. Imagine that there is a Spring web application and a couple of microservices that all produce logs in a defined format. The goal is to create a Logstash pipeline that analyzes those logs (multiple inputs), filters and aggregates the records and finaly outputs some useful information.
Creating the sample web application and the microservices as well as the logstash pipeline is in the scope of this challenge. Logstash filters should perorm some filtering and aggregation based on configured keywords. There are no hard requirements on the web application and microservice features, but bear in mind that the overall pipeline should demonstrate some useful real world scenario.
Here is one sample scenarios you can use but you are free to construct your own scenarios
Analyzing request response times for the web application, with a breakdown for each call from the web app to the microservices. In this case the remote calls are using request tagging for correlating the remote calls to the originating call to the web application (https://github.com/stevesaliman/spring-boot-starter-request-correlation). Keyword filtering could be used to filter only requests to speciffic endpoints
General Requirements
Services and the web application should be developed using Spring Boot, latest stable version. Unit tests, swagger and postman files for the services are NOT required as these are used only for demonstration purposes. Annotations are preferred over xml configuration where possible.
Services should produce logs in various outputs (file logs, syslog, db, etc) and they should be used by logstash pipeline with multiple inputs.
Deployment
Create a Docker image for the microservices. Images should have just the minimal environment and should start only the app process (so don't make the entrypoint be a bash shell). Application code should be embeded into the image, and not mapped as a volume. You are free to determine how the logs should be integrated (rsyslog, Filebeats, etc)
Include instructions for deploying the service locally (both with docker and within development environment), and Heroku (docker). If your log aggregation approach won't work on Heroku, you can skip this requirement, but do explain the reasoning behind the limitations.
Final Submission Guidelines
1) Web application
2) Microservices
3) Logstash configuration
4) Deployment guide
5) Verification guide