Grafana dashboard that we will setup in this blog
Grafana dashboard that we will setup in this blog
Grafana dashboard that we will setup in this blog

In this blog post, we will setup Kafka and Zookeeper cluster with following requirements:

  • System will handle 15K events per second of 1 KB each
  • System will be reliable and will not loose data
  • System will support monitoring with Prometheus and Grafana
  • System will be load tested with JMeter

Things that we will not consider in this blog:

  • No topic is partitioned by message key.
  • Consumer and their throughput/partition requirement
  • Security related changes

Kafka system requirements:

  • CPU & Memory: Since Kafka is light on the CPU, we will use m5.xlarge


Image Source: pixabay

Events are lifeline of an Event Driven Architecture. All microservices rely on the data in these events to be accurate and complete. One of the many ways to implement Event Driven Architecture is through Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS).


How to set up a logging Infrastructureusing using ElasticSearch, Fluentd, and Kibana.

Photo by Ugne Vasyliute on Unsplash

Setting up the right logging infrastructure helps us in finding what happened, debugging and monitoring the application. At a very basic level, we should expect the following from our infrastructure:

  • Ability to free text search on our logs
  • Ability to search for specific api logs
  • Ability to search per statusCode of all the APIs
  • System should scale as we add more data into our logs

Architecture


Learn how to CI/CD with GitHub Actions and Docker

Photo by Jannes Glas on Unsplash

This article will cover the following:

  • Use Docker instead of bare metal deployment
  • Use GitHub actions for continuous integration of your app
  • Use GitHub actions for continuous deployment by pushing the Docker image to a Docker registry (Docker Hub)

Our workflow will look like this


Rules for an optimized caching mechanism

For every complex problem, there is an answer that is clear, simple, and wrong.

source

Let’s look at how a typical cache miss workflow looks like


source

Why we might need one?

There can be scenarios where we have some code which is used by many micro-services. For example, we can have a code on how to make interservice calls with the circuit breaker, logging, timeouts and much more. We want every service in our infrastructure to adhere to same guidelines. It would be great if we could install these as npm packages and version manage the changes. This will give more confidence in rolling out new and breaking changes thus enabling higher velocity in the product development.

We will be using Verdaccio as a npm registry. From its readme.md

Verdaccio is…


Time series data in its most basic form is a sequence of data points measuring a particular thing over a period of time. For more information, you can read the first part of this post What the heck is time-series data?

Our use case is pretty simple. We have users who would like to store their medical observables either manually or through some device. At the very basic, think about saving heart rate/blood pressure/sugar several times a day with few hundred thousand active users a day (with a possibility to a much larger traffic). …


We were working for a telecom client where we were rolling out offers for our customers. They had tons of business logic based on which the user will be given an offer. In addition to that, they would create more categories of offer as time passes by. We saw the code that looked like this

Strategy Pattern: Not using strategy pattern

Every-time a new offer type was introduced, or existing rules needed modification, we needed to change User class.

Calling this class looked something like this

We have following issues with the code:


Choose between http/1.1 and gRPC. TL; DR: Use gRPC

Communicate effectively

How microservices communicate with each other can affect performance and scalability of the application. Communication between services can be synchronous or asynchronous. For this blog post, we will focus on synchronous. We have 2 common protocols at our disposal

  • HTTP/1.1: The “default” http call
  • gRPC: High performance remote procedure call (RPC framework). gRPC.io has a comprehensive documentation and how it works.

Let’s look at some sample code and run some performance test to pick our choices.

Non-cluster mode

Let’s start small and try to get one microservice talk to another one.


Design pattern for “when shit happens”

Problem Statement

We have a serviceA which has two APIs

  • /data which depends on serviceB
  • /data2 does not depend on any external service

Abhinav Dhasmana

Senior Staff Engineer @freshworks. Ex-McKinsey/Microsoft/Slideshare/SAP, Tech Enthusiast, Passionate about India. Opinions are mine

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store