Mar 29, 2018
System Design: Create a url shortening service (Part 5): Performance testing on AWS
This is part of a blog series where we design, develop, optimize, deploy and test a URL shortener service from scratch.
- Part 1: Overview
- Part 2: Design the write API
- Part 3: Read API, Load testing and Performance improvement
- Part 4: Deploy to AWS
- Part 5: Performance testing on AWS
In this article, we’ll talk about:
Install and run JMeter
To avoid latency in the result, Let’s spin up another EC2 instance (T2 Medium) in the same region.
Install JMeter
Change the heap size of JMeter. Default is 512Mb. I have put it as 80% of RAM. vi apache-jmeter-4.0/bin/jmeter . Change heap size : “${HEAP:=”-Xms4g -Xmx6g -XX:MaxMetaspaceSize=256m”}”
Also git clone the repo as it has JMeter test plan. Important settings are:
- 250 Users with 250 sec ramp up time
- Constant Throughput Timer at 60K requests/minute
- I have 4 EC2 instances running before starting the JMeter test. Easiest way to achieve this is to change the
DesriedandMinsetting in auto scaling group.
Start JMeter tests./apache-jmeter-4.0/bin/./jmeter -n -t tinyUrl/tinyUrl.jmx
After certain time, EC2 instances would start coming up. For me, AWS peaked at 8 instances
I ran the test for ~19 minutes with more than a total of 10 million request.
One of the target we set for ourself at the start of this series was the ability to serve 10K request/second. From the above log we can see we are able to serve that 👍
scp the log file scp -i “<your pem file>” ubuntu@<ip where JMeter is running>:/home/ubuntu/perf.jt in your box. Load it into the JMeter GUI.
We fulfilled our target of responding to 90% of all the request in less than 10 ms 👍
Stop the JMeter and EC2 instances would start shutting down till they reach the Min setting in the auto scaling group.
We can also look at the Activity History inside Auto Scaling Groups
Other Observations
100% cache hit rate: All of the data is coming from Redis which is really fast and we have 100% cache hit ratio. In real life, it would not be the case. This would affect our performance. If the postgres data size is greater than what redis can store, it would impact the performance of the application as well.
Read heavy application: We have only performed performance test on read as this type of application is supposed to be read heavy.
Scaling write heavy applications is harder: Write heavy applications are harder to scale. A typical workflow would look like
- Do vertical scaling
- Separate read server from write servers
- Configure the database in master-master config
- If possible, write async way.
Redundancy: We have many single point of failures.
- Single postgres server
- Single Redis
- Complete application is hosted in the same region of AWS
Latency: Users using this application from other regions of the world will face higher response time because of the latency.
Automation: Currently a lot of steps are manual in this setup. We can use Chef/Puppet etc to automate this whole process
If you found this story interesting or useful, please support it by clapping it👏
