Benchmark Overview

About

With this performance benchmark we wanted to show how well Webiny can handle the load, what performance you can expect, what cost is associated with different flows as well as what are the factors to additionally scale Webiny beyond the numbers shown here.

Benchmark Structure

Webiny has several apps and APIs. For this benchmark we decided to pick 3 actions we felt are most relevant:

1. Call the Headless CMS Manage API and insert a new record (view results)
2. Call the Headless CMS Preview API and retrieve a record (view results)
3. Deliver a page created by the Page Builder (view results)

The tests on the Headless CMS API provide a good reference on both how the Headless CMS scales and performs, but the same performance can be expected if you were to create a custom API endpoint on Webiny, not using the Headless CMS. The performance of the Preview API is expected to be identical to the Read API. We picked the Preview API because it was easier to setup the test since we didn’t need to publish records after we inserted them.

The Page Builder page delivery test is used to showcase how Webiny can serve a static page.

Execution

We deployed Webiny into an AWS region, in our case eu-west-2 (London). This is the basic vanilla Webiny setup you would get if you follow our install Webiny tutorial. We ran all 3 tests using this setup. Afterwards we modified one of the infrastructure resources and repeated tests 1 and 2. All the details can be found in the test results.

To run the test we deployed an EC2 machine in the same region. This machine was used to generate the load for the test. The machine instance type we used was c5.18xlarge. We used a fairly powerful instance as we needed to generate a lot of load for some of the tests and didn’t want to be CPU, memory or network bound.

The machine was a base Ubuntu instance with only Java and Apache JMeter installed. The tests were executed using the Apache JMeter CLI. For each of the tests we are providing a jmx file so you can run the same on your end.

Cost Benchmark

Besides covering the performance aspect of this benchmark, we also wanted to focus on the cost. A lot of people have questions and concerns when it comes to estimating cost of serverless infrastructure. To help answer those questions when it comes to Webiny, you can use the numbers from this benchmark to help you estimate your potential cost.

Scaling the Performance

The numbers we presented in our tests are not the maximum that Webiny can reach. Many of the serverless services are designed to scale far beyond our reported numbers. To achieve this scale, a “real life” example is needed. Users need to be accessing the API and the website from different networks and locations. Unfortunately creating such a load test is very difficult and in our test we spread the load across several IPs but that’s far from enough to achieve real life production scale.

If you require a greater scale, we recommend you reach out to us via slack and we’ll be happy to help you out.

Webiny vs Others

Webiny’s numbers are not directly comparable to systems running on virtual machines or containers. Webiny’s architecture is fault tolerant in almost all of it’s parts out of the box. The only exception is Elasticsearch, which can be made highly available by deploying to multiple AZs.

Running on serverless services overall improves the reliability and security of your application and makes the infrastructure easier to maintain. So if you do plan to make comparisons, we recommend you take all appropriate aspects into account.

Replicate the Test

All test files, reports from our run and instructions are available in this repository here: http://github.com/webiny/benchmark .