The Yahoo! Cloud Serving Benchmark Tool
The Yahoo! Cloud Serving Benchmark (YCSB), an open source framework for evaluating and comparing the performance of multiple types of data-serving systems (including NoSQL stores such as Apache HBase, Apache Cassandra, Redis, MongoDB, and Voldemort), has long been the de facto open standard for this purpose.
It has two parts:
1. The YCSB Client, an extensible workload generator.
2. The core workloads, a set of workload scenarios to be executed by the generator.
The core workloads provide a well-rounded picture of a system’s performance, and the client is extensible so that you can define additional workloads to examine system aspects or application scenarios not covered by the core workload. The client can also be extended to benchmark different databases. YCSB ships with bindings for a long list of databases including HBase, Cassandra, Apache Accumulo, MongoDB, and Voldemort, and you can add support for a different data store by writing an interface layer.
To benchmark multiple data stores and compare them, you can install those data stores within a single deployment (or alternatively on multiple instances of a selected hardware configuration). Then, the same workloads can be run against the various data stores. Next, plot the performance of each system, to see their relative performance profiles. One example of a good visualization to try is latency versus throughput curves.
To run a YCSB workload:
-Set up the database you will be testing.
-Create the table and load data into it, either using YCSB or manually.
-Choose the workload in YCSB. Some of the available workloads include read transactions, update transactions, and mixed transactions.
-Choose runtime parameters, such as the request distribution and a number of threads.
-Invoke the YCSB client to apply the workload.
-When it is done, the client will report throughput and latency results.