FuzzBench: Fuzzer Benchmarking As a Service

FuzzBench is a free service that evaluates fuzzers on a wide variety of real-world benchmarks, at Google scale. The goal of FuzzBench is to make it painless to rigorously evaluate fuzzing research and make fuzzing research easier for the community to adopt. We invite members of the research community to contribute their fuzzers and give us feedback on improving our evaluation techniques.

FuzzBench provides:

  • An easy API for integrating fuzzers.
  • Benchmarks from real-world projects. FuzzBench can use any OSS-Fuzz project as a benchmark.
  • A reporting library that produces reports with graphs and statistical tests to help you understand the significance of results.

To participate, submit your fuzzer to run on the FuzzBench platform by following our simple guide. After your integration is accepted, we will run a large-scale experiment using your fuzzer and generate a report comparing your fuzzer to others, such as AFL and libFuzzer. See a sample report.

Overview

FuzzBench Service diagram The process works like this:

  1. A fuzzer developer (or someone else interested) integrates a fuzzer with FuzzBench.
  2. The integration is merged into the FuzzBench repo.
  3. FuzzBench runs an experiment with the new fuzzer on the benchmarks.
  4. FuzzBench publishes a report comparing the performance of the fuzzer to other fuzzers both on individual benchmarks and in aggregate.

Adding a fuzzer

Follow this guide to add a fuzzer to FuzzBench, submit it, and get benchmark results.

Sample report

You can view our sample report here and our periodically generated reports here. The sample report is generated using 10 fuzzers against 24 real-world benchmarks, with 20 trials each and over a duration of 24 hours. The raw data in compressed CSV format can be found at the end of the report.

When analyzing reports, we recommend:

  • Checking the strengths and weaknesses of a fuzzer against various benchmarks.
  • Looking at aggregate results to understand the overall significance of the result.

Please provide feedback on any inaccuracies and potential improvements (such as integration changes, new benchmarks, etc.) by opening a GitHub issue here.

Contacts

Join our mailing list for discussions and announcements, or send us a private email at fuzzbench@google.com.