Analyzing fuzzer performance

ClusterFuzz automates fuzzing as much as possible, but it’s the responsibility of users to write and maintain fuzzers that can find security bugs. This page gives recommendations on how to analyze the performance of the fuzzers running on ClusterFuzz.

Note: this page only applies to fuzz targets doing coverage guided fuzzing with libFuzzer or AFL.

• Analyzing fuzzer performance
  • When to analyze fuzz target performance
  • Performance factors
  • Fuzzer stats
  • Coverage report
  • Fuzzer logs

---

When to analyze fuzz target performance

It’s important to regularly monitor the performance of fuzz targets, especially after a new target is created. If a target finds many new crashes, fixing them is probably more important than analyzing performance. But if a target has not found any crashes in a while, you should probably examine its performance.

Performance factors

• Speed is crucial for fuzzing. There is no minimum threshold. The faster a fuzz target generates testcases, the better.
• Code coverage should grow over time. A fuzz target should be continuously generating new “interesting” testcases that exercise various parts of the target program.
• Blocking issues should be resolved. If a fuzz target frequently reports a Timeout, Out-of-Memory, or other crashes, it will be blocking the target from finding more interesting issues.

Fuzzer stats

The Fuzzer stats page provides metrics on fuzzer performance. Using the filters on the page, you can see how those metrics (e.g. execution speed or number of crashes) change over time (if you choose “Group by Day”). You can compare different fuzzers to one another using “Group by Fuzzer”. There is also a “Group by Time” filter that shows fuzzer stats as charts rather than raw numbers.

This feature requires a production setup, as fuzzer stats are stored in BigQuery. The stats are usually delayed by up to 24 hours, as data is uploaded to BigQuery once a day.

Coverage report

Code coverage is a very important metric for evaluating fuzzer performance. Looking at the code coverage report, you can see which exact parts of the target program are tested by the fuzzer and which parts are never executed. If you set up a code coverage builder for ClusterFuzz, you can find links to the coverage reports on the Fuzzer stats page. Otherwise, you can generate code coverage reports locally. For C and C++ targets, we recommend using Clang Source-based Code Coverage.

Fuzzer logs

If none of the above gives you enough information about fuzzer performance, looking into the fuzzer logs may help. On the fuzzer stats page, you can find a link to the GCS bucket storing the logs. You can also navigate to that manually using the Google Cloud Storage web interface.