You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
409 lines
19 KiB
409 lines
19 KiB
===============================
|
|
Understanding the status screen
|
|
===============================
|
|
|
|
This document provides an overview of the status screen - plus tips for
|
|
troubleshooting any warnings and red text shown in the UI. See README for
|
|
the general instruction manual.
|
|
|
|
0) A note about colors
|
|
----------------------
|
|
|
|
The status screen and error messages use colors to keep things readable and
|
|
attract your attention to the most important details. For example, red almost
|
|
always means "consult this doc" :-)
|
|
|
|
Unfortunately, the UI will render correctly only if your terminal is using
|
|
traditional un*x palette (white text on black background) or something close
|
|
to that.
|
|
|
|
If you are using inverse video, you may want to change your settings, say:
|
|
|
|
- For GNOME Terminal, go to Edit > Profile preferences, select the "colors"
|
|
tab, and from the list of built-in schemes, choose "white on black".
|
|
|
|
- For the MacOS X Terminal app, open a new window using the "Pro" scheme via
|
|
the Shell > New Window menu (or make "Pro" your default).
|
|
|
|
Alternatively, if you really like your current colors, you can edit config.h
|
|
to comment out USE_COLORS, then do 'make clean all'.
|
|
|
|
I'm not aware of any other simple way to make this work without causing
|
|
other side effects - sorry about that.
|
|
|
|
With that out of the way, let's talk about what's actually on the screen...
|
|
|
|
1) Process timing
|
|
-----------------
|
|
|
|
+----------------------------------------------------+
|
|
| run time : 0 days, 8 hrs, 32 min, 43 sec |
|
|
| last new path : 0 days, 0 hrs, 6 min, 40 sec |
|
|
| last uniq crash : none seen yet |
|
|
| last uniq hang : 0 days, 1 hrs, 24 min, 32 sec |
|
|
+----------------------------------------------------+
|
|
|
|
This section is fairly self-explanatory: it tells you how long the fuzzer has
|
|
been running and how much time has elapsed since its most recent finds. This is
|
|
broken down into "paths" (a shorthand for test cases that trigger new execution
|
|
patterns), crashes, and hangs.
|
|
|
|
When it comes to timing: there is no hard rule, but most fuzzing jobs should be
|
|
expected to run for days or weeks; in fact, for a moderately complex project, the
|
|
first pass will probably take a day or so. Every now and then, some jobs
|
|
will be allowed to run for months.
|
|
|
|
There's one important thing to watch out for: if the tool is not finding new
|
|
paths within several minutes of starting, you're probably not invoking the
|
|
target binary correctly and it never gets to parse the input files we're
|
|
throwing at it; another possible explanations are that the default memory limit
|
|
(-m) is too restrictive, and the program exits after failing to allocate a
|
|
buffer very early on; or that the input files are patently invalid and always
|
|
fail a basic header check.
|
|
|
|
If there are no new paths showing up for a while, you will eventually see a big
|
|
red warning in this section, too :-)
|
|
|
|
2) Overall results
|
|
------------------
|
|
|
|
+-----------------------+
|
|
| cycles done : 0 |
|
|
| total paths : 2095 |
|
|
| uniq crashes : 0 |
|
|
| uniq hangs : 19 |
|
|
+-----------------------+
|
|
|
|
The first field in this section gives you the count of queue passes done so far
|
|
- that is, the number of times the fuzzer went over all the interesting test
|
|
cases discovered so far, fuzzed them, and looped back to the very beginning.
|
|
Every fuzzing session should be allowed to complete at least one cycle; and
|
|
ideally, should run much longer than that.
|
|
|
|
As noted earlier, the first pass can take a day or longer, so sit back and
|
|
relax. If you want to get broader but more shallow coverage right away, try
|
|
the -d option - it gives you a more familiar experience by skipping the
|
|
deterministic fuzzing steps. It is, however, inferior to the standard mode in
|
|
a couple of subtle ways.
|
|
|
|
To help make the call on when to hit Ctrl-C, the cycle counter is color-coded.
|
|
It is shown in magenta during the first pass, progresses to yellow if new finds
|
|
are still being made in subsequent rounds, then blue when that ends - and
|
|
finally, turns green after the fuzzer hasn't been seeing any action for a
|
|
longer while.
|
|
|
|
The remaining fields in this part of the screen should be pretty obvious:
|
|
there's the number of test cases ("paths") discovered so far, and the number of
|
|
unique faults. The test cases, crashes, and hangs can be explored in real-time
|
|
by browsing the output directory, as discussed in the README.
|
|
|
|
3) Cycle progress
|
|
-----------------
|
|
|
|
+-------------------------------------+
|
|
| now processing : 1296 (61.86%) |
|
|
| paths timed out : 0 (0.00%) |
|
|
+-------------------------------------+
|
|
|
|
This box tells you how far along the fuzzer is with the current queue cycle: it
|
|
shows the ID of the test case it is currently working on, plus the number of
|
|
inputs it decided to ditch because they were persistently timing out.
|
|
|
|
The "*" suffix sometimes shown in the first line means that the currently
|
|
processed path is not "favored" (a property discussed later on, in section 6).
|
|
|
|
If you feel that the fuzzer is progressing too slowly, see the note about the
|
|
-d option in section 2 of this doc.
|
|
|
|
4) Map coverage
|
|
---------------
|
|
|
|
+--------------------------------------+
|
|
| map density : 10.15% / 29.07% |
|
|
| count coverage : 4.03 bits/tuple |
|
|
+--------------------------------------+
|
|
|
|
The section provides some trivia about the coverage observed by the
|
|
instrumentation embedded in the target binary.
|
|
|
|
The first line in the box tells you how many branch tuples we have already
|
|
hit, in proportion to how much the bitmap can hold. The number on the left
|
|
describes the current input; the one on the right is the value for the entire
|
|
input corpus.
|
|
|
|
Be wary of extremes:
|
|
|
|
- Absolute numbers below 200 or so suggest one of three things: that the
|
|
program is extremely simple; that it is not instrumented properly (e.g.,
|
|
due to being linked against a non-instrumented copy of the target
|
|
library); or that it is bailing out prematurely on your input test cases.
|
|
The fuzzer will try to mark this in pink, just to make you aware.
|
|
|
|
- Percentages over 70% may very rarely happen with very complex programs
|
|
that make heavy use of template-generated code.
|
|
|
|
Because high bitmap density makes it harder for the fuzzer to reliably
|
|
discern new program states, I recommend recompiling the binary with
|
|
AFL_INST_RATIO=10 or so and trying again (see env_variables.txt).
|
|
|
|
The fuzzer will flag high percentages in red. Chances are, you will never
|
|
see that unless you're fuzzing extremely hairy software (say, v8, perl,
|
|
ffmpeg).
|
|
|
|
The other line deals with the variability in tuple hit counts seen in the
|
|
binary. In essence, if every taken branch is always taken a fixed number of
|
|
times for all the inputs we have tried, this will read "1.00". As we manage
|
|
to trigger other hit counts for every branch, the needle will start to move
|
|
toward "8.00" (every bit in the 8-bit map hit), but will probably never
|
|
reach that extreme.
|
|
|
|
Together, the values can be useful for comparing the coverage of several
|
|
different fuzzing jobs that rely on the same instrumented binary.
|
|
|
|
5) Stage progress
|
|
-----------------
|
|
|
|
+-------------------------------------+
|
|
| now trying : interest 32/8 |
|
|
| stage execs : 3996/34.4k (11.62%) |
|
|
| total execs : 27.4M |
|
|
| exec speed : 891.7/sec |
|
|
+-------------------------------------+
|
|
|
|
This part gives you an in-depth peek at what the fuzzer is actually doing right
|
|
now. It tells you about the current stage, which can be any of:
|
|
|
|
- calibration - a pre-fuzzing stage where the execution path is examined
|
|
to detect anomalies, establish baseline execution speed, and so on. Executed
|
|
very briefly whenever a new find is being made.
|
|
|
|
- trim L/S - another pre-fuzzing stage where the test case is trimmed to the
|
|
shortest form that still produces the same execution path. The length (L)
|
|
and stepover (S) are chosen in general relationship to file size.
|
|
|
|
- bitflip L/S - deterministic bit flips. There are L bits toggled at any given
|
|
time, walking the input file with S-bit increments. The current L/S variants
|
|
are: 1/1, 2/1, 4/1, 8/8, 16/8, 32/8.
|
|
|
|
- arith L/8 - deterministic arithmetics. The fuzzer tries to subtract or add
|
|
small integers to 8-, 16-, and 32-bit values. The stepover is always 8 bits.
|
|
|
|
- interest L/8 - deterministic value overwrite. The fuzzer has a list of known
|
|
"interesting" 8-, 16-, and 32-bit values to try. The stepover is 8 bits.
|
|
|
|
- extras - deterministic injection of dictionary terms. This can be shown as
|
|
"user" or "auto", depending on whether the fuzzer is using a user-supplied
|
|
dictionary (-x) or an auto-created one. You will also see "over" or "insert",
|
|
depending on whether the dictionary words overwrite existing data or are
|
|
inserted by offsetting the remaining data to accommodate their length.
|
|
|
|
- havoc - a sort-of-fixed-length cycle with stacked random tweaks. The
|
|
operations attempted during this stage include bit flips, overwrites with
|
|
random and "interesting" integers, block deletion, block duplication, plus
|
|
assorted dictionary-related operations (if a dictionary is supplied in the
|
|
first place).
|
|
|
|
- splice - a last-resort strategy that kicks in after the first full queue
|
|
cycle with no new paths. It is equivalent to 'havoc', except that it first
|
|
splices together two random inputs from the queue at some arbitrarily
|
|
selected midpoint.
|
|
|
|
- sync - a stage used only when -M or -S is set (see parallel_fuzzing.txt).
|
|
No real fuzzing is involved, but the tool scans the output from other
|
|
fuzzers and imports test cases as necessary. The first time this is done,
|
|
it may take several minutes or so.
|
|
|
|
The remaining fields should be fairly self-evident: there's the exec count
|
|
progress indicator for the current stage, a global exec counter, and a
|
|
benchmark for the current program execution speed. This may fluctuate from
|
|
one test case to another, but the benchmark should be ideally over 500 execs/sec
|
|
most of the time - and if it stays below 100, the job will probably take very
|
|
long.
|
|
|
|
The fuzzer will explicitly warn you about slow targets, too. If this happens,
|
|
see the perf_tips.txt file included with the fuzzer for ideas on how to speed
|
|
things up.
|
|
|
|
6) Findings in depth
|
|
--------------------
|
|
|
|
+--------------------------------------+
|
|
| favored paths : 879 (41.96%) |
|
|
| new edges on : 423 (20.19%) |
|
|
| total crashes : 0 (0 unique) |
|
|
| total tmouts : 24 (19 unique) |
|
|
+--------------------------------------+
|
|
|
|
This gives you several metrics that are of interest mostly to complete nerds.
|
|
The section includes the number of paths that the fuzzer likes the most based
|
|
on a minimization algorithm baked into the code (these will get considerably
|
|
more air time), and the number of test cases that actually resulted in better
|
|
edge coverage (versus just pushing the branch hit counters up). There are also
|
|
additional, more detailed counters for crashes and timeouts.
|
|
|
|
Note that the timeout counter is somewhat different from the hang counter; this
|
|
one includes all test cases that exceeded the timeout, even if they did not
|
|
exceed it by a margin sufficient to be classified as hangs.
|
|
|
|
7) Fuzzing strategy yields
|
|
--------------------------
|
|
|
|
+-----------------------------------------------------+
|
|
| bit flips : 57/289k, 18/289k, 18/288k |
|
|
| byte flips : 0/36.2k, 4/35.7k, 7/34.6k |
|
|
| arithmetics : 53/2.54M, 0/537k, 0/55.2k |
|
|
| known ints : 8/322k, 12/1.32M, 10/1.70M |
|
|
| dictionary : 9/52k, 1/53k, 1/24k |
|
|
| havoc : 1903/20.0M, 0/0 |
|
|
| trim : 20.31%/9201, 17.05% |
|
|
+-----------------------------------------------------+
|
|
|
|
This is just another nerd-targeted section keeping track of how many paths we
|
|
have netted, in proportion to the number of execs attempted, for each of the
|
|
fuzzing strategies discussed earlier on. This serves to convincingly validate
|
|
assumptions about the usefulness of the various approaches taken by afl-fuzz.
|
|
|
|
The trim strategy stats in this section are a bit different than the rest.
|
|
The first number in this line shows the ratio of bytes removed from the input
|
|
files; the second one corresponds to the number of execs needed to achieve this
|
|
goal. Finally, the third number shows the proportion of bytes that, although
|
|
not possible to remove, were deemed to have no effect and were excluded from
|
|
some of the more expensive deterministic fuzzing steps.
|
|
|
|
8) Path geometry
|
|
----------------
|
|
|
|
+---------------------+
|
|
| levels : 5 |
|
|
| pending : 1570 |
|
|
| pend fav : 583 |
|
|
| own finds : 0 |
|
|
| imported : 0 |
|
|
| stability : 100.00% |
|
|
+---------------------+
|
|
|
|
The first field in this section tracks the path depth reached through the
|
|
guided fuzzing process. In essence: the initial test cases supplied by the
|
|
user are considered "level 1". The test cases that can be derived from that
|
|
through traditional fuzzing are considered "level 2"; the ones derived by
|
|
using these as inputs to subsequent fuzzing rounds are "level 3"; and so forth.
|
|
The maximum depth is therefore a rough proxy for how much value you're getting
|
|
out of the instrumentation-guided approach taken by afl-fuzz.
|
|
|
|
The next field shows you the number of inputs that have not gone through any
|
|
fuzzing yet. The same stat is also given for "favored" entries that the fuzzer
|
|
really wants to get to in this queue cycle (the non-favored entries may have to
|
|
wait a couple of cycles to get their chance).
|
|
|
|
Next, we have the number of new paths found during this fuzzing section and
|
|
imported from other fuzzer instances when doing parallelized fuzzing; and the
|
|
extent to which identical inputs appear to sometimes produce variable behavior
|
|
in the tested binary.
|
|
|
|
That last bit is actually fairly interesting: it measures the consistency of
|
|
observed traces. If a program always behaves the same for the same input data,
|
|
it will earn a score of 100%. When the value is lower but still shown in purple,
|
|
the fuzzing process is unlikely to be negatively affected. If it goes into red,
|
|
you may be in trouble, since AFL will have difficulty discerning between
|
|
meaningful and "phantom" effects of tweaking the input file.
|
|
|
|
Now, most targets will just get a 100% score, but when you see lower figures,
|
|
there are several things to look at:
|
|
|
|
- The use of uninitialized memory in conjunction with some intrinsic sources
|
|
of entropy in the tested binary. Harmless to AFL, but could be indicative
|
|
of a security bug.
|
|
|
|
- Attempts to manipulate persistent resources, such as left over temporary
|
|
files or shared memory objects. This is usually harmless, but you may want
|
|
to double-check to make sure the program isn't bailing out prematurely.
|
|
Running out of disk space, SHM handles, or other global resources can
|
|
trigger this, too.
|
|
|
|
- Hitting some functionality that is actually designed to behave randomly.
|
|
Generally harmless. For example, when fuzzing sqlite, an input like
|
|
'select random();' will trigger a variable execution path.
|
|
|
|
- Multiple threads executing at once in semi-random order. This is harmless
|
|
when the 'stability' metric stays over 90% or so, but can become an issue
|
|
if not. Here's what to try:
|
|
|
|
- Use afl-clang-fast from llvm_mode/ - it uses a thread-local tracking
|
|
model that is less prone to concurrency issues,
|
|
|
|
- See if the target can be compiled or run without threads. Common
|
|
./configure options include --without-threads, --disable-pthreads, or
|
|
--disable-openmp.
|
|
|
|
- Replace pthreads with GNU Pth (https://www.gnu.org/software/pth/), which
|
|
allows you to use a deterministic scheduler.
|
|
|
|
- In persistent mode, minor drops in the "stability" metric can be normal,
|
|
because not all the code behaves identically when re-entered; but major
|
|
dips may signify that the code within __AFL_LOOP() is not behaving
|
|
correctly on subsequent iterations (e.g., due to incomplete clean-up or
|
|
reinitialization of the state) and that most of the fuzzing effort goes
|
|
to waste.
|
|
|
|
The paths where variable behavior is detected are marked with a matching entry
|
|
in the <out_dir>/queue/.state/variable_behavior/ directory, so you can look
|
|
them up easily.
|
|
|
|
9) CPU load
|
|
-----------
|
|
|
|
[cpu: 25%]
|
|
|
|
This tiny widget shows the apparent CPU utilization on the local system. It is
|
|
calculated by taking the number of processes in the "runnable" state, and then
|
|
comparing it to the number of logical cores on the system.
|
|
|
|
If the value is shown in green, you are using fewer CPU cores than available on
|
|
your system and can probably parallelize to improve performance; for tips on
|
|
how to do that, see parallel_fuzzing.txt.
|
|
|
|
If the value is shown in red, your CPU is *possibly* oversubscribed, and
|
|
running additional fuzzers may not give you any benefits.
|
|
|
|
Of course, this benchmark is very simplistic; it tells you how many processes
|
|
are ready to run, but not how resource-hungry they may be. It also doesn't
|
|
distinguish between physical cores, logical cores, and virtualized CPUs; the
|
|
performance characteristics of each of these will differ quite a bit.
|
|
|
|
If you want a more accurate measurement, you can run the afl-gotcpu utility
|
|
from the command line.
|
|
|
|
10) Addendum: status and plot files
|
|
-----------------------------------
|
|
|
|
For unattended operation, some of the key status screen information can be also
|
|
found in a machine-readable format in the fuzzer_stats file in the output
|
|
directory. This includes:
|
|
|
|
- start_time - unix time indicating the start time of afl-fuzz
|
|
- last_update - unix time corresponding to the last update of this file
|
|
- fuzzer_pid - PID of the fuzzer process
|
|
- cycles_done - queue cycles completed so far
|
|
- execs_done - number of execve() calls attempted
|
|
- execs_per_sec - current number of execs per second
|
|
- paths_total - total number of entries in the queue
|
|
- paths_found - number of entries discovered through local fuzzing
|
|
- paths_imported - number of entries imported from other instances
|
|
- max_depth - number of levels in the generated data set
|
|
- cur_path - currently processed entry number
|
|
- pending_favs - number of favored entries still waiting to be fuzzed
|
|
- pending_total - number of all entries waiting to be fuzzed
|
|
- stability - percentage of bitmap bytes that behave consistently
|
|
- variable_paths - number of test cases showing variable behavior
|
|
- unique_crashes - number of unique crashes recorded
|
|
- unique_hangs - number of unique hangs encountered
|
|
- command_line - full command line used for the fuzzing session
|
|
- slowest_exec_ms- real time of the slowest execution in ms
|
|
- peak_rss_mb - max rss usage reached during fuzzing in mb
|
|
|
|
Most of these map directly to the UI elements discussed earlier on.
|
|
|
|
On top of that, you can also find an entry called 'plot_data', containing a
|
|
plottable history for most of these fields. If you have gnuplot installed, you
|
|
can turn this into a nice progress report with the included 'afl-plot' tool.
|