Fuzzing Artifacts Archive 1.0
fuzzing-artifacts.tar[.xz|.bz2|.gz]
Abstract
This specification describes the format of the fuzzing artifacts
archive for usage with CI FUZZ on https://app.code-intelligence.com
or Code Intelligence on-premise solutions.
Table of Contents
1. Archive Contents
2. Fuzz targets
2.1 Fuzz targets for coverage measurement
3. Metadata
3.1. General properties
3.2. Fuzzer properties
Appendix A. Schema of cifuzz.yaml
Appendix B. Example of an exception policy JSON file
Appendix C. Schema of an exception policy JSON file
1. Archive Contents
The fuzzing artifacts archive MUST be a Tar archive (.tar), or a
[xz|bz2|gz]-compressed Tar archive (.tar.xz, .tar.bz2, .tar.gz).
It MUST contain the following files and directories:
cifuzz.yaml
work_dir/
All of the aforementioned directories MAY be empty. The archive MAY
contain arbitrary other files and directories. It SHOULD however be
limited in size to some degree, as it might be copied and
transferred several times during fuzzing.
2. Fuzz targets
Every fuzz target is a binary in the fuzzing artifact archive and
can be dependent on libraries (see 'library_paths' in Section 3.2.)
and other artifacts in the same archive. Every fuzz target needs an
entry in the fuzz target list 'fuzzers' in cifuzz.yaml.
2.1 Fuzz targets for coverage measurement
In order to measure the coverage of a fuzzing run, an additional
fuzz target for coverage measurement has to be configured. It has
to have the same name as the fuzz target it is measuring coverage
for and must use a coverage engine, currently only "LLVM_COV".
A fuzz target for coverage measurement can use the same binary as
a fuzz target for fuzzing, if the binary is instrumented for
fuzzing and coverage measurement.
For measuring Java coverage ("JAVA_LIBFUZZER"), no extra fuzz
target for coverage measurment is needed.
3. Metadata
The metadata file cifuzz.yaml contains all information necessary to
start a fuzzing run using the artifact. For a complete schema of
cifuzz.yaml, see Appendix A.
All paths in cifuzz.yaml are given relative to the archive's root
directory and are written WITHOUT a leading slash (/).
A typical cifuzz.yaml might look like this:
run_environment:
docker: ubuntu:rolling
fuzzers:
- target: .code-intelligence/fuzz_targets/fuzz_target.cpp
engine: LIBFUZZER
build_dir: "/home/ci/.local/share/code-intelligence/\
projects/blub-060bf9a1/llvm_cov"
path: libfuzzer/fuzz_target
library_paths:
- cifuzz-libs
max_run_time: 1800
- target: .code-intelligence/fuzz_targets/fuzz_target.cpp
engine: LLVM_COV
build_dir: "/home/ci/.local/share/code-intelligence/\
projects/blub-060bf9a1/llvm_cov"
path: llvm_cov/fuzz_target
library_paths:
- cifuzz-libs
max_run_time: 86400
Aside: Per convention, a fuzz target with file name <fuzz_target> and
engine <engine> usually has the path /<engine>/<fuzz_target>, and a
fuzz target with name <fuzz_target>, engine <engine> and using
sanitizer <sanitizer> usually has the path
/<engine>/<sanitizer>/<fuzz_target>. However, this is not necessary,
and path has to be given in all cases.
3.1. General properties
There are currently only two general properties that can be set in
cifuzz.yaml, 'run_environment' and 'code_revision':
'run_environment' specifies which environment to run the fuzzing
in. It currently only supports the subkey 'docker', which gives a
string—the docker container to run fuzzing in. Currently, every
container can be used as a run environment, but in the future this
may be limited to a set of containers deemed secure for usage.
'code_revision' specifies the version of the code according to the
source control management in use. Currently, only the key 'git' is
supported, giving a object with two keys, 'commit' and 'branch',
that give the current git commit hash and the current branch name
if applicable, respectively.
3.2. Fuzzer properties
Each fuzzer can take the following keys:
'target' [required] specifies the name of the fuzz target.
'engine' [required] specifies the fuzzing engine to be used. One of:
"LIBFUZZER", "AFL", "AFL_LLVM_MODE", "HONGGFUZZ", "LLVM_COV",
"UNIT_TEST_RUNNER", "JAVA_LIBFUZZER", "WEB_APP".
'path' [required] specifies the path of the fuzzing binary.
'sanitizer' specifies the sanitizer in LIBFUZZER mode. One of:
"UNKNOWN_SANITIZER", "ADDRESS", "MEMORY", "THREAD"
'build_dir' specifies the directory the fuzzers where built in.
This is used in order to differentiate functions from the
system under test from other functions (e.g. language or
framework) in stacktrace. Omitting 'build_dir' will thus lead
to empty stack traces.
'fuzzer_run_args' specifies a list of arguments to be given to the
fuzzer.
'library_paths' specifies the lists of paths to libraries needed to
run the fuzzer.
'seeds' specifies the path to directory containing the initial
corpora in the archive.
'dictionary' specifies the path to a dict file in the archive.
'engine_options' specifies options for the fuzzing engine. It has two
subkeys, 'flags' and 'env':
- 'flags' is a list of strings to be given to the engine as command
line arguments.
- 'env' is a list of strings to be given to the engine as
environment variables.
'max_run_time' specifies the maximum run time in seconds.
'parallel_executions' specifies the number of fuzzers to be run at
the same time.
Additional properties apply to web app fuzzing (out-of-process
fuzzing) only:
'exception_policy' specifies a path to an exception policy JSON file.
See Appendix B for an example file and Appendix C for a JSON schema
for exception policy JSON files.
'application_url' specifies the URL that the application is reachable
under.
'web_services' lists the webservices.
Appendix A. Schema of cifuzz.yaml
The following is the JSON Schema [http://json-schema.org/] of
cifuzz.yaml. Note that even though we are using JSON Schema as a
means of describing the format, the file format is still YAML.
See Section 3. "Metadata" for an example file.
{
"title": "cifuzz.yaml",
"format": "YAML",
"description": "This is a schema that describes the content of cifuzz.yaml in the top level of a fuzzing artifact archive.",
"type": "object",
"required": ["run_environment", "fuzzers"],
"properties": {
"run_environment": {
"type": "object",
"required": ["docker"],
"properties": {
"docker": {
"description": "The docker container to run the fuzzers in.",
"examples": ["debian:stable", "ubuntu:rolling"],
"type": "string"
}
}
},
"code_revision": {
"type": "object",
"properties": {
"git": {
"type": "object",
"properties": {
"commit": { "type": "string" },
"branch": { "type": "string" }
}
}
}
},
"fuzzers": {
"type": "array",
"items": {
"type": "object",
"required": ["target", "engine", "path"],
"properties": {
"target": {
"description": "The name of the fuzz target. Should be the name of the source file.",
"examples": [".code-intelligence/fuzz_targets/AllImagesParser.java"],
"type": "string"
},
"engine": { "enum": ["LIBFUZZER", "AFL", "AFL_LLVM_MODE", "HONGGFUZZ", "LLVM_COV", "UNIT_TEST_RUNNER", "JAVA_LIBFUZZER", "WEB_APP"] },
"sanitizer": { "enum": ["UNKNOWN_SANITIZER", "ADDRESS", "MEMORY", "THREAD"] },
"build_dir": { "type": "string" },
"path": {
"description": "The path of the fuzzer binary in the fuzzing artifact archive.",
"examples": ["java_libfuzzer/AllImagesParser.jar"],
"type": "string"
},
"library_paths": { "type": "array", "items": { "type": "string" }},
"seeds": { "type": "string" },
"dictionary": { "type": "string" },
"engine_options": {
"type": "object",
"properties": {
"flags": { "type": "array", "items": { "type": "string" }},
"env": { "type": "array", "items": { "type": "string" }}
}
},
"max_run_time": { "type": "number" },
"parallel_executions": { "type": "number" },
"skip_initial_corpus": { "type": "boolean" },
"exception_policy": { "type": "string" },
"application_url": { "type": "string" },
"web_services": { "type": "array", "items": { "type": "string" }},
}
}
}
}
}
Appendix B. Example of an exception policy JSON file
This is an example for an exception policy JSON file. It will
instruct the fuzzer monitor to ignore all exceptions from
responses with an HTTP return code of 4xx or 302 and all
exceptions of the type java.lang.IllegalArgumentException if
the stacktrace contains the string
"org.springframework.web.util.UriComponentsBuilder".
{
"policy": [
{
"enabled": true,
"responsePolicy": {
"responseMatcher": "CODE_STARTS_WITH",
"code": 4
}
},
{
"enabled": true,
"responsePolicy": {
"code": 302
}
},
{
"enabled": true,
"exceptionPolicy": {
"Exception": "java.lang.IllegalArgumentException",
"exceptionStrategy": "STACKTRACE_CONTAINS",
"content": "org.springframework.web.util.UriComponentsBuilder"
}
}
]
}
Appendix C. Schema of an exception policy JSON file
The following is the JSON Schema [http://json-schema.org/] of an
exception policy JSON file given in the fuzz target metadata key
'exception_policy'. Any matching policy _whitelists_ the given
exception or response, meaning that it is ignored and not considered
to be a finding. See Appendix B for an example file.
{
"title": "Exception policy",
"format": "JSON",
"description": "This is a schema that describes the content of a exception policy file in a fuzzing artifact archive.",
"type": "object",
"required": ["policy"],
"properties": {
"policy": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"required": ["enabled", "exceptionPolicy"],
"properties": {
"enabled": { "type": "boolean" },
"exceptionPolicy": {
"description": "Policy for exceptions.",
"type": "object",
"properties": {
"Exception": {"type": "string"},
"content": {"type": "string"},
"exceptionMatcher": {
"enum": ["EXCEPTION_IS", "EXCEPTION_FROM_PACKAGE", "EXCEPTION_STARTS_WITH"]
},
"exceptionStrategy": {
"enum": ["MATCH_ALL", "MESSAGE_CONTAINS", "MESSAGE_EXCLUDES", "STACKTRACE_CONTAINS"]
}
}
}
}
},
{
"type": "object",
"required": ["enabled", "responsePolicy"],
"properties": {
"enabled": { "type": "boolean" },
"responsePolicy": {
"description": "Policy for API responses.",
"type": "object",
"properties": {
"code": {"type": "number"},
"content": {"type": "string"},
"responseMatcher": {
"enum": ["CODE_IS", "CODE_STARTS_WITH"]
},
"responseStrategy": {
"enum": ["MATCH_ALL", "BODY_CONTAINS", "BODY_EXCLUDES"]
}
}
}
}
},
{
"type": "object",
"required": ["enabled", "zapPolicy"],
"properties": {
"enabled": { "type": "boolean" },
"zapPolicy": {
"description": "Policy for API responses.",
"type": "object",
"properties": {
"alert": {"type": "number"},
"content": {"type": "string"},
"alertMatcher": {
"enum": ["ALERT_NAME_IS", "ALERT_NAME_CONTAINS"]
},
"alertStrategy": {
"enum": ["MATCH_ALL", "URL_IS", "URL_CONTAINS"]
}
}
}
}
}
]
}
}
}
}