3ab6c43fb6
[tests] awesomize make output
...
Summary:
Hide the output of all the toplevel `make` actions and print messages instead, as if we had a better build system than Makefiles. When a command fails, print its stdout and stderr and prefix it with bold red letters. Also display timing information.
```
$ make -j 4 test
[17:39:18][27290] Facebook setup...
[17:39:18][27363] Building checkCopyright...
[17:39:18][27443] Validating facebook/...
[ 0s][27363] SUCCESS Building checkCopyright
[ 0s][27290] SUCCESS Facebook setup
[17:39:18][27863] Building clang plugin...
[ 0s][27443] SUCCESS Validating facebook/
[ 0s][27863] SUCCESS Building clang plugin
[17:39:18][27898] Building clang plugin OCaml interface...
[ 0s][27898] SUCCESS Building clang plugin OCaml interface
[17:39:18][27974] Testing Infer builds without warnings...
[17:39:18][28033] Building native Infer...
[ 12s][27974] SUCCESS Testing Infer builds without warnings
[17:39:30][28998] Testing Infer toplevel builds...
[ 6s][28998] SUCCESS Testing Infer toplevel builds
[17:39:36][31417] Running OCaml unit tests...
[17:39:36][31456] Testing infer OCaml REPL...
[ 6s][31417] SUCCESS Running OCaml unit tests
[ 9s][31456] SUCCESS Testing infer OCaml REPL
[ 92s][28033] SUCCESS Building native Infer
[17:40:50][ 2170] Building Java annotations...
[ 0s][ 2170] SUCCESS Building Java annotations
[17:40:50][ 2186] Building Infer models...
[ 11s][ 2186] SUCCESS Building Infer models
[17:41:01][ 2803] Running build assembly test...
[17:41:01][ 2861] Running build ck analytics test...
[17:41:01][ 2941] Running build ck imports test...
[17:41:01][ 3035] Running build clang compilation db escaped test...
[ 1s][ 3035] SUCCESS Running build clang compilation db escaped test
[17:41:02][ 3139] Running build clang compilation db relpath test...
[ 3s][ 2861] SUCCESS Running build ck analytics test
[17:41:04][ 3296] Running build clang multiple files test...
[*ERROR**][2803] *** ERROR Running build assembly test
[*ERROR**][2803] *** command: ( make INTERACTIVE=1 -C /home/jul/infer/infer/tests/build_systems/assembly test 2> >(grep -v warning: (ignoring old|overriding) (commands|recipe) for target) ; exit 0)
[*ERROR**][2803] *** CWD: /home/jul/infer
[*ERROR**][2803] *** stdout:
[*ERROR**][2803] make[1]: Entering directory '/home/jul/infer/infer/tests/build_systems/assembly'
[*ERROR**][2803] *** inner Testing infer/clang in build_systems/assembly
[*ERROR**][2803] *** inner command: /home/jul/infer/infer/bin/infer --results-dir infer-out --dump-duplicate-symbols --report-custom-error --developer-mode --project-root ../codetoanalyze --no-failures-allowed -a infer -- clang -c ../codetoanalyze/example.S ../codetoanalyze/hello.c
[*ERROR**][2803] *** inner CWD: /home/jul/infer/infer/tests/build_systems/assembly
[*ERROR**][2803] Capturing in make/cc mode...
[*ERROR**][2803] Starting analysis...
[*ERROR**][2803]
[*ERROR**][2803] legend:
[*ERROR**][2803] "F" analyzing a file
[*ERROR**][2803] "." analyzing a procedure
[*ERROR**][2803]
[*ERROR**][2803] Found 1 (out of 1) source files to be analyzed in /home/jul/infer/infer/tests/build_systems/assembly/infer-out
[*ERROR**][2803]
[*ERROR**][2803]
[*ERROR**][2803] No issues found
[*ERROR**][2803]
[*ERROR**][2803] ../../infer.make:30: recipe for target 'test' failed
[*ERROR**][2803] make[1]: Leaving directory '/home/jul/infer/infer/tests/build_systems/assembly'
[*ERROR**][2803] F.--- build_systems/assembly/issues.exp 2017-03-22 16:25:44.583854270 +0000
[*ERROR**][2803] +++ build_systems/assembly/issues.exp.test 2017-03-28 17:41:04.743678254 +0100
[*ERROR**][2803] @@ -1 +0,0 @@
[*ERROR**][2803] -hello.c, test, 2, NULL_DEREFERENCE, [start of procedure test()]
[*ERROR**][2803]
[*ERROR**][2803] Test output (build_systems/assembly/issues.exp.test) differs from expected test output build_systems/assembly/issues.exp
[*ERROR**][2803] Run the following command to replace the expected test output with the new output:
[*ERROR**][2803]
[*ERROR**][2803] make -C build_systems/assembly replace
[*ERROR**][2803]
[*ERROR**][2803] make[1]: *** [test] Error 1
[*ERROR**][2803] *** stderr:
Makefile:230: recipe for target 'build_assembly_test' failed
make: *** [build_assembly_test] Error 1
make: *** Waiting for unfinished jobs....
[...]
```
Reviewed By: jberdine
Differential Revision: D4781857
fbshipit-source-id: cbce26d
8 years ago
4ec3690413
[tests] add a `replace` target to all tests
...
Summary:
Most of our tests work by comparing the output of some command to a baseline
expected output. It's often needed to update that baseline.
Previously, that was crudely done by attempting to move every foo.exp.test file
to foo.exp. This does not work terribly well, in particular because
foo.exp.test might be stale.
Instead, add a `replace` target to every test that knows how to update the
baseline. This allows custom behaviours too, eg in the differential tests.
Most of the tests include base.make or differential.make, so add a replace target
there. A few tests are completely custom, add a replace target to them too.
Reviewed By: jeremydubreil
Differential Revision: D4754279
fbshipit-source-id: ec34273
8 years ago
f8a65e698c
Add differential and filtering functionalities to Infer, to compare two analyses
...
Summary:
Given two analysis results, it's now possible to compare them with the following command:
infer --diff --report-current reportA.json --report-previous reportB.json --file-renamings file_renamings.json
this command will then generate 3 files in `infer-out/differential/{introduced, fixed, preexisting}.json`, whose meaning is the following:
- `introduced.json` has all issues in `current` that are not in `previous`
- `fixed.json` has all issues in `previous` that are not in `current`
- `preexisting.json` has all issues that are in both `current` and `previous`
The json files generated can then be used to categorise results coming from incremental analyses of a codebase.
Reviewed By: jvillard
Differential Revision: D4482517
fbshipit-source-id: 1f7df3e
8 years ago
Jules Villard
Martino Luca