Summary:
This seems like a bug in opam, probably started after we moved the opam
files to opam/ in D27326495 (673c944fc3). I reported the strange behaviour in
https://github.com/ocaml/opam/issues/4313#issuecomment-853179145.
Reviewed By: rgrig
Differential Revision: D28837624
fbshipit-source-id: f91a58dbc
Summary:
This diff renames and moves the opam package definition files to
`<package>.opam` in an `opam` directory at the root of the
repository. This enables opam pinning the different packages contained
in the repo. It is necessary for these files to be either at the root
of the repo or in a directory named `opam` at the root of the repo, or
else opam cannot find them.
Reviewed By: skcho
Differential Revision: D27326495
fbshipit-source-id: 4c95c6955
Summary:
Make sure that we only download the clang tarballs needed to build clang
if:
1. clang is not installed
2. the user intends to build clang
Reviewed By: martintrojer
Differential Revision: D24137085
fbshipit-source-id: bfea1bf02
Summary:
When the switch changes to a new version of ocaml, opam may need updating to
see the new compiler version.
Reviewed By: jberdine
Differential Revision: D23678520
fbshipit-source-id: de1205de6
Summary:
Script changes to make the clang build work without the FCP submodule.
A llvm+clang+libraries src folder is downloaded and put together in the clang/src/download folder. This is the used for the clang build.
Reviewed By: ngorogiannis
Differential Revision: D23315610
fbshipit-source-id: 0cbce23e2
Summary:
Before: 3 modes: (where "lenient" build has warnings not crash the
build, while "strict" build errors on warning):
- opt (default): flambda optimisations + lenient build
- dev (recommand for dev): lenient build, no flambda
- test: strict build (used in tests), no flambda
Now:
- dev (default): *strict* build, no flambda
- opt: lenient build, flambda
- dev-noerror: lenient build, no flambda (use when you want to test
infer but there are build warnings)
The goal is to give faster feedback to infer developers and reduce the
amount of times diffs are sent with build warnings. Also it's now faster
to alternate between changing infer and running unit tests since test
mode is just dev mode.
Reviewed By: ngorogiannis
Differential Revision: D23167416
fbshipit-source-id: d663b6054
Summary:
Newer is better. Also upgrade other dependencies to their latest
version in opam.
allow-large-files
Reviewed By: skcho
Differential Revision: D21548074
fbshipit-source-id: 4061c71ee
Summary:
Not sure what's going on but even though `set -e` is passed,
`install_opam_deps` was seen as succeeding even when the `opam install`
command failed. This prevented `opam_retry` from getting triggered (and
the error was also silently ignored).
Also get rid of a couple of useless subshells in `opam_retry`.
Reviewed By: jberdine
Differential Revision: D18615860
fbshipit-source-id: 360232623
Summary:
Currently only user-specified opam switches that name compilers are
supported. This diff generalizes this so that the switch name and
compiler can be specified separately.
Reviewed By: jvillard
Differential Revision: D18477910
fbshipit-source-id: f17c6363d
Summary:
Previously it was required to provide SDKROOT during configure on Mojave
hosts to `make` the project which in scripts was messing up local clang
and somewhat error-prone. Instead we could use xcrun to find required SDK
paths automatically.
Reviewed By: jvillard
Differential Revision: D16072354
fbshipit-source-id: 93cbf3980
Summary: Turns out we were using it wrong and that's why the lock wasn't working. See https://github.com/ocaml/opam/issues/3694
Reviewed By: ezgicicek
Differential Revision: D13529136
fbshipit-source-id: 66c781e5f
Summary:
Avoid sandboxing issues and mark the incompatibility with javalib 3.0.
Also remove workaround for opam bug that has been fixed and that was
causing portability issues on osx where `realpath` isn't always
available.
Reviewed By: ngorogiannis
Differential Revision: D13450029
fbshipit-source-id: ea33b06d8
Summary:
Switches from opam 1 to opam 2.
Opam2 has some cool new features that simplify some of the scripting.
Notable changes:
1. Use the new `opam lock` *plugin* from https://github.com/AltGr/opam-lock/ instead of https://github.com/rgrinberg/opam-lock. This has a simpler interface for our purposes.
2. Change the way `./build-infer.sh` can be called to use an already existing switch: simply pass `--user-opam-switch` to the script and it won't attempt to create/set the current switch. This can be used to build infer in a local switch for instance.
3. Take advantage of automatic pinning where possible, eg to install infer deps without using opam.locked.
Reviewed By: ngorogiannis
Differential Revision: D13167863
fbshipit-source-id: 1a667c270
Summary:
Change the documentation to refer to (upcoming) binary releases. Update the
scripts to treat .release differently: now we want to build clang and the
plugins even in release mode, as that's just the preparation for the release
tarball containing only binaries.
Reviewed By: mbouaziz
Differential Revision: D8235388
fbshipit-source-id: bfb4ae8
Summary:
Change the license of the source code from BSD + PATENTS to MIT.
Change `checkCopyright` to reflect the new license and learn some new file
types.
Generated with:
```
git grep BSD | xargs -n 1 ./scripts/checkCopyright -i
```
Reviewed By: jeremydubreil, mbouaziz, jberdine
Differential Revision: D8071249
fbshipit-source-id: 97ca23a
Summary:
Users missing the development package for libsqlite3 would see only a failure to install "sqlite3". Adding conf-sqlite3 to the dependencies does not improve things that much but there is a bit more information in the console:
```
[ERROR] The compilation of conf-sqlite3 failed at "pkg-config sqlite3".
[ERROR] The compilation of sqlite3 failed at "jbuilder build -p sqlite3 -j 4".
#=== ERROR while installing conf-sqlite3.1 ====================================#
# opam-version 1.2.2
# os linux
# command pkg-config sqlite3
# path /home/jul/.opam/infer-4.06.1+flambda/build/conf-sqlite3.1
# compiler 4.06.1+flambda
# exit-code 1
# env-file /home/jul/.opam/infer-4.06.1+flambda/build/conf-sqlite3.1/conf-sqlite3-7504-7afd23.env
# stdout-file /home/jul/.opam/infer-4.06.1+flambda/build/conf-sqlite3.1/conf-sqlite3-7504-7afd23.out
# stderr-file /home/jul/.opam/infer-4.06.1+flambda/build/conf-sqlite3.1/conf-sqlite3-7504-7afd23.err
#=== ERROR while installing sqlite3.4.3.2 =====================================#
# opam-version 1.2.2
# os linux
# command jbuilder build -p sqlite3 -j 4
# path /home/jul/.opam/infer-4.06.1+flambda/build/sqlite3.4.3.2
# compiler 4.06.1+flambda
# exit-code 1
# env-file /home/jul/.opam/infer-4.06.1+flambda/build/sqlite3.4.3.2/sqlite3-7504-d2c37b.env
# stdout-file /home/jul/.opam/infer-4.06.1+flambda/build/sqlite3.4.3.2/sqlite3-7504-d2c37b.out
# stderr-file /home/jul/.opam/infer-4.06.1+flambda/build/sqlite3.4.3.2/sqlite3-7504-d2c37b.err
### stderr ###
# -> stdout:
# [...]
# | ast_impl_magic_number: Caml1999M022
# | ast_intf_magic_number: Caml1999N022
# | cmxs_magic_number: Caml1999D022
# | cmt_magic_number: Caml1999T022
# -> stderr:
# Fatal error: exception End_of_file
# Raised at file "src/import0.ml" (inlined), line 351, characters 22-32
# Called from file "src/configurator.ml", line 511, characters 13-22
# Called from file "src/config/discover.ml", line 42, characters 2-1023
Exception:
Reqs_error
(Process_error
("opam install -y ANSITerminal.0.8 atd.1.12.0 atdgen.1.12.0 base.v0.11.0 base64.2.2.0 bin_prot.v0.11.0 biniou.1.2.0 camlp4.4.06+1 camlzip.1.07 cmdliner.1.0.2 conf-aclocal.1.0.0 conf-autoconf.0.1 conf-m4.1 conf-pkg-config.1.0 conf-sqlite3.1 conf-which.1 conf-zlib.1 configurator.v0.11.0 core.v0.11.0 "... (* string length 1469; truncated *),
Unix.WEXITED 4)).
```
Also fix some issues with `build-infer.sh`:
- fix a problem where `SCRIPT_PATH` should be `SCRIPT_DIR`
- add `set -u` and `set -o pipefail` to make sure we don't miss errors in the future
- add quotes everywhere
- make number of `JOBS` user-configurable instead of hardcoding `$NCPU`
Reviewed By: mbouaziz
Differential Revision: D8201849
fbshipit-source-id: 19b7c77
Summary:
No need for our own patched version now that it's available in opam.
You should `opam pin remove --no-action javalib && ./build-infer.sh` to get rid of the previous version.
Reviewed By: mbouaziz
Differential Revision: D6063730
fbshipit-source-id: 8efd598
Summary:
With flambda (`-O3`), compilation time is ~5x slower, but the backend is ~25% faster!
To mitigate the atrocious compilation times, introduce a new `opt` build mode in the jbuilder files.
- build in "opt" mode by default from the toplevel (so that install scripts and external users get the fastest infer by default), in "default" mode by default from infer/src (since the latter is only called directly by infer devs, for faster builds)
- `make byte` is as fast as before in any mode
- `make test` will build "opt" by default, which is very slow. Solution for testing (or building the models) locally: `make BUILD_MODE=default test`.
- You can even change the default locally with `export BUILD_MODE=default`.
The benchmarks are to be taken with a sizable pinch of salt because I ran them only once and other stuff could be running in the background. That said, the perf win is consistent across all projects, with 15-20% win in wallclock time and around 25% win in total CPU time, ~9% win in sys time, and ~25% fewer minor allocations, and ~5-10% fewer overall allocations. This is only for the backend; the capture is by and large unaffected (either the same or a tad faster within noise range).
Here are the results running on OpenSSL 1.0.2d on osx (12 cores, 32G RAM)
=== base
infer binary: 26193088 bytes
compile time: 40s
capture:
```lang=text
real 1m7.513s
user 3m11.437s
sys 0m55.236s
```
analysis:
```lang=text
real 5m41.580s
user 61m37.855s
sys 1m12.870s
```
Memory profile:
```lang=json
{
...
"minor_gb": 0.1534719169139862,
"promoted_gb": 0.0038930922746658325,
"major_gb": 0.4546157643198967,
"allocated_gb": 0.6041945889592171,
"minor_collections": 78,
"major_collections": 23,
"compactions": 7,
"top_heap_gb": 0.07388687133789062,
"stack_kb": 0.3984375,
"minor_heap_kb": 8192.0,
...
}
```
=== flambda with stock options (no `-Oclassic`, just the same flags as base)
Exactly the same as base.
=== flambda `-O3`
infer binary: 56870376 bytes (2.17x bigger)
compile time: 191s (4.78x slower)
capture is the same as base:
```lang=text
real 1m9.203s
user 3m12.242s
sys 0m58.905s
```
analysis is ~20% wallclock time faster, ~25% CPU time faster:
```lang=text
real 4m32.656s
user 46m43.987s
sys 1m2.424s
```
memory usage is a bit lower too:
```lang=json
{
...
"minor_gb": 0.11583046615123749, // 75% of previous
"promoted_gb": 0.00363825261592865, // 93% of previous
"major_gb": 0.45415670424699783, // about same
"allocated_gb": 0.5663489177823067, // 94% of previous
"minor_collections": 73,
"major_collections": 22,
"compactions": 7,
"top_heap_gb": 0.07165145874023438,
"stack_kb": 0.3359375,
"minor_heap_kb": 8192.0,
...
}
```
=== flambda `-O2`
Not nearly as exciting as `-O3`, but the compilation cost is still quite high:
infer: 37826856 bytes
compilation of infer: 100s
Capture and analysis timings are mostly the same as base.
Reviewed By: jberdine
Differential Revision: D4867979
fbshipit-source-id: 99230b7
Summary:
Installing with `opam pin` risks upgrading a lot of package in the current
switch, which is very slow. It's also wasted work since we'll install back
their older versions afterwards with `opam lock`. Moreover, that second step
can now fail if javalib needs to be recompiled (which it seems it does),
because we delete the javalib sources after having pinned them.
Solution: 1. do not install javalib at `opam pin` time, and 2. do not remove its source tree after pinning.
Reviewed By: martinoluca
Differential Revision: D5764153
fbshipit-source-id: 9c9b1c7
Summary:
For instance, `./build-infer.sh --opam-switch "4.04.2+spacetime"` now builds an
actual spacetime switch instead of a clone of the 4.04.2 switch called
"4.04.2+spacetime".
Reviewed By: jberdine
Differential Revision: D5631258
fbshipit-source-id: e6295c5
Summary:
Useful for hacking on infer, and when eg the default opam switch changes and
one has to remember which development packages to install all over again
(merlin, etc.).
Reviewed By: mbouaziz
Differential Revision: D5328701
fbshipit-source-id: 7b7694e
Summary:
Let's try this one more time.
- change build-infer.sh so that it also tries to init opam even if a switch is
specified. This shouldn't matter according to my testing and that means that
it's easier to have `./build-infer.sh --opam-switch 4.04.0` work out of the box
without needing to initialise the `4.04.0` opam switch ourselves beforehand.
- use that in .travis.yml
Reviewed By: jeremydubreil
Differential Revision: D5301320
fbshipit-source-id: 12f91ae
Summary: This uses opam.lock which is less likely to break due to external opam changes.
Reviewed By: cristianoc
Differential Revision: D5208622
fbshipit-source-id: af6f03c
Summary: This makes the build less likely to break because of package upgrades in opam.
Reviewed By: mbouaziz
Differential Revision: D4937908
fbshipit-source-id: 5ce9ec7
Summary:
Opam can give cryptic errors when all you need to do is `opam update`, so try
that for the user.
Reviewed By: mbouaziz
Differential Revision: D4867878
fbshipit-source-id: f740b13
Summary:
This enables the following workflow:
```
$ ./build-infer.sh
[...]
$ opam upgrade # takes infer dependencies into account
```
To do so, we install a temporary `infer-deps-XXXX` package as before (the temp
name is because opam thinks there's nothing to do otherwise), then we install
an `infer-deps` package permanently.
Reviewed By: mbouaziz
Differential Revision: D4867195
fbshipit-source-id: da806f9
Jules Villard
Radu Grigore
Josh Berdine
HARUYAMA Seigo
Martin Trojer
Anton Kochkov
Sungkeun Cho
Artem Pianykh
Phoebe Nichols
Sam Blackshear