documentation for new libraries

Summary:
The documentation had gone out of sync with the new library names. Add
or copy some short documentation for the main libraries, i.e. all of
them except individual analyses (and scripts, third party, ..).

The idea is that each library has some toplevel documentation
`infer/src/<library_dir>/<LibraryName>.mld` that is linked to from the
main entry point of the document infer/infer.mld. We can link to some
important modules for each library from within their toplevel
documentation, then the actual documentation should live inside the
.mli's of the modules of the library as appropriate.

Hopefully this leads to better documentation over time. At least now we
can write some docs and they'll end up somewhere nice. Lots can be
improved still at this point.

Reviewed By: ngorogiannis

Differential Revision: D21551955

fbshipit-source-id: 69a0cfa44

infer/IR.mld

@@ -1,11 +0,0 @@
-{1 Library InferIR: Intermediate Representation}
-The Intermediate Representation, called [SIL], is a format used by the back-end for analysis. It is produced by the front-ends.
-
-The entry point of this library is the module:
-{!module-InferIR}.
-
-The main entry point is the intermediate language in {!module-InferIR.Sil}.
-
-The control flow graph module is {!module-InferIR.Cfg}.
-
-The type environment module is {!module-InferIR.Tenv}.

infer/atd.mld

@@ -1,5 +0,0 @@
-{1 Library InferGenerated}
-Source code that is generated from ATD definitions.
-
-The entry point of this library is the module:
-{!module-InferGenerated}.

infer/base.mld

@@ -1,6 +0,0 @@
-{1 Library InferBase}
-Utility modules that are somewhat specific to infer, usually because
-they depend on Config.ml.
-
-The entry point of this library is the module:
-{!module-InferBase}.

infer/dune

@@ -5,7 +5,4 @@
 
 (dirs src bin)
 
-(documentation
- (package infer)
- (mld_files :standard)
-)
+(documentation (package infer) (mld_files :standard))

infer/index.mld

@@ -1,7 +0,0 @@
-{1 Infer Libraries}
-
-- {{:infer.html}infer}: most of the modules: capture, analysis, integrations, ...
-- {{:IR.html}IR}: the intermediate language used by the analysis
-- {{:base.html}base}: configuration and utilities
-- {{:atd.html}atd}: auto-generated datatypes
-- {{:istd.html}}: additions to the Core standard library

infer/infer.mld

@@ -1,63 +1,45 @@
-{1 Library infer}
-These are the modules used by infer that are not packaged into sub-libraries.
+{0 Infer}
 
-The entry point of this library is the module:
-{!module-InferModules}.
+{1 Backend}
 
+Want to write your own analysis? You probably want the
+{{:Absint.html}Absint} library. {{:Backend.html}Backend} is in charge
+of orchestrating the various analyses on a collection of procedures.
 
-The modules are split into several directories in the source tree of infer.
+Some of the simplest analyses can be found in the {{:Checkers.html} Checkers} library.
 
-{2 absint/}
-Code + utilities for the Infer.AI abstract interpretation framework. Start at {!module-InferModules.AbstractInterpreter}.
+{1 Frontends}
 
-{2 backend/}
-The backend is responsible for the analysis of a project starting from an intermediate representation stored in the results directory, typically [infer-out].
+The frontends translate the source code to analyze into Infer's
+intermediate representation (see {{:IR.html} IR}) to be analyzed by
+the backend.
 
-The entry point for the analysis is the module {!module-InferModules.InferAnalyze}.
+There are two frontends:
+- {{:ClangFrontend.html} Clang}
+- {{:JavaFrontend.html} Java}
 
-The module {!module-InferModules.InferPrint} is used to export the analysis results.
+See also {{:ASTLanguage.html} AST Language}, and
+{{:TestDeterminators.html} TestDeterminators}, which operate right
+after the frontends.
 
-{2 checkers/}
-Transfer functions, abstract domains, and checker configuration.
+{1 Utilities}
 
-{2 clang/}
-This is the frontend for the clang compiler. The main entry point is {!module-InferModules.ClangWrapper}.
+{{:IStdlib.html} IStdlib} groups non-Infer-specific modules, and
+{{:IBase.html} IBase} contains more Infer-specific modules that other
+Infer libraries need, usually depending on the {!module-IBase.Config}
+module. Some datatypes and their accompanying utilities are defined in
+the {{:ATDGenerated.html} ATDGenerated} library.
 
-{3 General structure}
-+ Given a compilation command from the build system, sanitize it, attach [ASTExporter] clang plugin to the relevant commands and run it.
-+ Parse the AST (Abstract Syntax Tree) from the Biniou format into an OCaml data structure.
-+ If enabled, invoke translation to [SIL] via {!module-InferModules.CFrontend}.
-+ If enabled, invoke linters via {!module-InferModules.CFrontend_checkers_main}. More on linters here: {:http://fbinfer.com/docs/linters.html}.
+{1 Integrations and Infer executables}
 
-{3 Format of the AST}
-The OCaml data type is defined in [atd] format. The [clang_ast_t.atd] file is generated from the comments in the {{:https://github.com/facebook/facebook-clang-plugins/blob/master/libtooling/ASTExporter.h} [ASTExporter.h]} file.
+{{:Integration.html} Integration} ties the frontends and backend
+together, and deals with build system integrations.
 
-For more information, refer to the relevant documentation in [facebook-clang-plugins]:
-- {{:https://github.com/facebook/facebook-clang-plugins/blob/master/libtooling/ATD_GUIDELINES.md} [libtooling/ATD_GUIDELINES.md]}
-- {{:https://github.com/facebook/facebook-clang-plugins/blob/master/clang-ocaml/README.md} [clang-ocaml/README.md]}
+You'll find the source code for the [infer] command-line interface and
+others in the
+{{:https://github.com/facebook/infer/blob/master/infer/src/}infer/src/}
+directory.
 
-{3 Hacking on the [.atd] file}
-+ Create a simple example ([example.cpp]) source file with construct that needs to be exported. The smaller the better.
-+ Export the extra information by changing the code in {{:https://github.com/facebook/facebook-clang-plugins/blob/master/libtooling/ASTExporter.h} [libtooling/ASTExporter.h]}. For more information, refer to the {{:https://github.com/facebook/facebook-clang-plugins/blob/master/libtooling/ATD_GUIDELINES.md} [ATD_GUIDELINES]}.
-+ Compile Infer with the new version of [facebook-clang-plugins]. Running [make] from top level of Infer repository will do that. Sometimes there may be compilation errors due to [.atd] file changes - they need to be fixed.
-+ Use newly exported information information in the frontend.
+{1 Index}
 
-Tips & Tricks:
-- To view the AST in a human readable version, Infer can generate [.bdump] file: [infer -g -- clang -c example.cpp && sh example.cpp.ast.sh]. Then open [example.cpp.ast.bdump].
-- To inspect AST dump visually: [clang -c example.cpp -Xclang -ast-dump]. It doesn't include all the information that Infer sees, but it's pretty concise
-- If running [bdump] is failing (it happens on huge sources sometimes), there is a way to view it in "Yojson" format. To do that, replace all occurrences of [BiniouASTExporter] with [YojsonASTExporter] in the [.ast.sh] script.
-
-{2 eradicate/ }
-Eradicate [@Nullable] Checker
-
-Eradicate is a type checker for [@Nullable] annotations for Java. The goal is to eradicate null pointer exceptions. See the online docs at {:http://fbinfer.com/docs/eradicate}.
-
-The main entry point is module {!module-InferModules.Eradicate}.
-
-{2 java/}
-Java Frontend; this is the frontend for Java compilers.
-
-The main entry point is {!module-InferModules.JMain}.
-
-{2 infer executable}
-You'll find the infer binary in infer.ml. It is not part of this library.
+{{:index.html}List of all libraries.}

infer/istd.mld

@@ -1,6 +0,0 @@
-{1 Library InferStdLib}
-Utility modules not specific to infer. In particular, nothing here
-should depend on Config.ml.
-
-The entry point of this library is the module:
-{!module-InferStdlib}.

infer/src/IR/IR.mld

@@ -0,0 +1,11 @@
+{0 Library IR: Intermediate Representation}
+
+The Intermediate Representation, called [SIL], is a format used by the back-end for analysis. It is produced by the front-ends.
+
+The entry point of this library is the module {!module-IR}.
+
+The main entry point is the intermediate language in {!module-IR.Sil}.
+
+The control flow graph module is {!module-IR.Cfg}.
+
+The type environment module is {!module-IR.Tenv}.

infer/src/IR/dune

@@ -12,3 +12,5 @@
  (libraries core zarith IStdlib ATDGenerated IBase)
  (preprocess
   (pps ppx_compare)))
+
+(documentation (package infer) (mld_files IR))

infer/src/absint/Absint.mld

@@ -0,0 +1,7 @@
+{0 Abstract Interpretation Framework}
+
+{!module-Absint} is Infer's abstract interpretation framework, the recommended way to write analyses in Infer.
+
+Important modules:
+- {!module-Absint.AbstractInterpreter}
+- {!module-Absint.AbstractDomain}

infer/src/absint/AbstractDomain.mli

@@ -7,6 +7,8 @@
 
 open! IStd
 
+(** {1 Abstract domains and domain combinators} *)
+
 module Types : sig
   type 'astate bottom_lifted = Bottom | NonBottom of 'astate
 
@@ -17,8 +19,6 @@ end
 
 open! Types
 
-(** Abstract domains and domain combinators *)
-
 module type NoJoin = sig
   include PrettyPrintable.PrintableType
 

infer/src/absint/AbstractInterpreter.mli

@@ -31,7 +31,7 @@ module type S = sig
     -> initial:TransferFunctions.Domain.t
     -> Procdesc.t
     -> TransferFunctions.Domain.t option
-  (** compute and return the postcondition for the given {Procdesc.t} starting from [initial].
+  (** compute and return the postcondition for the given {!Procdesc.t} starting from [initial].
       [pp_instr] is used for the debug HTML and passed as a hook to handle both SIL and HIL CFGs. *)
 
   val exec_cfg :

infer/src/absint/dune

@@ -12,3 +12,5 @@
  (libraries core IStdlib ATDGenerated IBase IR)
  (preprocess (pps ppx_compare))
 )
+
+(documentation (package infer) (mld_files Absint))

infer/src/al/ASTLanguage.mld

@@ -0,0 +1,5 @@
+{0 AST Language (AL)}
+
+Linter framework based on the {{:ClangFrontend.html} Clang frontend}.
+
+All modules: {!module-ASTLanguage}

infer/src/al/dune.in

@@ -23,7 +23,9 @@ let al =
      -open IBase -open IR -open Absint -open ClangFrontend))
  (libraries core IStdlib ATDGenerated IBase IR Absint ClangFrontend)
  (preprocess (pps ppx_compare))
-)|}
+)
+
+(documentation (package infer) (mld_files ASTLanguage))|}
     (if clang then ":standard" else "")
 
 

infer/src/atd/ATDGenerated.mld

@@ -0,0 +1,5 @@
+{0 Library ATDGenerated}
+
+Source code that is generated from ATD definitions.
+
+The entry point of this library is the module {!module-ATDGenerated}.

infer/src/atd/dune

@@ -12,6 +12,8 @@
  (preprocess
   (pps ppx_compare)))
 
+(documentation (package infer) (mld_files ATDGenerated))
+
 ; Rules below are boilerplatey, but this is benign and very easy to
 ; understand boilerplate.
 

infer/src/backend/Backend.mld

@@ -0,0 +1,9 @@
+{0 Orchestration of the Backend Analyses}
+
+Important modules:
+- {!module-Backend.InferAnalyze}
+- {!module-Backend.Ondemand}
+- {!module-Backend.RegisterCheckers}
+- {!module-Backend.Summary}
+
+All modules: {!module-Backend}

infer/src/backend/InferAnalyze.mli

@@ -11,7 +11,6 @@ open! IStd
 (** Main module for the analysis after the capture phase *)
 
 val main : changed_files:SourceFile.Set.t option -> unit
-(** Given a name of the Makefile to use for multicore analysis, analyze the captured code *)
 
 val invalidate_changed_procedures : SourceFile.Set.t option -> unit
 (** Invalidate specs files for procedures that have changed. Used for incremental analysis. *)

infer/src/backend/dune

@@ -16,3 +16,5 @@
             Checkers Costlib Quandary TOPLlib Concurrency Labs)
  (preprocess (pps ppx_compare ppx_fields_conv))
 )
+
+(documentation (package infer) (mld_files Backend))

infer/src/base/IBase.mld

@@ -0,0 +1,6 @@
+{0 Library IBase}
+
+Utility modules that are somewhat specific to infer, usually because
+they depend on {!Config}.
+
+The entry point of this library is the module {!module-IBase}.

infer/src/base/dune

@@ -11,3 +11,5 @@
    IStdlib OpenSource)
  (preprocess
   (pps ppx_compare ppx_enumerate)))
+
+(documentation (package infer) (mld_files IBase))

infer/src/checkers/Checkers.mld

@@ -0,0 +1,5 @@
+{0 Checkers Library}
+
+A collection of various analyses.
+
+All modules: {!module-Checkers}

infer/src/checkers/dune

@@ -12,3 +12,5 @@
  (libraries base64 core ocamlgraph IStdlib ATDGenerated IBase IR Absint Biabduction BO Pulselib)
  (preprocess (pps ppx_compare))
 )
+
+(documentation (package infer) (mld_files Checkers))

infer/src/clang/Capture.mli

@@ -10,6 +10,15 @@ open! IStd
 val run_clang : ClangCommand.t -> (In_channel.t -> 'a) -> 'a
 
 val capture : ClangCommand.t -> unit
+(** If the command is detected to correspond to a source file, translate it.
+
+    + Given a compilation command, attach our [ASTExporter] clang plugin to the command and run it.
+    + Our clang plugin emits the AST (Abstract Syntax Tree) as Biniou data that we deserialize. The
+      AST format is described in {!module-ATDGenerated.Clang_ast_t} (and its Biniou API is in
+      {!module-ATDGenerated.Clang_ast_b}).
+    + If enabled, invoke translation to {!module-IR.Sil} via {!module-CFrontend}.
+    + If enabled, invoke linters callbacks from [ASTLanguage] (see below). More on linters here:
+      {:http://fbinfer.com/docs/linters.html}. *)
 
 val al_callback_ref : (CFrontend_config.translation_unit_context -> Clang_ast_t.decl -> unit) ref
 (** callback set by AL to avoid circular dependencies between clang/ and al/ without having to

infer/src/clang/ClangFrontend.mld

@@ -0,0 +1,25 @@
+{0 Clang Frontend}
+
+Translates code that {{:https://clang.llvm.org/}[Clang]} understands into Infer's intermediate representation language {!module-IR.Sil}.
+
+All modules: {!module-ClangFrontend}
+
+{1 Capturing source files}
+
+Start at {!module-ClangFrontend.Capture} for how Infer uses Clang to parse source files, using our Clang plugin. For more information on our clang plugin itself, refer to the relevant documentation in [facebook-clang-plugins]:
+
+- {{:https://github.com/facebook/facebook-clang-plugins/blob/master/libtooling/ATD_GUIDELINES.md} [facebook-clang-plugins/libtooling/ATD_GUIDELINES.md]}
+- {{:https://github.com/facebook/facebook-clang-plugins/blob/master/clang-ocaml/README.md} [facebook-clang-plugins/clang-ocaml/README.md]} 
+
+{1 Quick guide to changing the AST format}
+
++ Create a simple example ([example.cpp]) source file with construct that needs to be exported. The smaller the better.
++ Export the extra information by changing the code in {{:https://github.com/facebook/facebook-clang-plugins/blob/master/libtooling/ASTExporter.h} [libtooling/ASTExporter.h]}. For more information, refer to the {{:https://github.com/facebook/facebook-clang-plugins/blob/master/libtooling/ATD_GUIDELINES.md} [ATD_GUIDELINES]}.
++ Compile Infer with the new version of [facebook-clang-plugins]. Running [make] from top level of Infer repository will do that. Sometimes there may be compilation errors due to [.atd] file changes --they need to be fixed.
++ Use newly-exported information in the frontend as you please.
+
+{1 Tips & Tricks}
+
+- To view the AST in a human readable version, Infer can generate [.bdump] files: [infer -g -- clang -c example.cpp && sh example.cpp.ast.sh]. Then open [example.cpp.ast.bdump].
+- To inspect the AST visually: [clang -c example.cpp -Xclang -ast-dump]. It doesn't include all the information that Infer sees, but it's often a good place to start.
+- If running [bdump] is failing (it happens on huge sources sometimes), there is a way to view it in "Yojson" format. To do that, replace all occurrences of [BiniouASTExporter] with [YojsonASTExporter] in the [.ast.sh] debug script.

infer/src/clang/ClangWrapper.ml

@@ -6,9 +6,6 @@
  *)
 
 open! IStd
-(** Given a clang command, normalize it via `clang -###` if needed to get a clear view of what work
-    is being done and which source files are being compiled, if any, then replace compilation
-    commands by our own clang with our plugin attached for each source file. *)
 
 module L = Logging
 

infer/src/clang/ClangWrapper.mli

@@ -8,3 +8,7 @@
 open! IStd
 
 val exe : prog:string -> args:string list -> unit
+(** Given a clang command, normalize it via [clang -###] if needed to get a clear view of what work
+    is being done and which source files are being compiled, if any. Pass the resulting files to
+    compile to {!module-Capture} to be captured, i.e., parsed and translated into Infer's IR
+    {!module-IR.Sil}. *)

infer/src/clang/dune.in

@@ -23,5 +23,7 @@ let clang =
     (if clang then ":standard \\ ClangFrontendStubs" else "ClangFrontendStubs")
 
 
+let documentation = "(documentation (package infer) (mld_files ClangFrontend))"
+
 ;;
-Jbuild_plugin.V1.send clang
+Jbuild_plugin.V1.send (String.concat "\n" [clang; documentation])

infer/src/integration/Integration.mld

@@ -0,0 +1,3 @@
+{0 Integration}
+
+All modules: {!module-Integration}

infer/src/integration/dune.in

@@ -22,7 +22,8 @@ let library =
      TestDeterminators ClangFrontend ASTLanguage JavaFrontend)
  (preprocess (pps ppx_compare))
 )
-|}
+
+(documentation (package infer) (mld_files Integration))|}
     (if clang then "" else "-open ClangFrontendStubs")
     (if java then "" else "-open JavaFrontendStubs")
 

infer/src/istd/IStdlib.mld

@@ -0,0 +1,5 @@
+{0 Library IStdlib}
+
+Additions to the standard library that are not specific to infer.
+
+The entry point of this library is the module {!module-IStdlib}.

infer/src/istd/dune

@@ -11,3 +11,5 @@
  (libraries ANSITerminal core str yojson)
  (preprocess
   (pps ppx_compare)))
+
+(documentation (package infer) (mld_files IStdlib))

infer/src/java/JavaFrontend.mld

@@ -0,0 +1,5 @@
+{0 Java Frontend}
+
+This is the frontend for Java compilers.
+
+All modules: {!module-JavaFrontend}

infer/src/java/dune.in

@@ -27,5 +27,7 @@ let java =
     (if java then "javalib sawja" else "")
 
 
+let documentation = "(documentation (package infer) (mld_files JavaFrontend))"
+
 ;;
-Jbuild_plugin.V1.send (String.concat " " [lexer; java])
+Jbuild_plugin.V1.send (String.concat "\n" [lexer; java; documentation])

infer/src/nullsafe/Nullsafe.mld

@@ -0,0 +1,8 @@
+{0 Nullsafe [@Nullable] Checker}
+
+Nullsafe (also called Eradicate) is a type checker for [@Nullable]
+annotations in Java. The goal is to eradicate null pointer
+exceptions. See the online docs at
+{:http://fbinfer.com/docs/eradicate}.
+
+All modules: {!module-Nullsafe}.

infer/src/nullsafe/dune

@@ -12,3 +12,5 @@
  (libraries core IStdlib ATDGenerated IBase IR Absint Biabduction)
  (preprocess (pps ppx_compare ppx_sexp_conv))
 )
+
+(documentation (package infer) (mld_files Nullsafe))

infer/src/test_determinator/TestDeterminators.mld

@@ -0,0 +1,6 @@
+{0 Test Determinators}
+
+Call graph analyses that can be used to determine which tests to run
+in a given project as a result of a code change.
+
+All modules: {!module-TestDeterminators}

infer/src/test_determinator/dune

@@ -13,3 +13,5 @@
  (libraries core IStdlib ATDGenerated IBase IR CStubs)
  (preprocess (pps ppx_compare))
 )
+
+(documentation (package infer) (mld_files TestDeterminators))