Add chapter Packages and dependencies to the guide

doc/src/guide/book.asciidoc

@@ -18,6 +18,8 @@ include::limitations.asciidoc[Limitations]
 
 include::app.asciidoc[Building]
 
+include::deps.asciidoc[Packages and dependencies]
+
 include::compat.asciidoc[Compatibility with other build tools]
 
 = Advanced

doc/src/guide/deps.asciidoc

@@ -0,0 +1,463 @@
+== Packages and dependencies
+
+Erlang.mk can fetch and compile the dependencies that your
+project requires. Erlang.mk improves upon the concepts
+introduced by Rebar, so they should be familiar to many
+seasoned Erlang developers.
+
+Erlang.mk is not a package manager, nor is it trying to be,
+but it does include an index of Erlang packages to make
+discovering useful projects easier.
+
+This chapter will explain how to use packages, add
+dependencies to your project or bundle them directly
+in a single repository.
+
+=== Searching packages
+
+Erlang.mk gives you access to nearly 500 packages, with more
+being added regularly.
+
+To find a package, search for it:
+
+[source,bash]
+$ make search q=pool
+
+This will return all packages matching this word, like worker
+pool and acceptor pool projects.
+
+You can also list everything and use regular command line
+tools to find what you need, for example:
+
+[source,bash]
+$ make search | less
+
+// @todo Simplify adding packages, add a new chapter explaining
+// everything, then link to this new chapter from here.
+
+=== Adding dependencies to your project
+
+Once you find the package you need, adding it as a dependency
+to your project is a one-liner:
+
+[source,make]
+DEPS = cowboy
+
+And that's it! The next time you run `make`, Erlang.mk will
+fetch and compile Cowboy. Erlang.mk will also ensure Cowboy
+is available whenever you use the shell, run tests and any
+other operations.
+
+Erlang.mk will fill in the application resource file with
+all applications found in `DEPS`. But not all dependencies
+are Erlang applications, and not all dependencies need to
+be a runtime dependency. That's where the `BUILD_DEPS`
+variable comes in: it works just like `DEPS`, except the
+dependencies listed there will not be added as runtime
+dependencies.
+
+For example, you could add a parse transform project like
+this to make it available only at build time:
+
+[source,make]
+BUILD_DEPS = erlando
+
+Or you could depend on a C project directly, if you are
+building a NIF:
+
+[source,make]
+BUILD_DEPS = leveldb
+dep_leveldb = git https://github.com/basho/leveldb 2.1.3
+
+This dependency will be built before your application, so
+you could easily copy the resulting shared file into your
+'priv/' directory as part of the build process. More information
+about that in the link:ports.asciidoc[NIFs and port drivers]
+chapter.
+
+Another variable, `LOCAL_DEPS`, allows specifying runtime
+dependencies which are part of Erlang/OTP itself, but also
+dependencies that are included in the repository. Since they
+are already on your system, there is no need to fetch them.
+Do note that there is no way to choose the version, the
+application used will be the one already on your system.
+
+You could depend on the Crypto application, for example:
+
+[source,make]
+LOCAL_DEPS = crypto
+
+Erlang.mk comes with additional types of dependencies.
+It has `TEST_DEPS` for dependencies used only for testing:
+
+[source,make]
+TEST_DEPS = ct_helper
+dep_ct_helper = git https://github.com/ninenines/ct_helper master
+
+`DOC_DEPS` for dependencies used only when building documentation:
+
+[source,make]
+DOC_DEPS = edown
+
+`REL_DEPS` for dependencies required to build the release,
+or to include extra applications in the release:
+
+[source,make]
+REL_DEPS = recon
+
+And `SHELL_DEPS` for dependencies to make available when running
+the `make shell` command:
+
+[source,make]
+SHELL_DEPS = tddreloader
+
+All these will be documented in more details in their respective
+chapters.
+
+==== Modifying the dependency source or version
+
+By default, Erlang.mk will look into its package index to
+find the project you are looking for, if you only provide
+its name. This is this case:
+
+[source,make]
+DEPS = cowboy
+
+If you need a different version, you need to define another
+variable. There are two ways to do this, each being useful
+for different reasons.
+
+If you simply want to change the commit number, all you
+need to do is to define the `dep_$(DEP_NAME)_commit`
+variable. In the case of Cowboy, this would look like this:
+
+[source,make]
+DEPS = cowboy
+dep_cowboy_commit = 2.0.0-pre.2
+
+Erlang.mk will use the package index to get all information
+about Cowboy, except the commit number which will be overriden.
+
+If you need to set the fetch method or repository information
+too, for example because you want to use your own fork, or
+simply because the project is missing from the index, you
+can define the `dep_$(DEP_NAME)` variable with everything:
+
+[source,make]
+DEPS = cowboy
+dep_cowboy = git https://github.com/essen/cowboy 2.0.0-pre.2
+
+This will fetch Cowboy from your fork at the given commit.
+
+==== Fetch methods
+
+Erlang.mk comes with a number of different fetch methods.
+You can fetch from Git, Mercurial, SVN, to name a few.
+There are fetch methods that will work everywhere, and
+fetch methods that will only work in a given environment.
+
+The following table lists all existing methods:
+
+[cols="<,3*^",options="header"]
+|===
+| Name           | Format          | Description
+| git            | git repo commit | Clone the Git repository and checkout the given version
+| hg             | hg repo commit  | Clone the Mercurial repository and update to the given version
+| svn            | svn repo        | Checkout the given SVN repository
+| cp             | cp path/to/repo | Recursively copy a local directory
+| hex            | hex version     | Download the given project version from hex.pm
+| fail           | N/A             | Always fail, reserved for internal use
+| legacy         | N/A             | Legacy Erlang.mk fetcher, reserved for internal use
+|===
+
+The `git` and `hg` methods both have a repository and commit.
+You can use any valid commit, tag or branch in that repository
+for the commit value.
+
+For example, to fetch Cowboy with tag 2.0.0-pre.2 from Git:
+
+[source,make]
+dep_cowboy = git https://github.com/ninenines/cowboy 2.0.0-pre.2
+
+Or to fetch Ehsa tag 4.0.3 from Mercurial:
+
+[source,make]
+dep_ehsa = hg https://bitbucket.org/a12n/ehsa 4.0.3
+
+The `svn` method only has a repository value, but that's
+simply because the SVN repository URL can also contain
+the path and commit.
+
+This would fetch an example project from the trunk:
+
+[source,make]
+dep_ex1 = svn https://example.com/svn/trunk/project/ex1
+
+And this would fetch a separate example project from a
+specific commit:
+
+[source,make]
+dep_ex2 = svn svn://example.com/svn/branches/erlang-proj/ex2@264
+
+You can copy a directory from your machine using the `cp` method.
+It only takes the path to copy from:
+
+[source,make]
+dep_cowboy = cp $(HOME)/ninenines/cowboy
+
+Finally, you can use a package from the
+link:https://hex.pm/[Hex repository]:
+
+[source,make]
+dep_cowboy = hex 1.0.3
+
+==== Custom fetch methods
+
+If none of the existing methods fit your use, you can simply
+define your own. Erlang.mk will consider all variables that
+are named as `dep_fetch_$(METHOD)` to be available fetch
+methods. You can do anything inside this variable, as long
+as you create a folder named '$(DEPS_DIR)/$(call dep_name,$1)'.
+Or in layman terms, if your dependency is Cowboy, this would
+become 'deps/cowboy'.
+
+To give an example, this is what the Git method does:
+
+[source,make]
+----
+define dep_fetch_git
+	git clone -q -n -- $(call dep_repo,$1) $(DEPS_DIR)/$(call dep_name,$1); \
+	cd $(DEPS_DIR)/$(call dep_name,$1) && git checkout -q $(call dep_commit,$1);
+endef
+----
+
+Note that, like dependency information, this custom fetch method
+must be written before including 'erlang.mk'.
+
+=== How deps are fetched and built
+
+The order in which dependencies are fetched and built is well
+defined. This means that Erlang.mk will get the same applications
+regardless of the command or options being used.
+
+In tree traversal terms, where the list of dependencies is a
+tree, Erlang.mk fetches everything using the pre-order traversal
+method. The steps can be summarized like this, starting from
+the root application:
+
+. Fetch all dependencies for the application
+. Build first dependency
+. Build Nth dependency
+. Build last dependency
+
+Every time a dependency is built, these same steps are followed,
+recursively.
+
+Do note that the first step, fetching all dependencies of
+an application, is not guaranteed to be ordered. The reason
+for this is that it is not possible to have the same dependency
+listed twice in a single application, and therefore there can
+be no conflicts. Remember, this step only fetches, at no point
+are different applications built in parallel.
+
+What about conflicts between the dependencies of different
+applications? Simple. Since builds are ordered, this means
+that the first version of an application that is fetched
+will be the one that wins.
+
+This means that if project A depends on projects B and C,
+in this order, and that both B and C depend on a different
+version of D, it will always be B's version of D that wins,
+because we fetch the dependencies of B before fetching
+those from C.
+
+Similarly, if project A depends on projects B, C and D,
+regardless of the order, and A, B and C depend on a
+different version of D, it will always be A's version
+that wins, because we fetch all dependencies of A before
+fetching those from B or C.
+
+=== Ignoring unwanted dependencies
+
+Sometimes, you may want to ignore dependencies entirely.
+Not even fetch them. You may want to do this because a
+project you depend on depends on an application you do
+not need (like a dependency for building documentation
+or testing). Or maybe the dependency is already installed
+on your system.
+
+To ignore a dependency, simply add it to the `IGNORE_DEPS`
+variable:
+
+[source,make]
+IGNORE_DEPS += edown proper
+
+This will only ignore dependencies that are needed for
+building. It is therefore safe to write:
+
+[source,make]
+IGNORE_DEPS += edown proper
+TEST_DEPS = proper
+
+The PropEr application will be fetched as intended when
+running `make tests` or `make check`. It will however
+not be fetched when running `make` or `make deps`.
+
+=== Dependencies directory
+
+Dependencies are fetched in '$(DEPS_DIR)'. By default this is
+the 'deps' directory. You can change this default, but you
+should only do so if it was not defined previously. Erlang.mk
+uses this variable to tell dependencies where to fetch their
+own dependencies.
+
+You will therefore need to use `?=` instead of `=`. Of course,
+if you know you will never use this project as a dependency,
+`=` will work. But to avoid it biting you later on, do this:
+
+[source,make]
+DEPS_DIR ?= $(CURDIR)/libs
+
+The `$(CURDIR)` part is important, otherwise dependencies of
+dependencies will be fetched in the wrong directory.
+
+Erlang.mk will also export the `REBAR_DEPS_DIR` variable for
+compatibility with Rebar build tools, as long as they are
+recent enough.
+
+=== Dependencies local to the repository
+
+In addition to the dependencies that are fetched, Erlang.mk
+also allows you to have dependencies local to your repository.
+This kind of layout is sometimes called multi-application
+repositories, or repositories with multiple applications.
+
+They work exactly the same as remote dependencies, except:
+
+* They are not fetched
+* They are not autopatched
+* They are not deleted on `make distclean`
+* They are not automatically added to the application resource file
+
+To properly fill the application resource file, you will
+need to define the `LOCAL_DEPS` variable for each relevant
+application, the same as for OTP applications.
+
+If there is a conflict between a local dependency and a
+remote dependency, then the local dependency always wins;
+an error will be triggered when trying to fetch the
+conflicting remote dependency.
+
+To start using dependencies local to the repository, simply
+create a folder named '$(APPS_DIR)'. By default, this folder
+is the 'apps/' directory.
+
+You can use Erlang.mk to bootstrap local dependencies by
+using the command `make new-app` or `make new-lib`. This
+command will create the necessary directories and bootstrap
+the application.
+
+For example, to create a full fledged OTP application as
+a local dependency:
+
+[source,bash]
+$ make new-app in=webchat
+
+Or, the same as an OTP library:
+
+[source,bash]
+$ make new-lib in=webchat
+
+Templates also work with local dependencies, from the root
+directory of the project. You do need however to tell
+Erlang.mk to create the files in the correct application:
+
+[source,bash]
+$ make new t=gen_server n=my_server in=webchat
+
+=== Repositories with no application at the root level
+
+It's possible to use Erlang.mk with only applications in
+'$(APPS_DIR)', and nothing at the root of the repository.
+Just create a folder, put the 'erlang.mk' file in it,
+write a Makefile that includes it, and start creating
+your applications.
+
+Similarly, it's possible to have a repository with only
+dependencies found in '$(DEPS_DIR)'. You just need to
+create a Makefile and specify the dependencies you want.
+This allows you to create a repository for handling the
+building of releases, for example.
+
+=== Autopatch
+
+Erlang.mk will automatically patch all the dependencies it
+fetches. It needs to do this to ensure that the dependencies
+become compatible with not only Erlang.mk, but also with
+the version of Erlang.mk that is currently used.
+
+When fetching a dependency, the following operations are
+performed:
+
+* Fetch the dependency using the configured fetch method
+* If it contains a 'configure.ac' or 'configure.in' file, run `autoreconf -Wall -vif -I m4`
+* If it contains a 'configure' script, run it
+* Run autopatch on the project
+
+Autopatch first checks if there is any project-specific patch
+enabled. There are currently two: `RABBITMQ_CLIENT_PATCH` for
+the `amqp_client` dependency, and `RABBITMQ_SERVER_PATCH` for
+the `rabbit` dependency. These are needed only for RabbitMQ
+versions before 3.6.0 (assuming you are using upstream RabbitMQ,
+and not a fork).
+
+Otherwise, autopatch performs different operations depending
+on the kind of project it finds the dependency to be.
+
+* Rebar projects are automatically converted to use Erlang.mk
+as their build tool. This essentially patches Rebar out, and
+fixes and converts the project to be compatible with Erlang.mk.
+
+* Erlang.mk projects have their 'erlang.mk' file redirect to
+the top-level project's Erlang.mk. This is to ensure that
+functionality works across all dependencies, even if the
+dependency's Erlang.mk is outdated.
+
+* Other Erlang projects get a small Erlang.mk Makefile
+generated automatically.
+
+* Projects with no source directory and no Makefile get an
+empty Makefile generated, for compatibility purposes.
+
+* Other projects with no Makefile are left untouched.
+
+You can disable the replacing of the 'erlang.mk' file by
+defining the `NO_AUTOPATCH_ERLANG_MK` variable:
+
+[source,make]
+NO_AUTOPATCH_ERLANG_MK = 1
+
+You can also disable autopatch entirely for a few select
+projects using the `NO_AUTOPATCH` variable:
+
+[source,make]
+NO_AUTOPATCH = cowboy ranch cowlib
+
+=== Skipping deps
+
+It is possible to temporarily skip all dependency operations.
+This is done by defining the `SKIP_DEPS` variable. Use cases
+include being somewhere with no connection to download them,
+or perhaps a peculiar setup.
+
+A typical usage would be:
+
+[source,bash]
+$ make SKIP_DEPS=1
+
+When the variable is defined:
+
+* Dependencies will not be compiled or downloaded when required
+* The dependency directory '$(DEPS_DIR)' will not be removed on `make distclean`
+
+This variable only applies to remote dependencies.