Automation and Make: Running Make

Key Points

Introduction	Make allows us to specify what depends on what and how to update things that are out of date.
Makefiles	Use `#` for comments in Makefiles. Write rules as `target: dependencies`. Specify update actions in a tab-indented block under the rule. Use `.PHONY` to mark targets that don’t correspond to files.
Automatic Variables	Use `$@` to refer to the target of the current rule. Use `$^` to refer to the dependencies of the current rule. Use `$<` to refer to the first dependency of the current rule.
Dependencies on Data and Code	Make results depend on processing scripts as well as data files. Dependencies are transitive: if A depends on B and B depends on C, a change to C will indirectly trigger an update to A.
Pattern Rules	Use the wildcard `%` as a placeholder in targets and dependencies. Use the special variable `$*` to refer to matching sets of files in actions.
Variables	Define variables by assigning values to names. Reference variables using `$(...)`.
Functions	Make is actually a small programming language with many built-in functions. Use `wildcard` function to get lists of files matching a pattern. Use `patsubst` function to rewrite file names.
Self-Documenting Makefiles	Document Makefiles by adding specially-formatted comments and a target to extract and format them.
LaTeX Example	make can document how you got your results. make can simplify trying a variation of your analysis. make saves time by automating repetitive work. make saves time by only rebuilding things that depend on modified files.

Running Make

To run Make:

$ make

Make will look for a Makefile called Makefile and will build the default target, the first target in the Makefile.

To use a Makefile with a different name, use the -f flag e.g.

$ make -f build-files/analyze.mk

To build a specific target, provide it as an argument e.g.

$ make isles.dat

If the target is up-to-date, Make will print a message like:

make: `isles.dat' is up to date.

To see the actions Make will run when building a target, without running the actions, use the --dry-run flag e.g.

$ make --dry-run isles.dat

Alternatively, use the abbreviation -n.

$ make -n isles.dat

Trouble Shooting

If Make prints a message like,

Makefile:3: *** missing separator.  Stop.

then check that all the actions are indented by TAB characters and not spaces.

If Make prints a message like,

No such file or directory: 'books/%.txt'
make: *** [isles.dat] Error 1

then you may have used the Make wildcard, %, in an action in a pattern rule. Make wildcards cannot be used in actions.

Makefiles

Rules:

target : dependency1 dependency2 ...
	action1
	action2
		...

Each rule has a target, a file to be created, or built.
Each rule has zero or more dependencies, files that are needed to build the target.
: separates the target and the dependencies.
Dependencies are separated by spaces.
Each rule has zero or more actions, commands to run to build the target using the dependencies.
Actions are indented using the TAB character, not 8 spaces.

Dependencies:

If any dependency does not exist then Make will look for a rule to build it.
The order of rebuilding dependencies is arbitrary. You should not assume that they will be built in the order in which they are listed.
Dependencies must form a directed acyclic graph. A target cannot depend on a dependency which, in turn depends upon, or has a dependency which depends upon, that target.

Comments:

# This is a Make comment.

Line continuation character:

ARCHIVE = isles.dat isles.png \
		  abyss.dat abyss.png \
		  sierra.dat sierra.png

If a list of dependencies or an action is too long, a Makefile can become more difficult to read.
Backslash,\, the line continuation character, allows you to split up a list of dependencies or an action over multiple lines, to make them easier to read.
Make will combine the multiple lines into a single list of dependencies or action.

Phony targets:

.PHONY : clean
clean :
	   rm -f *.dat

Phony targets are a short-hand for sequences of actions.
No file with the target name is built when a rule with a phony target is run.

Automatic variables:

$< denotes ‘the first dependency of the current rule’.
$@ denotes ‘the target of the current rule’.
$^ denotes ‘the dependencies of the current rule’.
$* denotes ‘the stem with which the pattern of the current rule matched’.

Pattern rules:

%.dat : books/%.txt $(COUNT_SRC)
		$(COUNT_EXE) $< $@

The Make wildcard, %, specifies a pattern.
If Make finds a dependency matching the pattern, then the pattern is substituted into the target.
The Make wildcard can only be used in targets and dependencies.
e.g. if Make found a file called books/abyss.txt, it would set the target to be abyss.dat.

Defining and using variables:

COUNT_SRC=wordcount.py
COUNT_EXE=python $(COUNT_SRC)

A variable is assigned a value. For example, COUNT_SRC is assigned the value wordcount.py.
$(...) is a reference to a variable. It requests that Make substitutes the name of a variable for its value.

Suppress printing of actions:

.PHONY : variables
variables:
		@echo TXT_FILES: $(TXT_FILES)

Prefix an action by @ to instruct Make not to print that action.

Include the contents of a Makefile in another Makefile:

include config.mk

wildcard function:

TXT_FILES=$(wildcard books/*.txt)

Looks for all files matching a pattern e.g. books/*.txt, and return these in a list.
e.g. TXT_FILES is set to books/abyss.txt books/isles.txt books/last.txt books/sierra.txt.

patsubst (‘path substitution’) function:

DAT_FILES=$(patsubst books/%.txt, %.dat, $(TXT_FILES))

Every string that matches books/%.txt in $(TXT_FILES) is replaced by %.dat and the strings are returned in a list.
e.g. if TXT_FILES is books/abyss.txt books/isles.txt books/last.txt books/sierra.txt this sets DAT_FILES to abyss.dat isles.dat last.dat sierra.dat.

Default targets:

In Make version 3.79 the default target is the first target in the Makefile.
In Make 3.81, the default target can be explicitly set using the special variable .DEFAULT_GOAL e.g.

.DEFAULT_GOAL := all

Manuals

GNU Make Manual. Reference sections include:

Summary of Options for the make command.
Quick Reference of Make directives, text manipulation functions, and special variables.
Automatic Variables.
Special Built-in Target Names

Glossary

action: The steps a build manager must take to create or update a file or other object.
assignment: A request that Make stores something in a variable.
automatic variable: A variable whose value is automatically redefined for each rule. Make’s automatic variables include $@, which holds the rule’s target, $^, which holds its dependencies, and, $<, which holds the first of its dependencies, and $*, which holds the stem with which the pattern was matched. Automatic variables are typically used in pattern rules.
build file: A description of dependencies and rules for a build manager.
build manager: A program, such as Make, whose main purpose is to build or update software, documentation, web sites, data files, images, and other things.
default rule: The rule that is executed if no target is specified when a build manager is run.
default target: The target of the default rule.
dependency: A file that a target depends on. If any of a target’s dependencies are newer than the target itself, the target needs to be updated. A target’s dependencies are also called its prerequisites. If a target’s dependencies do not exist, then they need to be built first.
false dependency: This can refer to a dependency that is artificial. e.g. a false dependency is introduced if a data analysis script is added as a dependency to the data files that the script analyses.
function: A built-in Make utility that performs some operation, for example gets a list of files matching a pattern.
incremental build: The feature of a build manager by which it only rebuilds files that, either directory or indirectly, depend on a file that was changed.
macro: Used as a synonym for variable in certain versions of Make.
Make: A popular build manager, from GNU, created in 1977.
Makefile: A build file used by Make, which, by default, are named Makefile.
pattern rule: A rule that specifies a general way to build or update an entire class of files that can be managed the same way. For example, a pattern rule can specify how to compile any C file rather than a single, specific C file, or, to analyze any data file rather than a single, specific data file. Pattern rules typically make use of automatic variables and wildcards.
phony target: A target that does not correspond to a file or other object. Phony targets are usually symbolic names for sequences of actions.
prerequisite: A synonym for dependency.
reference: A request that Make substitutes the name of a variable for its value.
rule: A specification of a target’s dependencies and what actions need to be executed to build or update the target.
stem: The part of the target that was matched by the pattern rule. If the target is file.dat and the target pattern was %.dat, then the stem $* is file.
target: A thing to be created or updated, for example a file. Targets can have dependencies that must exist, and be up-to-date, before the target itself can be built or updated.
variable: A symbolic name for something in a Makefile.
wildcard: A pattern that can be specified in dependencies and targets. If Make finds a dependency] matching the pattern, then the pattern is substituted into the target. wildcards are often used in pattern rules. The Make wildcard is %.