Dependency (*.d) Files (Prev) Makefiles (Next)

Each namespace-level subdirectory of the src/ directory has a makefile fragment named "patterns.mk". This file defines several pattern rules used to create targets in the corresponding directory of the bld directory.

Example

As an example, here is the body of the file src/rpc/patterns.mk that controls compilation of files located in the src/rpc namespace-level directory

#-----------------------------------------------------------------------
# Variable definitions used in pattern rules

# PSCF-specific static libraries needed in src/rpc (the order matters)
# Variables $(rpc_LIB) etc. are defined in namespace config.mk files
PSCF_LIBS= $(rpc_LIB) $(prdc_LIB) $(pscf_LIB) $(util_LIB) 

# All libraries needed by main program pscf_pc (including external libs)
LIBS=$(PSCF_LIBS)

# Add header include and library paths to Gnu scientific library
INCLUDES+=$(GSL_INC)
LIBS+=$(GSL_LIB) 

# Add header and library paths to FFTW Fast Fourier transform library
INCLUDES+=$(FFTW_INC)
LIBS+=$(FFTW_LIB) 

# Arguments for MAKEDEP
MAKEDEP_ARGS=$(CPPFLAGS) $(INCLUDES)
MAKEDEP_ARGS+= -A$(BLD_DIR)/config.mk
MAKEDEP_ARGS+= -S$(SRC_DIR)
MAKEDEP_ARGS+= -B$(BLD_DIR)

#-----------------------------------------------------------------------
# Pattern rules

# Pattern rule to compile *.cpp class source files in src/rpc
# Note: Creates a *.d dependency file as a side effect
$(BLD_DIR)/%.o: $(SRC_DIR)/%.cpp
   @SDIR=$$(dirname "$@"); if [ ! -d "$$SDIR" ]; then mkdir -p "$$SDIR"; fi
   $(CXX) $(CPPFLAGS) $(INCLUDES) $(CXXFLAGS) -c -o $@ $<
   $(MAKEDEP) $(MAKEDEP_CMD) $(MAKEDEP_ARGS) $<

# Pattern rule to link executable Test programs in src/rpc/tests
$(BLD_DIR)/%Test: $(BLD_DIR)/%Test.o $(PSCF_LIBS)
   $(CXX) $(LDFLAGS) -o $@ $< $(LIBS)

# Note: In the linking rule for tests, we include the list $(PSCF_LIBS) 
# of PSCF-specific libraries as prerequisites but link to the list 
# $(LIBS) of libraries that includes external libraries

# Note: There are no *.cu CUDA source files in src/rpc, and so no rule to 
# compile them is defined in this directory.

This file is divded into two main sections. The first section defines several variables that are used in pattern rules, and the second defines several pattern rules.

Makefile variable definitions

The first part of this file defines several variables that are used in the actual pattern rules, some of which are also used in makefiles that include this pattern rule file. These are:

PSCF_LIB : List of paths to relevant static libraries that are created by the PSCF build system
LIBS : A list of paths or names of all relevant libraries, including those associated with any relevant external packages
INCLUDES : A list of any additional paths to non-standard directories containing header files for external libraries
MAKEDEP_ARGS : A list of options that is passed to a python script that is used to construct dependency files.

Several of these variables are discussed in more detail below.

PSCF_LIB : The variable PSCF_LIB is constructed by concatenating values of variables such as $(util_LIB), $(pscf_LIB), etc. that contain absolute paths to static library files that are created by the PSCF build system. Each of these variables is defined in a sources.mk file is located in the same namepace-level directory as the source files that are compiled to create the relevant library. The value of PSCF_LIBS constructed in the patterns.mk file in each namespace-level directory only includes libraries associated with that namespace-level directory and other such directories that it depends upon. In this example, the value of PSCF_LIB thus contains paths to libraries associated with the util, pscf, prdc, and rpc namespace-level directories, but not with the r1d or rpg program-level directories.

LIBS : The variable LIBS is created by initializing its value to PSCF_LIB, and then adding makefile variables that contain compiler options to add names of external libraries that should also be linked to create executable files. In the above example, the value of LIBS is given by the concatenation

LIBS = $(PSCF_LIBS) $(GSL_LIB) $(FFTW_LIB)

Here and are compiler options that link to the GSL and FFTW libraries, respectively. Values of these variables are set in the config.mk file, and are usually constructed using the -l compiler option to link to specific libraries, and the -L option to add additional directories in which to search for library files. For example, the variable FFTW_LIB is normally assigned a value

FFTW_LIB=-lfftw3

by default on a standard linux system.

INCLUDES : The variable INCLUDES may contains compiler options that add paths to non-standard directories that should be searched for header files that are included into PSCF source code files via C preprocessor #include directives. Such additional search paths are generally needed only to find header files for external libraries have been installed in non-standard locations. The INCLUDES variable is initialized to an empty string in the config.mk file, and is then completed within each patterns.mk file by adding variable values that may contain options to add additional search paths for directories containing header files for particular libraries. In the above example, the final value of INCLUDES is given by a concatenation

INCLUDES = $(GSL_INC) $(FFTW_INC)

The value of each variable such as GSL_INC and FFTW_INC may be empty, or may contain an option to add a non-standard directory that contains the header for a particular library to the list of directories that will be searched by the preprocessor. The value of each variable such as GSL_INC and FFTW_INC is defined in the config.mk file, and is set to an empty string for any library whose header file is installed in a standard locations such as /usr/include or /usr/local/include that the compiler searches by default.

For each library that installs a header file in a non-standard standard location, the associated *_INC variable may instead be set to a string that starts with the "-I" compiler option, followed by the absolute path to a directory containing any required header file. For example, on a particular Apple laptop on which the GSL library was installed by homebrew, the value

GSL_INC=-I /opt/homebrew/Cellar/gsl/2.8/include

was required to tell the compiler where to find header files for a particular version of GSL.

Use of GSL is simplified by the fact this package installs a utility named gsl-config that reports where different types of file were installed. This utility is used by the PSCF configure script to automatically generate the required paths. Paths related to the GSL package are thus usually set correctly in the config.mk files that are installed by the configure script.

Pattern Rules

Each patterns.mk file defines pattern rules for creating *.o object files by compiling *.cpp and/or *.cu source files, and for creating executable files that run unit tests. Analogous rules in different patterns.mk files have very similar structure.

We first consider pattern rules for creating *.o object file targets. The patterns.mk file in each namespace-level directory that contains C++ source files with file name extension .cpp contains a rule for compiling such *.cpp file to create a *.o object file. In directories that contain CUDA C++ source files with file name extension .cu, the patterns.mk file will contain a pattern rule for compiling these files, which uses the NVIDIA CUDA compiler rather than a C++ compiler. Both rules may exist in patterns.mk directories that contain both types of source file.
Both types of rule creating an object file with file name extension *.o in a subdirectory of the build directory, $(BLD_DIR), by compiling a .cpp or *.cu source file with the same base name in the corresponding subdirectory of the source directory, $(SRC_DIR).

Rule: Compiling C++ source files

A pattern rule for compiling standard C++ source files always looks something like this:

$(BLD_DIR)/%.o: $(SRC_DIR)/%.cpp
    @SDIR=$$(dirname "$@"); if [ ! -d "$$SDIR" ]; then mkdir -p "$$SDIR"; fi
    $(CXX) $(CPPFLAGS) $(INCLUDES) $(CXXFLAGS) -c -o $@ $<
    $(MAKEDEP) $(MAKEDEP_CMD) $(MAKEDEP_ARGS) $<

Pattern rules for compiling CUDA files are closely analogous, and will be discussed separately below.

In this pattern rule, the symbol % is a wildcard that represents a string that appears in the path for the target file that may also appear in one or more pre-requisites. The rule shown above thus describes how to create a target object file in the build directory with a path of the form $(BLD_DIR)%.o by compiling a corresponding source file in the source directory with a path of the form $(SRC_DIR)/%.cpp, in which the symbol % represents a string that appears in both paths. The shared part of these paths may include one or more directory separator symbols and the name or names of subdirectories of the src directory, as well as the base file name. The PSCF bld directory is required to have the same internal directory structure as the src directory in order to allow this pattern rule to work correctly for out-of-source builds,

The recipe of the rule shown above contains three lines with different purposes. The first defines a sequence of shell command that checks if the directory in which the target should be installed exists, and attempts to create it if that directory does not yet exist. This part of the recipe is a convenience, and is not an essential part of the rule. The second line of the recipe is the heart of the matter: This line is a shell command that invokes the compiler command, given by the makefile variable value $(CXX), in order to compile the source file and create the object file target. The third line of the recipe calls a script, whose name is given by the variable value $(MAKEDEP), that creates a corresponding dependency file with extension .d.

The two essential actions taken by this pattern rule are thus to compile the source file to create an *.o object file, and to create a *.d dependency file as a side-effect of compilation. These two actions are discussed separately below.

Source file compilation :

The line of the recipe that actually compiles the source file in this example is given by

    $(CXX) $(CPPFLAGS) $(INCLUDES) $(CXXFLAGS) -c -o $@ $<

In this recipe, a standard makefile symbol $@ is used to represents the target, which must match the pattern $(BLD_DIR)/%.o, while $< represents the first prerequisite, which is a source file that must match the pattern $(SRC_DIR)/%.o. The following additional makefile variables are used in the second line of the recipe to define a rule for compiling the source file:

CXX: name of the C++ compiler command
CPPFLAGS: standard flags for the C preprocessor
INCLUDES: additional options to add directories to search for included header files for external libraries (if any)
CXXFLAGS: flags for the C++ compiler (optimization, warnings, etc.)

Values for the variables BLD_DIR, SRC_DIR, CXX, CPPFLAGS, and CXXFLAGS are all defined in the relevant config.mk configuration file. The value of the INCLUDES variable is initialized to an empty string in the config.mk file and then added to as needed within each patterns.mk file, as discussed above.

Default values for some of these variables on a linux system, as assigned by the file make/compiler/linux-default are:

  CXX = g++
  CPPFLAGS = -I $(SRC_DIR) -D UTIL_CXX11
  CXXFLAGS = -Wall --std=c++11 -O3

The value of INCLUDES may be empty, or may contain options to add non-standard directories to the search path for included files, as discussed above.

The option -I $(SRC_DIR) in the value of $(CPP_FLAGS) instructs the preprocessor to always search the PSCF src directory for header files. The design of the PSCF source code assumes that this option will always be present. This option allows the PSCF source code to use a path that is defined relative to the src directory to include a header file that is located anywhere in the PSCF src directory tree.

The "-D" compiler option is used to define C/C++ preprocessor macros when calling the compiler. The option "-D UTIL_CXX11" option is necessary because the src/util directory contains a few files that optionally compile code that uses features of C++11 language standard only if the preprocessor macro UTIL_CXX11 is defined. Most of the PSCF source code is designed to use the C++11 language standard, but a few older files in the util directory were designed to use C++99 by default and to use new features of C++11 only if this macro is defined.

If the makefile variable UTIL_DEBUG is defined, then an additional compiler option string "-D UTIL_DEBUG" will be appended to the end of $(CPPFLAGS) within the config.mk file, giving a modified value

   CPPFLAGS = -I $(SRC_DIR) -D UTIL_CXX11 -D UTIL_DEBUG

This added option defines a C/C++ preprocessor macro named UTIL_DEBUG that enables conditional compilation of additional run-time sanity checks, such as run-time checks on the bounds of array indices. Note that the makefile variable UTIL_DEBUG and the preprocessor macro UTIL_DEBUG have the same name, and similar purposes (both enable additional run-time checks) but are used in different contexts (i.e., in makefiles vs. in C++ code).

Automatic generation of a dependency file :

The variables MAKEDEP, MAKEDEP_CMD, and MAKEDEP_ARGS are used only in the third line of the example shown above, in order to construct a *.d dependency file. Values of MADEDEP and MAKEDEP_CMD are defined in the config.mk file, while the value of MAKEDEP_ARGS is defined in a patterns.mk file.

The value of MAKEDEP is the absolute path of an executable script that is used to generate dependency files for C++ files. The value of this variable is assigned in the config.mk file, where it is assigned a value $(BIN_DIR)/makeDepCpp by default. Generation of dependency files can be suppressed by commenting out the line that defines MAKEDEP.

The value of MAKEDEP_CMD is the name of a compiler command and compiler options required to use a compiler to generate a list of pre-requisites of a C++ by analyzing preprocessor #include directives. The specified compiler command is called internally by the makeDep script to generate a list of dependencies. The MAKEDEP script then does some further manipulation to convert paths that are reported by the compiler as relative paths into absolute paths. The variable MAKEDEP_CMD is assigned a value in the main config.mk file, where it is set to "-C\$(CXX) -MM -MF" by default.

The value of MAKEDEP_ARGS is a string of other arguments required by the MAKEDEP script. The value of this variable is defined in the patterns.mk file.

Rule: Compiling CUDA C++ source files

In namespace-level directories that contain CUDA code, the patterns.mk file contains a separate pattern rule for compiling CUDA C++ source files with file name extension *.cu. Such a rule always looks like this:

$(BLD_DIR)/%.o: $(SRC_DIR)/%.cu
    @SDIR=$$(dirname "$@"); if [ ! -d "$$SDIR" ]; then mkdir -p "$$SDIR"; fi
    $(NVXX) $(CPPFLAGS) $(INCLUDES) $(NVXXFLAGS) -c -o $@ $<
    $(MAKEDEP_CUDA) $(MAKEDEP_CUDA_CMD) $(MAKEDEP_ARGS) $<

This pattern rule is closely analogous to that used to compile *.cpp files. The differences are:

The file name pattern for the source file prerequisite uses a file name extension .cu rather than .cpp
The name of the compiler is represented by a variable value $(NVXX), which gives the name of the NVIDA nvxx CUDA C++ compiler.
The variable value $(NVXXFLAGS) contains a set of compiler flags that are designed for the NVIDIA compiler, which are generally different from those used by the C++ compiler.
The rule for creating a dependency file uses different variables $(MAKEDEP_CUDA) and $(MAKEDEP_CUDA_CMD) to represent the name of the script that is used to create the dependency list and the name of the compiler command that it calls internall, respectively.

Values of NVXX, NVXXFLAGS, MAKEDEP_CUDA and MAKEDEP_CUDA_CMD are all defined in the config.mk file, while the value of MAKE_DEP_ARGS is defined in the patterns.mk file.

Rule: Creating unit test executable files

Every namespace-level subdirectory of the src directory has a subdirectory named "tests" that contains code for unit tests. Within each such "tests" directory are one or more source files that can be compiled and linked to create executables that run unit tests. By convention, such files are compiled and linked to create executable files with names that end in Test (e.g., Test, cpuTest or cudaTest). The name of the object file used to create each such executable is given by the name of the executable followed by a .o filename extension. Each tests directory may contain multiple source files for unit tests, some which may be located in subdirectories of the "tests" directory, each of which is designed to run a different subset of the available unit tests. Because there may be multiple source files for unit tests that obey a set of conventions, a pattern rule is used.

The pattern.mk file in each namespace-level directory contains a pattern rule that is designed to create executable files by linking a corresponding object file to a list of relevant libraries. The rule generally in a directory that contains only standard C++ files generally looks like this

$(BLD_DIR)/%Test: $(BLD_DIR)/%Test.o $(PSCF_LIBS)
    $(CXX) $(LDFLAGS) -o $@ $< $(LIBS)

The symbols $@ and $< refer to the target of the rule (the executable) and the first prerequisite (the object file). The meanings of the variables used in this rule are:

CXX is the name of the compiler that is used as a linker
LDFLAGS is the a set of flags that is passed to the linker, which is usually empty.
PSCF_LIBS is a list of relevant static libraries located within namespace-level directories in the PSCF repository.
LIBS is a list of compiler options that add all relevant libraries, including libraries associated with external packages, including, e.g., the FFTW and GSL libraries.

The variable CXX is replaced by NVXX in some subdirectories that contain CUDA code. External libraries are added to the LIBS variable using the "-l" option before the name of each library. Values of the compiler name variable (CXX and NVXX) and LDFLAGS are defined in the config.mk file. Values of PSCF_LIBS and LIBS are defined within the patterns.mk file.

Note that the makefile rule for unit test executables uses the value $(LIBS) in the recipe but $(PSCF_LIBS) in the prerequisite list. This usage guarantees that all relevant libraries are linked by the linker recipe, including external libraries, but that the PSCF makefile system is only responsible for updating out-of-date libraries that are part of the PSCF package.

Dependency (*.d) Files (Prev) Build System (Up) Makefiles (Next)