|
PSCF v1.1
|
Directory Structure (Prev) Coding Standards (Next)
PSCF is compiled using a system of unix makefiles. All development was done using gnu make (gmake).
All C++ and CUDA files for PSCF, are in the pscfpp/src/ directory tree. The header and source file for each class are in the same directory. The name of each C++ file is the same as the class name, followed by an extension to indicate file type. We use extension .h to indicate a header file, .tpp to indicate the implementation of a class template, .cpp to indicate a C++ source file, and .cu to indicate a CUDA source file. All class names and corresponding file names are upper space camel (like Util::Vector or Pscf::Basis).
The source code in pscfpp/src is divided among two top-level namespaces, named Util and Pscf.
The Util namespace contains a collection of utilities for scientific computation that is also used by other projects. All code in the Util namespace is contained in the src/util directory. This directory contains the contents of a separate github git repository (repository dmorse/util) that is imported into the pscfpp as a submodule.
The Pscf namespace contains all C++ and CUDA code that is specific to the PSCF project. The Pscf namespace contains several enclosed namespaces that each contain code that is used only by one program or set of closely related programs.
The main subdirectories of src/ are:
These five directories will be referred to in what follows as "namespace" level subdirectories of src/. The PSCF makefile system constructs a static library in each of these namespace level directories that contains compiled code for all of the classes and global functions defined in that directory. The src/fd1d, src/pspc, and src/pspg each contain one or more main program source files that are also compiled and installed by the build system.
Each of the five namespace level source directories contains a subdirectory named tests/ that contains unit tests for classes defined in the associated name space. These unit tests are not automatically compiled or run by the build system that compiles the source code.
When source files from the src/ directory are compiled, the resulting object (*.o) files and other files generated by compilation are placed in a directory that we refer to as the build directory. The PSCF makefile system can implement either an "out-of-source" build, in which the build directory tree is distinct from the src/ directory, or an "in-source-build", in which all generated files are placed in the src/ directory alongside the source files. The pscfpp/bld/ directory is used as the build directory for out-of-source compilation.
The choice between in-source and out-of-source builds is determined by where one invokes the "make" command: Invoking "make" from within the src/ directory causes the code to be built in source, while invoking "make" from the pscfpp/bld directory causes the code to be built out-of-source in pscfpp/bld.
Compiling a source file normally creates an object file and a dependency file with the same base name as the source file but with file name extension *.o and *.d, respectively. Each dependency file contains a makefile target rule that lists all the files within the src/ directory tree upon which the associated *.o object file depends, including all the header files that it includes. The object and dependency fields are placed in the same subdirectory of the build directory tree.
After the setup script has been run, but before any code has been compiled, each of the two possible build directories (src/ and bld/) contain the following files and namespace level subdirectories:
Here, we use the BLD_DIR to represent the path to a specific build directory. The purposes of the files in this directory are:
After setup but before compilation, each of the five namespace level subdirectories (util/, pscf/, fd1d/, pscp/, and pspg/) of each build directory also contains a few makefile fragments with similar names, of the form:
Here, we use "namespace/" to denote a namespace level directory name, such as "util/" or "pspc/". The purposes of these files are:
The makefile in each namespace level subdirectory contains an "all" target that compiles all of the source files in the corresponding namespace (or subdirectory), creates a static library that contains all of the resulting objects, and compiles and installs any main programs in that directory. After setup but before compilation, the bld/ directory is empty except for the files listed above. After setup, the src/ directory also contains copies of these files, in addition to source files and other makefile fragments.
The config.mk files in the root of each build directory tree (i.e., in src/ and bld/) and in its namespace level subdirectories will be referred to in what follows as build configuration files. Each such build configuration file file is a makefile fragment that is installed by the setup script by making copies of default versions stored in the directory make/config.
The main config.mk in each build directory defines makefile variables that contain paths to the source, build and binary directories, and other variables that define the choice of compiler and various compiler options.
The following variables defined in the main config.mk file define paths:
In the main config.mk in each build directory, BLD_DIR is given by the absolute path for that directory. By default, BIN_DIR is the absolute path to the pscfpp/bin directory. The values of the above three variables are expressed in terms of a variable ROOT_DIR that contains the absolute path to the pscfpp/ root directory. The correct value for ROOT_DIR is set by the setup script. Here is an example of the relevant part of the file bld/serial/config.mk:
Here, for simplicity, we have assumed that the pscfpp/ root directory is a subdirectory of the users home directory. In the actual file, this would be replaced by the literal absolute path to whatever parent directory contains the pscfpp/ directory.
sub
The "config.mk" file in each of the namespace level subdirectories of src/ and bld/ contains definitions of variables that are used in the pattern rules used to compile code in that directory.
The makefile in each namespace level subdirectory of each build directory defines an "all" target. The "all" target compiles all of the source files in the corresponding subdirectory of src/, or the corresponding namespace.
The "all" target in each namespace level directory also creates a static library that contains the object files for all classes in the associated namespace. The base name of each such library is given by a prefix "lib" followed by the namespace level directory name with a file extension ".a". Each such library is placed in the corresponding namespace level build directory. For example, the makefile in the util/ subdirectory creates a static library that (by default) is named BLD_DIR/util/libutil.a. Executable files are built by linking main program files to these namespace level libraries.
The "all" targets of the makefiles in the fd1d, pspc, and pspg namespace level subdirectories also compile main programs and install the resulting executables in the BIN_DIR directory. Entering "make all" from the BLD_DIR/fd1d directory compiles and links the main program src/fd1d/pscf_fd.cpp, in addition to compiling all of the required class files and aggregating the source files into a static library.
Every subdirectory of src/ (other than the tests/ directories) contains a makefile fragment named "sources.mk". Each such file defines a variable that contains a list of the source files in that directory and all of its subdirectories (if any). In each such subdirectory of src/, this variable has a name of the form [directory]_, where "[directory]" represents a mangled form of the subdirectory name. Specifically, the [directory] string is constructed by taking the path from the src/ directory to the subdirectory of interest and replacing each "/" directory separator by an underscore ("_"). For example, the file src/util/sources.mk defines a variable util_ that expands to a list of all of the source files in the directory tree rooted at src/util. The file src/util/space/sources.mk defines a corresponding variable named util_space_. The value of the [directory]_ variable is a list of relative paths for all source files in the corresponding directory and its descendant subdirectories, in which the path to each source file is expressed relative to the pscfpp/src/ directory.
The code that assigns a value to a [directory]_ source list variable is straightforward in directories that contain source files but no subdirectories, such as the util/space directory. For example, here is the definition of util_space_ from the file src/util/space/sources.mk:
The directory src/util/space contains four sources files named Grid.cpp, IntVector.cpp, Vector.cpp and Tensor.cpp, and has no subdirectories.
In source file directories that do contain subdirectories, however, the source file list is constructed by concatenating corresponding lists defined in the source.mk files of subdirectories. In this case, the "sources.mk" file in the parent directory must include the sources.mk files from all of subdirectories. The value of the [directory]_ variable for the parent directory is then constructed by concatenating values of the [subdirectory]_ variables, and then appending additional source files from the parent directory, if any. For example, here is the makefile code that defines the variable util_ in the file src/util/sources.mk:
The first segment of the file is a set of include statements that include the sources.mk files from all subdirectories of src/util. The definition of util_ then simply concatenates the corresponding source file lists for all subdirectories.
The sources.mk file in each such subdirectory of src/ also defines two variables that expand into lists of absolute paths for all source files and all object file targets in that subdirectory and its subdirectories. These variables have names of the form [directory]_SRCS and [directory]_OBJS, respectively, where [directory] again represents the mangled directory name. Thus, for example, the file src/util/sources.mk defines variables named util_SRCS and util_OBJS, in addition to util_. The [directory]_SRCS and [directory]_OBJS variables are constructed by using the gmake "addprefix" function to add absolute paths for the source and build directories as prefixes to each relative path listed in the [directory]_ variable. The relevant lines in the file src/util/sources.mk look like this:
Here, SRC_DIR expands to the absolute path to the pscfpp/src directory and BLD_DIR expands to the absolute path to the relevant build directory. Note that the only difference between the paths in a [directory]_ variable (e.g., util_) and those in a [directory]_SRCS variable (e.g., util_SRCS) is that the [directory]_ variable contains relative paths, relative to src/, while the [directory]_SRCS variable contains corresponding absolute paths.
Each namespace level subdirectory of the src/ directory has a makefile fragment named "patterns.mk". This file defines pattern rules for compiling source files in that directory or namespace. The pattern rule has the same structure in all five namespace level directories. In directories that contain C++ files, the main rule always looks like this:
This target pattern creates a rule for creating an object file with file extension *.o in a subdirectory of the build directory BLD_DIR by compiling a *.cpp source file with the same base name in the corresponding subdirectory of the source directory SRC_DIR. The first command actually compiles the source file and creates the object file. The second command, which is within an ifdef MAKEDEP .. endif block, generates a dependency file with extension *.d as a side-effect of compilation.
The following variables used in the above pattern are defined in the main config.mk file file in the build directory from which make was invoked:
The same values are used for the above variables in all five namespaces. Two additional variables are defined within each patterns.mk file that are assigned different values in different namespaces. These are:
The DEFINES variable, if not empty, contains a series of preprocessor macro definitions, defined using the "-D" compiler option. Thus, for example, if debugging is enabled, the value of DEFINES within the file src/inter/patterns.mk might expand to
The definitions of DEFINES and MAKE_DEPS within the file src/pspc/patterns.mk are shown below as an example:
The DEFINES variable is constructing by concatenating the NAMESPACE_DEFS variables (UTIL_DEFS, PSPC_DEFS, etc.) for this namespace (i.e., PSPC_DEFS) and for all namespaces upon which this namespace depends (i.e., UTIL_DEFS). For the Pscf::Pspc namespace, this yields a concatenation of UTIL_DEFS and PSPC_DEFS. The resulting string contains all of the preprocessor definitions that are allowed to be used by code in this namespace.
The MAKE_DEPS variable contains a list of build configuration files whose contents can effect compilation of source code in this namespace. This list always contains the main config.mk file from the parent build directory, the config.mk file for the namespace of interest, and config.mk files for any namespace(s) upon which this namespace depends. For the Pscf::Pspc namespace, MAKE_DEPS thus contains paths to the config.mk files in the pspc/ and util/ subdirectories. These build configuration files are added to the list of dependencies for every object file in this namespace because changes to these config.mk files can change which preprocessor macros are defined or which compiler options are set, and thus can change the contents of the resulting object files.
If automatic dependency generation is not disabled, a dependency file will be generated for each source file whenever the file is compiled, as a side effect of compilation. Each dependency file defines a makefile rule for constructing the corresponding object file. This rule can be used to determine when the source file needs to be recompiled as a result of changes in header files or other files upon which it depends. The rule consists of the name of the object file target followed by a colon and list of files upon which it depends, of the form
The list of dependencies for each object (*.o) file target includes the corresponding source (*.cpp) file, all of the header files that this source file directly or indirectly includes. The end of this list also includes all of the configuration files specified by the MAKE_DEPS variable for the relevant namespace. Each such makefile rule only lists dependencies, but does not contain an explicit command, because the required compiler command is given implicitly by the appropriate pattern rule.
Though not shown in the above example, all paths to the target and its dependencies are actually given in dependency files as absolute paths.
Dependency files are created by the bash script pscfpp/bin/makeDep. The pattern rules defined in the namespace level patterns.mk files apply the makeDep script to a source file whenever that file is compiled. The "makeDep" script works by calling the gnu g++ compiler with the -MM option to determine dependencies, and using a separate python script to edit the resulting file.
In each makefile at the namespace level and below, all relevant dependency files are included into the makefile by a command of the form
in which [directory] represents the mangled name of the directory containing the makefile. For example, the file src/mcMd/makefile contains a line
This command attempts to include a *.d dependency file for each *.o object file in the object file list. The dash in front of "include" instructs "make" to continue quietly if no dependency file is found for any object file target.
As an example of how the system works, here are the essential parts of the namespace level makefile from the directory bld/serial/inter. The actual makefile also includes "clean" and "veryclean" targets that have been excluded for clarity. Corresponding makefiles in bld/parallel/inter and src/inter are identical:
This simplified makefile contains three blocks:
(1) The block of include statements at the top of this file include all of the relevant build configuration files, the appropriate patterns.mk file, and the appropriate source file lists. The main config.mk file must be included using a relative path BLD_DIR_REL_=.. to the parent build directory. The main config.mk file defines variables BLD_DIR and SRC_DIR that contain absolute paths to the object and source directories, which can be used in the remainder of this makefile, and in makefile fragments that it includes. Next, the makefile includes the namespace level config.mk files for this namespace and any namespaces that it depends on. In this example, it thus includes the inter/config.mk and util/config.mk files from the build directory, because the Pscf::Pspc namespace depends upon the Util namespace. Finally, the makefile includes the patterns.mk file and sources.mk file list from the associated namespace level source directory src/inter.
(2) The "all" target builds all of the object files in the Pscf::Pscf namespace, which are listed in the variable inter_OBJS, and builds the associated library, whose name given by the variable inter_LIB. The file inter/sources.mk defines both the object file list inter_OBJS and a target rule for building the library inter_LIB. The library name inter_LIB is defined in the namespace level configuration file inter/config.mk.
(3) The last line of the above makefile fragment instructs make to include all of the dependency files for all of the source files in this namespace. The syntax $(inter_OBJS:.o=.d) creates a list of vfiles in which the .o extension of each object file is replaced by a .d extension. The dash in front of the -include command tells make to continue quietly if any dependency file does not exist.
The first time "make all" is invoked in this directory, neither the object file nor the associated dependency files exist. In this case, since each object target file is absent, make knows that it must be built, and uses the pattern rule defined in inter/patterns.mk to create each object file listed in inter_OBJS. If "make all" is a called again from the same directory, however, make can use the information in the dependency files to determine which object files needs to be rebuilt as a result in changes in C++ or configuration files.
The target rule for library inter_LIB lists all of the object files in inter_OBJS as dependencies. The library will thus be built only after the object files are all built, and will automatically be rebuilt if any object file in the Pscf::Pspc namespace is rebuilt.
The makefile for the inter/ namespace level directory, which we used above as an example, does not build any executable files. Corresponding makefiles for the mcMd/, ddMd/, and tools/ subdirectories of each build directory, however, also contain targets that that build executables. Each executable is built by linking an object file that is created by compiling a main program file to several namespace-level static libraries. For example, the mcSim and mdSim executables are created by compiling the main program source files src/mcMd/mcSim.cpp and src/mcMd/mdSim.cpp and then linking the resulting object files to static libraries from the mcMd/, inter/ and util/ build directories. The target rules for the final executables list the required static libraries as dependencies.
The fact that executable targets depend on libraries that are defined in other namespaces requires the following further changes to the structure of makefiles that contain executable targets:
Thus, for example, the mcMd/makefile in each build directory must include the sources.mk files from the inter/ and util/ subdirectories of the src/ directory, and must include all dependency files from corresponding subdirectories of BLD_DIR, in addition to files from the mcMd/ subdirectory. Inclusion of these additional files from other namespace level directories allows make to decide when any required library needs to be rebuilt, and provides instructions how to rebuild any required source file or library if necessary.
As a convenience for developers, every subdirectory src/ at the namespace level and below contains a makefile. The "all" target of each such src/ subdirectory makefile executes an in-source build of the files in that subdirectory and any descendant subdirectories. The structure of the makefiles in subdirectories of the namespace level directories is very similar to that of the namespace level makefiles, except for the absence of targets to build a library or executables.
During development, the most convenient way to test whether a source file that you have just modified compiles correctly is usually to just enter
from the directory that contains the new or newly modified source file. If automatic dependency generation is enabled, and if you only modify one class at a time, this will normally cause the build system to compile only the source file that you have just modified. If a directory contains more than one incomplete or broken file, you should temporarily comment out all but one of these files (the one you are working on) in the definition of the [directory]_ variable in the corresponding sources.mk file, and then uncomment each such file when you are ready to work on it.
The file name conventions used in the PSCF build system make it awkward for a user to explictly request compilation of a specific source file. The build system uses absolute paths for all source and object files. The "make" program is not smart enough to recognize when an absolute and relative path are equivalent. As a result, the only way explicitly request compilation of a specific source file is to use the absolute path to the object file as a makefile target. For example, if the absolute path to the root PSCF directory within a user account named "george" was "/home/george/pscfpp/", you (or George) could compile the source file src/mcMd/simulation/Simulation.cpp in-source by entering
or
from either the src/mcMd/simulation or src/mcMd/ directory. Most programmers, however, are unwilling to repeatedly type absolute paths. This is why we recommend getting in the habit of simply using "make all" from the directory that contains the file of interest, and working on only one file at a time.
Directory Structure (Prev) Developer Information (Up) Coding Standards (Next)
1.9.5