|
Simpatico
v1.10
|
2.11.2 Multi-system simulations: ddSim (Next)
MPI-enabled versions of mcSim and mdSim allow different processors within an MPI communicator to simulate different systems, with one system per processor. The names of MPI-enabled versions of these programs include a suffix _m (for MPI), and are called mcSim_m and mdSim_m if no other optional features are enabled.
The multi-processor versions of mcSim or mdSim can be invoked in either "independent" mode or "perturbation" mode. In "independent" mode, these programs simply run a set of completely independent simulations, with completely separate input files for every processor. In "perturbation" mode, these programs use different input configuration files for different processors but use a single parameter and command file to control all processors. In perturbation mode, the parameter file contains instructions for assigning a sequence of slightly different set of parameter values to different processors. Both modes are described in more detail below.
In either mode, conventions for paths to input and ouput file names are different for multi-system simulations than those for single-system simulations. Each processor in an MPI job is assigned a integer "rank" in the range n = 0, ..., np-1, where np is the total number of processors. In a multiprocessor mcSim or mdSim simulation, all or many of the files associated with a processor with rank "n" are placed in a numbered subdirectory of the working directory named "n/", e.g., in a directory named "3/" for the processor with MPI rank 3.
To use a parallel version of mcSim or mdSim to run n completely independent simulations, one simply enters the name of the executable from, such as "mcSim_m" or "mdSim_m", from a directory that contains a set of numbered directories named 0/, 1/, 2/, .. n-1/ associated with different processors. In this mode of operation, every processor reads a completely different set of param, command and configuration input files, all of which are located in a numbered directory associated with that processor. This file convention is enforced by prepending the name of the numbered directory to the names of all input and output files that are opened by the progam, e.g., by prepending a prefix "3/" to the names of any files that are open by the processor with MPI rank 3. This manipulation of file names is done by the FileMaster class.
In independent mode, the paths to the parameter and command file for each system are files in the associated numbered directory with base names specified as arguments to the -p and -c command line options. Paths to other input files (such as the input configuration file) and output files (such as those created by analyzer classes) are also defined relative to the relevant numbered directory. All such path names are obtained obtained by adding a the prefix "n/" for processor n before the parameter and command file names, and before the inputPrefix or outputPrefix strings specified in the parameter file.
As an example, consider a set of MC simulations on 3 processors. Assume that the param file for each processor sets the inputPrefix string to "in/" and the outputPrefix to "out/". Also assume that each command file contains a command "READ_CONFIG config" that read the initial configuration from a file named "config" in the appropriate data input directory. Such a simulation can be run by entering
from root of a directory tree with a structure
The command "READ_CONFIG config" in each command file will cause each processor to read the input configuration file "in/config" in the appropriate numbered directory for that processor. When the simulation is run, all configuration and data analysis output files for each processor will be written to the "out" directory of the corresponding numbered directory.
To invoke an parallel version of mcSim or mdSim in "perturbation" mode, simply invoke the program with the "-f" option. This command line option only functions in versions of the code that were compiled with MPI enabled. In this mode, simulations carried out on different processors are controlled by a single parameter file and a single command file, but different parameter values are assigned to different processors. This mode of operation is required for either free energy perturbation or replica exchange simulations.
For example, to run a MC program in this mode on 3 processors, one might thus enter
from the root directory of a directory tree with a structure
In this case, a single parameter file named "param" and command file named "commands" is read from the common root directory, and used to control the simulations on all three processors. In this mode, separate input configuration files are read from the in/ subdirectory of the associated numbered subdirectory, and output files produced by each processor are written to files within the associated numbered directory.
The method by which a single parameter file is used to set different parameter values on different processors involves the use of a Perturbation class, and is discussed below.
In perturbation mode, the master processor reads a parameter file, and broadcasts the contents of that file to all the other processors. This establishes a baseline set of parameters that are the same on all processors. The parameter file used in such simulations must also, however, contain a polymorphic block associated with a subclass of the McMd::Perturbation abstract base class. The "Pertubation" object must define how a sequence of different parameter values should be assigned on different processors. The block appears at the end of the block associated with the McSystem or MdSystem.
Each subclass of Perturbation defines how a sequence of slightly different sets of different physical parameters should be assigned to different processors in an MPI communicator, using the parameters set in the remainder of the parameter file as a common starting point. A subclass of Perturbation may assign different values to different processors for any parameters that effect the statistical weight for a microstate. To do so, it may modify values of either macroscopic state variables such as temperature or pressure, or parameters in the potential energy. The default implementation of the parameter file block associated with a Perturbation reads in an array of values for some parameter, with a different value is assigned to each processor. The implementation of each subclass of Perturbation must define how these input parameters should be used to reset values of one more more of the physical parameters of a simulation. Different parameters are set on different processors by first setting the parameters on all processors to common baseline values, which are determined by the value given for quantities such as the potential energy parameters and temperature in the rest of the parameter block, and then calling a virtual method of the Perturbation object to reset specific parameters to different values on different processors.
As a simple example of this, McEnergyPerturbation is a subclass of Perturbation that sets every processor in an MC simulation to a different temperature, or inverse temperature beta, while using the same values for all potential energy parameters on all processors. The required values of inverse temperature for different processors are input as an array in the parameter file block that is read by McEnergyPerturbation::readParam(), listed in order of the MPI ranks of corresponding processors. The contents of this entire array is broadcast to all processors, and each processor then resets its temperature to a value obtained from the appropriate elements of the array (i.e., processor 4 resets its temperature to the value corresponding to element 4 in this array).
Simpatico provides a replica exchange algorithm, which can be used in multiprocessor MC simulations with any associated Perburbation. The replica exchange algorithm is implemented by the class McMd::ReplicaMove, which implements a Monte Carlo move that exchanges configurations between processors with neighboring MPI ranks. Please see the documentation of the "ReplicaMove" class for further information.
In the parameter file format for an MC simulation in perturbation mode, the block associated with the Perturbation must be followed by a line containing a boolean parameter "hasReplicaMove", which may take on values 1 (true) or 0 (false). This parameter is required only in multi-system replicated simulations. If "hasReplicaMove" is true (1), it must be followed by a parameter block associated with the ReplicaMove class. The ReplicaMove parameter file block a single "interval" parameter that specifies the interval (in MC steps) between subsequent attempted MC moves.
Show below is an example of a parameter file for a replicated mcSim simulation of a polymer blend, which is simulated on three processors. This example uses the McPairPerturbation subclass of Perturbation to define a sequence of systems with different values of the epsilon parameter for interactions between A and B atoms, and uses a replica exchange move. The parameter block associated with the McPairPerturbation and ReplicaMove appear at the end of the McSystem block.
Is is also possible to restart multi-processor mcSim and mdSim simulations. During the original simulation, the frequency with which restart files should be written and the base name for these files is specified in the parameter file, exactly as for single-system simulations. In multi-system simulations, each processor writes a separate restart file, for both independent and replicate simulations. The restart file for each processor is placed in the associated numbered directory. For example, if the saveFileName parameter in the original parameter file is "restart", the restart file for processor 2 would be repeatedly written to the file 2/restart.rst.
Different conventions are used for the placement of restart command files for "independent" and "replicated" simulations. Independent simulations use a separate restart command file for each processor, each of which is placed in the same numbered directory as the associated restart file. Replicated simulations use a single restart command file, which must be placed in the same directory as the single restart file, and the single parameter file and command file for the original simulation. In either case, the base name of the restart command file is passed as command line parameter of the -c option.
The command to restart a multi-processor simulation is similar to that for a single-processor simulation. The command to restart a 3 processor mcSim simulation of either independent or replicated systems, using the mcSim_m executable and the mpirun command to launch an MPI program, using a restart file named "restart" and a command file named "commands.rst", is:
Information about whether the original simulations were independent or replicated is stored within all restart files, and does not need to be be specified by a command line option when the simulation is restarted. Invoking an mcSim or mdSim executable with both the "-r" and "-f" options is thus illegal, and will cause execution to halt with an error message.
2.11 Multi-System Simulations (Up) 2.11.2 Multi-system simulations: ddSim (Next)
1.8.11