CHOMBO

CHOMBO

Installation

This page describes what I (Gethin) did to install and run CHOMBO. In the first instance, I worked on dylan, an x86_64 machine (paths begin with /gsb6/ggdagw). I have also followed these instructions to install on dartagnan as root, which has the same x86_64 architecture (paths begin with /opt).

First we will look at the prerquisites, an MPI implementation and HDF5. Then CHOMBO itself.

MPICH

The CHOMBO documentation calls for a MPI 1.2 compliant C binding (when running CHOMBO in parallel). Therefore I started with MPICHv1.2.6.

MPICH v1.2.6

There are problems getting gfortran to work with both MPICH v1.2.5 and v1.2.6, so I will omit it here. Beware, however of picking up e.g. the Sun fortran compiler as f77 and f90 and thus getting a mixed compiler MPICH build.

First, get tar ball:

wget ftp://ftp.mcs.anl.gov/pub/mpi/old/mpich-1.2.6.tar.gz

Treating dylan as an SMP machine (multi-core with shared memory), we will build a shared memory MPI installation (the -with-device=ch_shmem bit):

CC=gcc FC=ifort F90=ifort CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" \
./configure --prefix=/opt/mpich/mpich-1.2.6/ch_shmem --with-device=ch_shmem -c++=g++ >& config.out
make
make testing
make install

The cflags enable files > 2GB.

NB Fortran compiler:

• Using gfortran the Fortran bindings will not be built. MPICH complains that it can't access the command line through gfortran.
• Using ifort, things are OK.

NB C/C++ compiler:

• I needed to link mpicxx to mpiCC in the install directory, for some reason..

MPICH2v1.0.8

I've also built MPICH2v1.0.8, but not tried it with CHOMBO. I used:

wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.0.8/mpich2-1.0.8.tar.gz

./configure --prefix=/gsb6/ggdagw/mpich2/mpich2-1.0.8/intel-fc-10.1/ch3shm --with-device=ch3:shm \
FC=ifort F90=ifort CC=gcc CXX=g++

Note that you will need to start a daemon called mpd to use MPICH2. See the MPICH2 guide for details on how this is done. Fear not, however. It's quite easy.

HDF5

The CHOMBO documentation calls for HDF5v1.4.4. You will see from the below that this is a pain to build. CHOMBO also works with HDF5v1.6.7, which is much easier to build.

HDF5 v1.4.4

Get from:

wget ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases/hdf5-1.4.4.tar.gz

For Serial Build

 
./configure --prefix=/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial --enable-production

For Parallel Build

Tested MPI-IO with the program Sample_mpio.c from PHDF5 instructions (appendix) in conjunction with shmem MPI build and tests passed OK.

Straight from the box, we would have a broken build, due to problems with mpicc and DL_OPEN support. So, we will make a cheeky edit. In the file config/commence.in, change the line that defines the LT_LINK_EXE variable to remove the -dlopen self -rpath $(libdir) options.

Note that we've disabled linking against shared libraries. We could have gone the other way and added '-fPIC to the build flags.

Now we configure and build:

CC=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpicc ./configure --disable-shared \
--prefix=/gsb6/ggdagw/hdf5/hdf5-1.4.4/parallel
make
make check
make install

Note the H5dump tests don't pass, but all the PHDF5 ones do, so we'll run with that for the moment.

HDF5 v1.6.7

This works with CHOMBO too.

wget ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases/hdf5-1.6.7/src/hdf5-1.6.7.tar.gz

For a parallel build, for example:

CC=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpicc ./configure --disable-shared \
--prefix=/gsb6/ggdagw/hdf5/hdf5-1.6.7/parallel
make
make check
make install

Note the H5dump tests don't pass, but all the PHDF5 ones do, so we'll run with that for the moment.

CHOMBO

The Oct07 release is installed in /opt/chombo/Chombo-2.0. The Apr09 release is installed in /opt/chombo/Chombo-3.0

Without HDF5

First:

cd lib/mk
cp Make.defs.local.template Make.defs.local

Then edit to match:

DIM           = 2
DEBUG         = TRUE
OPT           = TRUE
PRECISION     = DOUBLE
PROFILE       = FALSE
CXX           = g++
FC            = gfortran
MPI           = FALSE
USE_HDF       = FALSE

and to test:

make lib
make test
make run

Looks like the tests pass BUT uses -m32 during compilation and so a 32bit executable results. This is fine here (since x86_64 will also handle 32bit exes), but it causes a problem when we try and link against 64bit dependencies, such as HDF5 (see below for remedy).

With HDF5 compiled in serial mode

USE_64 = TRUE

adds -m64 to the GNU compiler flags and so object code is created for the x86_64 architecture, and so it will link against the HDF libraries which are built for x86_64. This is not what the documentation says the makefile variable is for! Hey ho. Add it in.

• e.g. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial/lib

required to find shared object files, else tests will not run. Tests look OK for both v1.4.4 and v1.6.7 of HDF.

With MPI and parallel HDF5

Add in to Make.defs.local:

MPICXX        = /opt/mpich/mpich-1.2.6/ch_shmem/bin/mpiCC
HDFMPIINCFLAGS= -I/opt/hdf5/hdf5-1.6.7/parallel/include
HDFMPILIBFLAGS= -L/opt/hdf5/hdf5-1.6.7/parallel/lib -lhdf5 -lz

For Centos5.2 I also needed to add, in order to make test to go:

lddbgflags    = -ldl

cd chombo/lib
make lib
make test
make run

Tests look to pass OK for both HDF5v1.4.4 and v1.6.7.

Running the AMRUpwind example

• Ensure that the appropriate version of mpiCC is in your path. For example, you might add:

export PATH=$PATH:/opt/mpich/mpich-1.2.6/ch_shmem/bin

to your ~/.bashrc file. (Don't forget to source it, or login again.)

• cd to where you want to work and copy the code for the examples to your working directory:

cp -r /opt/chombo/Chombo-2.0/example .

• We're going to compile and run AMRUpwind:

cd AMRUpwind

• Edit two makefiles, such that they contain the correct paths which point to the installation of CHOMBO under /opt. In ./GNUmakefile, e.g. (if you have copied the examples to your home dir):

CHOMBO_HOME = ../../../../../opt/chombo/Chombo-3.0/lib

• In exec/GNUmakefile, e.g.:

CHOMBO_HOME = ../../../../../../opt/chombo/Chombo-3.0/lib

• make clean
• make example
• make run

Useful informatiomn on files within example

After compilation, you will find 4 directories created within the example directory. Each contains subdirectories based on the particular build (e.g., 2d.Linux.64.mpiCC.gfortran.DEBUG.OPT.MPI/). These are:

• d which contains paths for any files used in the compilation; most usefully .H etc files.
• f which contains fortan code after it has been parsed; particularly useful in the case of ChF files (.f).
• o which contains usual .o objects.
• p which also contains parsed versions of any fortran code (.cpre is same as .f above).

Changes to Makefiles for Fortran90

In Chombo-3.0/lib/mk/Make.example:

src_extensions := .cpp .ChF .F .f .c .f90
vpath %.cpp $(src_dirs)
vpath %.ChF $(src_dirs)
vpath %.F   $(src_dirs)
vpath %.f   $(src_dirs)
vpath %.c   $(src_dirs)
vpath %.f90 $(src_dirs)

In Chombo-3.0/lib/mk/Make.rules:

line 104:

src_wildcard := *.cpp *.ChF *.F *.f *.f90

add after line ~350:

o/$(config)/%.o : %.f90  d/$(config)/%.d
        $(QUIET)$(FC) $(FFLAGS) $(XTRAFFLAGS) $(fcompflag) $< $(fobjflag)$@

add after line ~413:

d/$(config)/%.d : %.f90 d/$(config)/.dummy
        @if [ x"$(QUIET)" != x"@" ]; then echo Depending $< ...; fi

In Chombo-3.0/lib/mk/compiler/Make.defs.Intel (when using ifort), line 184:

deffcomflags = -w95 -u

i.e. remove -FI, as it causes trouble with f90 (and is an overspec here?)

Tested example/expHeat/parVersion by replacing heatfort.F with heatfort.f90.