Difference between revisions of "CHOMBO"

From SourceWiki
Jump to navigation Jump to search
(added section on contents of d/f/o/p directories)
 
(28 intermediate revisions by the same user not shown)
Line 5: Line 5:
 
=Installation=
 
=Installation=
  
This page describes what I (Gethin) did to install and run CHOMBO.  In the first instance, I worked on dyln (an x86_64 machine) and used the GNU C and Fortran compilers (gcc, g++ and gfortran).
+
This page describes what I (Gethin) did to install and run CHOMBO.  In the first instance, I worked on dylan, an x86_64 machine (paths begin with /gsb6/ggdagw).  I have also followed these instructions to install on dartagnan as root, which has the same x86_64 architecture (paths begin with /opt).
  
First we will look at the prerquisites, an MPI implementation and HDF5, and then CHOMBO itself.
+
First we will look at the prerquisites, an MPI implementation and HDF5.  Then CHOMBO itself.
 +
 
 +
=MPICH=
 +
 
 +
The CHOMBO documentation calls for a MPI 1.2 compliant C binding (when running CHOMBO in parallel).  Therefore I started with MPICHv1.2.6.
  
 
==MPICH v1.2.6==
 
==MPICH v1.2.6==
  
Only an MPI 1.2 compliant C binding is required to run CHOMBO in parallel.  There are problems getting gfortran to work with both MPICH v1.2.5 and v1.2.6, so I will omit it here.  '''Beware''', however of picking up e.g. the Sun fortran compiler as '''f77''' and '''f90''' and thus getting a mixed compiler MPICH build.
+
There are problems getting gfortran to work with both MPICH v1.2.5 and v1.2.6, so I will omit it here.  '''Beware''', however of picking up e.g. the Sun fortran compiler as '''f77''' and '''f90''' and thus getting a mixed compiler MPICH build.
 +
 
 +
First, get tar ball:
  
Get tar ball:
 
 
<pre>
 
<pre>
 
wget ftp://ftp.mcs.anl.gov/pub/mpi/old/mpich-1.2.6.tar.gz
 
wget ftp://ftp.mcs.anl.gov/pub/mpi/old/mpich-1.2.6.tar.gz
 
</pre>
 
</pre>
  
Treating dylan as an SMP machine (multi-core with shared memory), we will build a shared memory MPI installation:
+
Treating dylan as an SMP machine (multi-core with shared memory), we will build a shared memory MPI installation (the -with-device=ch_shmem bit):
  
 
<pre>
 
<pre>
./configure --prefix=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem --with-device=ch_shmem -cc=gcc -c++=g++ \
+
CC=gcc FC=ifort F90=ifort CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" \
-fc=gfortran -f90=gfortran -cflags="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" \
+
./configure --prefix=/opt/mpich/mpich-1.2.6/ch_shmem --with-device=ch_shmem -c++=g++ >& config.out
>& config.out
 
 
make
 
make
 
make testing
 
make testing
Line 31: Line 35:
 
The cflags enable files > 2GB.
 
The cflags enable files > 2GB.
  
'''Note''' that Fortran binding will not be built, as MPICH will complain that it can't access the command line through gfortran. '''Also''' that I needed to copt '''mpicxx''' to '''mpiCC''' in the install directory, for some reason..
+
'''NB Fortran compiler:'''
 +
* '''Using gfortran''' the Fortran bindings will not be builtMPICH complains that it can't access the command line through gfortran.
 +
* '''Using ifort''', things are OK.
 +
 
 +
'''NB C/C++ compiler:'''
 +
* I needed to link '''mpicxx''' to '''mpiCC''' in the install directory, for some reason..
 +
 
 +
==MPICH2v1.0.8==
 +
 
 +
I've also built MPICH2v1.0.8, but not tried it with CHOMBO.  I used:
 +
 
 +
<pre>
 +
wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.0.8/mpich2-1.0.8.tar.gz
 +
</pre>
 +
 
 +
<pre>
 +
./configure --prefix=/gsb6/ggdagw/mpich2/mpich2-1.0.8/intel-fc-10.1/ch3shm --with-device=ch3:shm \
 +
FC=ifort F90=ifort CC=gcc CXX=g++
 +
</pre>
 +
 
 +
Note that you will need to start a daemon called '''mpd''' to use MPICH2.  See [http://www.mcs.anl.gov/research/projects/mpich2/documentation/files/mpich2-1.0.8-installguide.pdf the MPICH2 guide] for details on how this is done.  Fear not, however.  It's quite easy.
 +
 
 +
=HDF5=
 +
 
 +
The CHOMBO documentation calls for HDF5v1.4.4.  You will see from the below that this is a pain to build.  CHOMBO also works with HDF5v1.6.7, which is much easier to build.
  
 
==HDF5 v1.4.4==
 
==HDF5 v1.4.4==
Line 38: Line 66:
  
 
<pre>
 
<pre>
ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases
+
wget ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases/hdf5-1.4.4.tar.gz
 
</pre>
 
</pre>
  
Line 64: Line 92:
 
make install
 
make install
 
</pre>
 
</pre>
 +
 +
'''Note''' the H5dump tests don't pass, but all the PHDF5 ones do, so we'll run with that for the moment.
  
 
==HDF5 v1.6.7==
 
==HDF5 v1.6.7==
  
We could use this too. For a parallel build, for example:
+
This works with CHOMBO too.
 +
 
 +
<pre>
 +
wget ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases/hdf5-1.6.7/src/hdf5-1.6.7.tar.gz
 +
</pre>
 +
 
 +
For a parallel build, for example:
  
 
<pre>
 
<pre>
Line 77: Line 113:
 
</pre>
 
</pre>
  
Seems to be OK.
+
'''Note''' the H5dump tests don't pass, but all the PHDF5 ones do, so we'll run with that for the moment.
 +
 
 +
=CHOMBO=
 +
<strike>The Oct07 release is installed in '''/opt/chombo/Chombo-2.0'''.</strike>
 +
The Apr09 release is installed in '''/opt/chombo/Chombo-3.0'''
 +
 
 +
==Without HDF5==
 +
 
 +
First:
 +
 
 +
<pre>
 +
cd lib/mk
 +
cp Make.defs.local.template Make.defs.local
 +
</pre>
  
==CHOMBO==
+
Then edit to match:
The Oct07 release is installed in '''/gsb6/ggdagw/chombo/Chombo-2.0'''.
 
  
===Without HDF5===
 
lib/mk/Make.defs.local:
 
 
<pre>
 
<pre>
 
DIM          = 2
 
DIM          = 2
Line 96: Line 142:
 
</pre>
 
</pre>
  
 +
and to test:
 +
 +
<pre>
 
make lib
 
make lib
 
make test
 
make test
 
make run
 
make run
 +
</pre>
  
 
Looks like the tests pass '''BUT uses -m32''' during compilation and so a 32bit executable results.  This is fine here (since x86_64 will also handle 32bit exes), but it causes a problem when we try and link against 64bit dependencies, such as HDF5 (see below for remedy).
 
Looks like the tests pass '''BUT uses -m32''' during compilation and so a 32bit executable results.  This is fine here (since x86_64 will also handle 32bit exes), but it causes a problem when we try and link against 64bit dependencies, such as HDF5 (see below for remedy).
  
===With HDF5 compiled in serial mode===
+
==With HDF5 compiled in serial mode==
  
 
<pre>
 
<pre>
Line 108: Line 158:
 
</pre>
 
</pre>
  
adds -m64 to the GNU compiler flags and so object code is created for the x86_64 architecture, and so it will link against the HDF libraries which are built for x86_64.  This is not what the documentation says the makefile variable is for!  Hey ho.  Add it in.
+
adds '''-m64''' to the GNU compiler flags and so object code is created for the x86_64 architecture, and so it will link against the HDF libraries which are built for x86_64.  This is not what the documentation says the makefile variable is for!  Hey ho.  Add it in.
  
 
* e.g. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial/lib
 
* e.g. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial/lib
Line 114: Line 164:
 
required to find shared object files, else tests will not run.  Tests look OK for both v1.4.4 and v1.6.7 of HDF.
 
required to find shared object files, else tests will not run.  Tests look OK for both v1.4.4 and v1.6.7 of HDF.
  
===With MPI and parallel HDF5===
+
==With MPI and parallel HDF5==
  
 
Add in to Make.defs.local:
 
Add in to Make.defs.local:
  
 
<pre>
 
<pre>
MPICXX        = /gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpiCC
+
MPICXX        = /opt/mpich/mpich-1.2.6/ch_shmem/bin/mpiCC
HDFMPIINCFLAGS= -I/gsb6/ggdagw/hdf5/hdf5-1.4.4/parallel/include
+
HDFMPIINCFLAGS= -I/opt/hdf5/hdf5-1.6.7/parallel/include
HDFMPILIBFLAGS= -L/gsb6/ggdagw/hdf5/hdf5-1.4.4/parallel/lib -lhdf5 -lz
+
HDFMPILIBFLAGS= -L/opt/hdf5/hdf5-1.6.7/parallel/lib -lhdf5 -lz
 +
</pre>
 +
 
 +
For Centos5.2 I also needed to add, in order to '''make test''' to go:
 +
 
 +
<pre>
 +
lddbgflags    = -ldl
 
</pre>
 
</pre>
  
Line 133: Line 189:
 
Tests look to pass OK for both HDF5v1.4.4 and v1.6.7.
 
Tests look to pass OK for both HDF5v1.4.4 and v1.6.7.
  
===Running the AMRUpwind example from Gethin's build===
+
==Running the AMRUpwind example==
 +
 
 +
* Ensure that the appropriate version of '''mpiCC''' is in your path.  For example, you might add:
 +
 
 +
<pre>
 +
export PATH=$PATH:/opt/mpich/mpich-1.2.6/ch_shmem/bin
 +
</pre>
 +
 
 +
to your ~/.bashrc file.  (Don't forget to source it, or login again.)
 +
 
 +
* cd to where you want to work and copy the code for the examples to your working directory:
 +
 
 +
<pre>
 +
cp -r /opt/chombo/Chombo-2.0/example .
 +
</pre>
 +
 
 +
* We're going to compile and run '''AMRUpwind''':
 +
 
 +
<pre>
 +
cd AMRUpwind
 +
</pre>
 +
 
 +
* Edit two makefiles, such that they contain the correct paths which point to the installation of CHOMBO under /opt.  In '''./GNUmakefile''', e.g. (if you have copied the examples to your home dir):
 +
 
 +
<pre>
 +
CHOMBO_HOME = ../../../../../opt/chombo/Chombo-3.0/lib
 +
</pre>
 +
 
 +
* In '''exec/GNUmakefile''', e.g.:
 +
 
 +
<pre>
 +
CHOMBO_HOME = ../../../../../../opt/chombo/Chombo-3.0/lib
 +
</pre>
  
* Ensure that '''/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin''' is in your path
 
* copy '''~ggdagw/chombo/Chombo-2.0/example''' to your location of choice
 
* cd AMRUpwind
 
* edit:
 
** '''GNUmakefile''' such that: '''CHOMBO_HOME = ../../chombo/Chombo-2.0/lib'''
 
** '''exec/GNUmakefile''' such that: '''CHOMBO_HOME = ../../../chombo/Chombo-2.0/lib'''
 
 
* make clean
 
* make clean
 
* make example
 
* make example
 
* make run
 
* make run
  
===Useful informatiomn on files within example===
+
==Useful informatiomn on files within example==
  
 
After compilation, you will find 4 directories created within the example directory. Each contains subdirectories based on the particular build (e.g., 2d.Linux.64.mpiCC.gfortran.DEBUG.OPT.MPI/).  These are:
 
After compilation, you will find 4 directories created within the example directory. Each contains subdirectories based on the particular build (e.g., 2d.Linux.64.mpiCC.gfortran.DEBUG.OPT.MPI/).  These are:
Line 154: Line 236:
 
* '''p''' which also contains parsed versions of any fortran code (.cpre is same as .f above).
 
* '''p''' which also contains parsed versions of any fortran code (.cpre is same as .f above).
  
===Changes to Makefiles for Fortran90===
+
==Changes to Makefiles for Fortran90==
  
In '''Chombo-2.0/lib/mk/Make.example''':
+
In '''Chombo-3.0/lib/mk/Make.example''':
  
 
<pre>
 
<pre>
Line 168: Line 250:
 
</pre>
 
</pre>
  
In '''Chombo-2.0/lib/mk/Make.rules''':
+
In '''Chombo-3.0/lib/mk/Make.rules''':
  
 
line 104:
 
line 104:
Line 186: Line 268:
 
         @if [ x"$(QUIET)" != x"@" ]; then echo Depending $< ...; fi
 
         @if [ x"$(QUIET)" != x"@" ]; then echo Depending $< ...; fi
 
</pre>
 
</pre>
 +
 +
In Chombo-3.0/lib/mk/compiler/Make.defs.Intel (when using '''ifort'''), line 184:
 +
 +
<pre>
 +
deffcomflags = -w95 -u
 +
</pre>
 +
 +
i.e. remove -FI, as it causes trouble with f90 (and is an overspec here?)
  
 
Tested '''example/expHeat/parVersion''' by replacing '''heatfort.F''' with '''heatfort.f90'''.
 
Tested '''example/expHeat/parVersion''' by replacing '''heatfort.F''' with '''heatfort.f90'''.

Latest revision as of 12:15, 8 June 2009

CHOMBO

Introduction

Installation

This page describes what I (Gethin) did to install and run CHOMBO. In the first instance, I worked on dylan, an x86_64 machine (paths begin with /gsb6/ggdagw). I have also followed these instructions to install on dartagnan as root, which has the same x86_64 architecture (paths begin with /opt).

First we will look at the prerquisites, an MPI implementation and HDF5. Then CHOMBO itself.

MPICH

The CHOMBO documentation calls for a MPI 1.2 compliant C binding (when running CHOMBO in parallel). Therefore I started with MPICHv1.2.6.

MPICH v1.2.6

There are problems getting gfortran to work with both MPICH v1.2.5 and v1.2.6, so I will omit it here. Beware, however of picking up e.g. the Sun fortran compiler as f77 and f90 and thus getting a mixed compiler MPICH build.

First, get tar ball:

wget ftp://ftp.mcs.anl.gov/pub/mpi/old/mpich-1.2.6.tar.gz

Treating dylan as an SMP machine (multi-core with shared memory), we will build a shared memory MPI installation (the -with-device=ch_shmem bit):

CC=gcc FC=ifort F90=ifort CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" \
./configure --prefix=/opt/mpich/mpich-1.2.6/ch_shmem --with-device=ch_shmem -c++=g++ >& config.out
make
make testing
make install

The cflags enable files > 2GB.

NB Fortran compiler:

  • Using gfortran the Fortran bindings will not be built. MPICH complains that it can't access the command line through gfortran.
  • Using ifort, things are OK.

NB C/C++ compiler:

  • I needed to link mpicxx to mpiCC in the install directory, for some reason..

MPICH2v1.0.8

I've also built MPICH2v1.0.8, but not tried it with CHOMBO. I used:

wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.0.8/mpich2-1.0.8.tar.gz
./configure --prefix=/gsb6/ggdagw/mpich2/mpich2-1.0.8/intel-fc-10.1/ch3shm --with-device=ch3:shm \
FC=ifort F90=ifort CC=gcc CXX=g++

Note that you will need to start a daemon called mpd to use MPICH2. See the MPICH2 guide for details on how this is done. Fear not, however. It's quite easy.

HDF5

The CHOMBO documentation calls for HDF5v1.4.4. You will see from the below that this is a pain to build. CHOMBO also works with HDF5v1.6.7, which is much easier to build.

HDF5 v1.4.4

Get from:

wget ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases/hdf5-1.4.4.tar.gz

For Serial Build

 
./configure --prefix=/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial --enable-production

For Parallel Build

Tested MPI-IO with the program Sample_mpio.c from PHDF5 instructions (appendix) in conjunction with shmem MPI build and tests passed OK.

Straight from the box, we would have a broken build, due to problems with mpicc and DL_OPEN support. So, we will make a cheeky edit. In the file config/commence.in, change the line that defines the LT_LINK_EXE variable to remove the -dlopen self -rpath $(libdir) options.

Note that we've disabled linking against shared libraries. We could have gone the other way and added '-fPIC to the build flags.

Now we configure and build:

CC=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpicc ./configure --disable-shared \
--prefix=/gsb6/ggdagw/hdf5/hdf5-1.4.4/parallel
make
make check
make install

Note the H5dump tests don't pass, but all the PHDF5 ones do, so we'll run with that for the moment.

HDF5 v1.6.7

This works with CHOMBO too.

wget ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases/hdf5-1.6.7/src/hdf5-1.6.7.tar.gz

For a parallel build, for example:

CC=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpicc ./configure --disable-shared \
--prefix=/gsb6/ggdagw/hdf5/hdf5-1.6.7/parallel
make
make check
make install

Note the H5dump tests don't pass, but all the PHDF5 ones do, so we'll run with that for the moment.

CHOMBO

The Oct07 release is installed in /opt/chombo/Chombo-2.0. The Apr09 release is installed in /opt/chombo/Chombo-3.0

Without HDF5

First:

cd lib/mk
cp Make.defs.local.template Make.defs.local

Then edit to match:

DIM           = 2
DEBUG         = TRUE
OPT           = TRUE
PRECISION     = DOUBLE
PROFILE       = FALSE
CXX           = g++
FC            = gfortran
MPI           = FALSE
USE_HDF       = FALSE

and to test:

make lib
make test
make run

Looks like the tests pass BUT uses -m32 during compilation and so a 32bit executable results. This is fine here (since x86_64 will also handle 32bit exes), but it causes a problem when we try and link against 64bit dependencies, such as HDF5 (see below for remedy).

With HDF5 compiled in serial mode

USE_64 = TRUE

adds -m64 to the GNU compiler flags and so object code is created for the x86_64 architecture, and so it will link against the HDF libraries which are built for x86_64. This is not what the documentation says the makefile variable is for! Hey ho. Add it in.

  • e.g. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial/lib

required to find shared object files, else tests will not run. Tests look OK for both v1.4.4 and v1.6.7 of HDF.

With MPI and parallel HDF5

Add in to Make.defs.local:

MPICXX        = /opt/mpich/mpich-1.2.6/ch_shmem/bin/mpiCC
HDFMPIINCFLAGS= -I/opt/hdf5/hdf5-1.6.7/parallel/include
HDFMPILIBFLAGS= -L/opt/hdf5/hdf5-1.6.7/parallel/lib -lhdf5 -lz

For Centos5.2 I also needed to add, in order to make test to go:

lddbgflags    = -ldl
cd chombo/lib
make lib
make test
make run

Tests look to pass OK for both HDF5v1.4.4 and v1.6.7.

Running the AMRUpwind example

  • Ensure that the appropriate version of mpiCC is in your path. For example, you might add:
export PATH=$PATH:/opt/mpich/mpich-1.2.6/ch_shmem/bin

to your ~/.bashrc file. (Don't forget to source it, or login again.)

  • cd to where you want to work and copy the code for the examples to your working directory:
cp -r /opt/chombo/Chombo-2.0/example .
  • We're going to compile and run AMRUpwind:
cd AMRUpwind
  • Edit two makefiles, such that they contain the correct paths which point to the installation of CHOMBO under /opt. In ./GNUmakefile, e.g. (if you have copied the examples to your home dir):
CHOMBO_HOME = ../../../../../opt/chombo/Chombo-3.0/lib
  • In exec/GNUmakefile, e.g.:
CHOMBO_HOME = ../../../../../../opt/chombo/Chombo-3.0/lib
  • make clean
  • make example
  • make run

Useful informatiomn on files within example

After compilation, you will find 4 directories created within the example directory. Each contains subdirectories based on the particular build (e.g., 2d.Linux.64.mpiCC.gfortran.DEBUG.OPT.MPI/). These are:

  • d which contains paths for any files used in the compilation; most usefully .H etc files.
  • f which contains fortan code after it has been parsed; particularly useful in the case of ChF files (.f).
  • o which contains usual .o objects.
  • p which also contains parsed versions of any fortran code (.cpre is same as .f above).

Changes to Makefiles for Fortran90

In Chombo-3.0/lib/mk/Make.example:

src_extensions := .cpp .ChF .F .f .c .f90
vpath %.cpp $(src_dirs)
vpath %.ChF $(src_dirs)
vpath %.F   $(src_dirs)
vpath %.f   $(src_dirs)
vpath %.c   $(src_dirs)
vpath %.f90 $(src_dirs)

In Chombo-3.0/lib/mk/Make.rules:

line 104:

src_wildcard := *.cpp *.ChF *.F *.f *.f90

add after line ~350:

o/$(config)/%.o : %.f90  d/$(config)/%.d
        $(QUIET)$(FC) $(FFLAGS) $(XTRAFFLAGS) $(fcompflag) $< $(fobjflag)$@

add after line ~413:

d/$(config)/%.d : %.f90 d/$(config)/.dummy
        @if [ x"$(QUIET)" != x"@" ]; then echo Depending $< ...; fi

In Chombo-3.0/lib/mk/compiler/Make.defs.Intel (when using ifort), line 184:

deffcomflags = -w95 -u

i.e. remove -FI, as it causes trouble with f90 (and is an overspec here?)

Tested example/expHeat/parVersion by replacing heatfort.F with heatfort.f90.