GENIE experiments - Revision history

Genie-user: /* Howto */

2010-03-23T09:30:53Z

Howto

Genie-user at 09:20, 23 March 2010

2010-03-23T09:20:15Z

Genie-user: /* build_experiment.sh */

2009-06-09T22:43:20Z

build_experiment.sh

Genie-user: /* File lists */

2009-06-09T22:42:32Z

File lists

Genie-user at 22:39, 9 June 2009

2009-06-09T22:39:29Z

Genie-user: /* You will need */

2009-06-09T21:19:22Z

You will need

Genie-user: /* creating and configuring GENIE experiments */

2009-06-09T20:55:36Z

creating and configuring GENIE experiments

Genie-user: /* build_experiment.sh */

2009-06-09T20:52:42Z

build_experiment.sh

Genie-user: /* Summary */

2009-06-09T20:52:20Z

Summary

Genie-user: /* Summary */

2009-06-09T20:50:28Z

Summary

@@ Line 79: / Line 79: @@
 # a text editor
-You will also need to make sure the OUTROOT environment variable is set correctly in user.sh:
+You will also need to make sure the OUTROOT and RUNTIME_ROOT environment variables are set correctly in user.sh:
 <pre>
-CODEDIR=/path/to/genie    // ~/genie by default
+CODEDIR             =/path/to/genie    // ~/genie by default
-OUTROOT=${CODEDIR}/genie-experiments
+OUTROOT             =${CODEDIR}/genie-experiments
 </pre>
-and OUT_DIR in user.mak:
+and also OUT_DIR and RUNTIME_ROOT in user.mak:
 <pre>
 GENIE_ROOT          = /path/to/genie
 OUT_DIR             = $(GENIE_ROOT)/genie-experiments
 </pre>

← Older revision		Revision as of 09:20, 23 March 2010
Line 1:		Line 1:
−	'''N.B.''' the genie-experiments approach to configuring and running GENIE is currently only on the ENGAGE branch of the genie repository, and is probably of most use at the moment to people building portable experiments for the ALADDIN Launchpad. We will be merging these new features into the trunk in the near future. If you're getting tired waiting for this to happen, please email martin.johnson@uea.ac.uk and give me a hard time.
−
	This page introduces a new way to configure and run traceable (i.e. reproduceable) GENIE experiments. We hope that this new method of approaching the organisation of GENIE files will make more sense to new users too.		This page introduces a new way to configure and run traceable (i.e. reproduceable) GENIE experiments. We hope that this new method of approaching the organisation of GENIE files will make more sense to new users too.

← Older revision		Revision as of 22:43, 9 June 2009
Line 179:		Line 179:
	note that the -x command-line argument to genie_example.job supresses the execution of the model i.e. compilation and file copying to experiment directory only.		note that the -x command-line argument to genie_example.job supresses the execution of the model i.e. compilation and file copying to experiment directory only.

−	Your GENIE experiment can now be moved to anywhere and run from there (by ./genie_wrapper.sh for logged output or ./genie.exe for command-line output) or used with ALADDIN.	+	Your GENIE experiment can now be moved to anywhere and run from there (by ./genie_wrapper.sh for logged output or ./genie.exe for command-line output) or used with ALADDIN, independent from the genie source tree. Yay!

← Older revision		Revision as of 22:42, 9 June 2009
Line 104:		Line 104:

	These filelists must contain soleley a list of files required and all of the files must be situated within the genie source tree, preferably in their default locations, so that minimum changes are required to experiment config file to point GENIE to the correct place in the experiment directory when these files have been copied across by build_experiment.sh.		These filelists must contain soleley a list of files required and all of the files must be situated within the genie source tree, preferably in their default locations, so that minimum changes are required to experiment config file to point GENIE to the correct place in the experiment directory when these files have been copied across by build_experiment.sh.
		+
		+	e.g. a filelist for a standard 8 level eb_go_gs_el run might look like this:
		+
		+
		+	<pre>
		+	genie-embm/data/input/worbe2.k1
		+	genie-embm/data/input/taux_u.interp
		+	genie-embm/data/input/taux_v.interp
		+	genie-embm/data/input/tauy_u.interp
		+	genie-embm/data/input/tauy_v.interp
		+	genie-embm/data/input/uncep.silo
		+	genie-embm/data/input/vncep.silo
		+	genie-embm/data/input/ta_ncep.silo
		+	genie-embm/data/input/qa_ncep.silo
		+	genie-goldstein/data/input/worbe2.k1
		+	genie-goldstein/data/input/worbe2.psiles
		+	genie-goldstein/data/input/worbe2.paths
		+	genie-goldstein/data/input/tempann.silo
		+	genie-goldstein/data/input/saliann.silo
		+	genie-goldsteinseaice/data/input/worbe2.k1
		+	genie-ents/data/k_constants.dat
		+	genie-ents/config/ents_config.par
		+	genie-ents/data/atm_albedo_monthly.dat
		+	genie-ents/data/uvic_windx.silo
		+	genie-ents/data/uvic_windy.silo
		+	genie-ents/data/monthly_windspd.silo
		+	genie-ents/data/inv_linterp_matrix.dat
		+	genie-ents/data/orography.dat
		+	genie-ents/data/icemask.dat
		+	genie-ents/config/sealevel_config.par
		+	</pre>

	==== input (and output) directory information in my_experiment.xml ====		==== input (and output) directory information in my_experiment.xml ====

@@ Line 13: / Line 13: @@
 There's nothing at all insensible about most of this - having the compiled model executable and namelist parameters along with the model output is good for traceability (in theory, if nothing in the source tree has been changed since you ran the experiment, you should be able to navigate to genie_output/my_experiment and launch genie.exe, and exactly reproduce your previous results. In practice, it's rarely that simple and there are some things we can do to improve traceability; but the main problem with the above is the misleading name of the genie_output folder and the difficulty of having it outside the source tree. Therefore we have made a new directory in the genie tree, called genie-experiments, from which experiments can be configured, built and run with relative ease and better traceability.
-== genie-experiments ==
+== GENIE experiments ==
 The motivation behind the changes outlined here is to i) make GENIE easier to configure and run for new users by making it more obvious what's what and ii) to facilitate the creation of full traceable and portable (i.e. source-tree-independent) compiled experiments, which can be run manually (i.e. by executing genie.exe) or through a front end (e.g. the [[GENIE_ALADDIN2|ALADDIN launchpad]]). Thus we have come up with the following specification for the structure and content of a GENIE experiment directory, which is portable and traceable. It is as follows
@@ Line 71: / Line 71: @@
 The optional mapVarData and graphVarData files are specific to ALADDIN, where they are used to interpret ascii output files to extract parameters to be visualised in ALADDIN during the experiment. See [[GENIE_ALADDIN2_varData_files]]
-=== You will need ===
+=== Howto ===
 # a working version of the genie source code and its prerequisites
@@ Line 79: / Line 81: @@
 # a text editor
-You will need to make sure the OUTROOT environment variable is set correctly in user.sh (note that user.sh notes tell us that RUNTIME_OUTDIR should always be "." - this will be useful later).
+You will also need to make sure the OUTROOT environment variable is set correctly in user.sh:
 <pre>
@@ Line 86: / Line 88: @@
 </pre>
-and OUT_DIR in user.mak
+and OUT_DIR in user.mak:
 <pre>
@@ Line 103: / Line 105: @@
 These filelists must contain soleley a list of files required and all of the files must be situated within the genie source tree, preferably in their default locations, so that minimum changes are required to experiment config file to point GENIE to the correct place in the experiment directory when these files have been copied across by build_experiment.sh.
-==== input directory information in my_experiment.xml ====
+==== input (and output) directory information in my_experiment.xml ====
-A traditional genie xml experiment configuration will point to data input files using the
+Note that the output directory for EMBM is already set correctly to meet with the specification of a GENIE experiment. However, it could be simplified by replacing '<varref>RUNTIME_OUTDIR</varref>' with '.' as genie.exe is executed from within the experiment directory.
 === build_experiment.sh ===
-The build_experiment.sh script takes files in the file list e.g. genie-goldstein/data/input/worap2.psiles. build_experiment.sh
+The build_experiment.sh script does the following:

@@ Line 92: / Line 92: @@
 OUT_DIR             = $(GENIE_ROOT)/genie-experiments
 </pre>
 Now, because we are creating a standalone experiment, we need to make sure that (i) all the necessary input files (topography, restart files, tracer initialisations, forcings etc) are in the correct places within the genie experiment structure (see specification) and (ii) that your xml config file is pointing GENIE to the correct place to look for them.
-The first part of this is non-trivial - as well as knowing how many files of what type are required for a particular run/ setup, you also need to know which of a selection of files are most appropriate. This isn't obvious, or documented anywhere at the moment, but I hope in the future [[GENIE_input_files]] might contain what you need to know. For now, a good starting point is to look around in the genie/genie-<module-name>/data/input directories and see what you can see. To make life easier, there are a number of file lists provided for 'standard' model setups in genie-experiments/files. If you need to make a new one for a particular config / model setup, we'd be grateful if you'd add a filelist with a self-explanatory filename into the files directory and document it in [[GENIE_input_files]] if appropriate.
+The first part of this is non-trivial - as well as knowing how many files of what type are required for a particular run/ setup, you also need to know which of a selection of files are most appropriate. This isn't obvious, or documented anywhere at the moment, but I hope in the future [[GENIE_input_files]] might contain what you need to know. For now, a good starting point is to look around in the genie/genie-<module-name>/data/input directories and see what you can see. Also trial and error works OK - GENIE is pretty good at telling you when it's missing a file and the correct files can be diagnosed from this.
-These filelists are key to constructing a standalone experiment and are passed to build_experiment.sh along with the experiment config file.
+A traditional genie xml experiment configuration will point to data input files using the
 === build_experiment.sh ===
 The build_experiment.sh script takes files in the file list e.g. genie-goldstein/data/input/worap2.psiles. build_experiment.sh

@@ Line 56: / Line 56: @@
 <code>*</code> the commands svn info and svn diff together in theory allow the recreation of any particular state of the genie source tree, whether a committed version or one containing user-changed code which wasn't in the repository at the time of doing the experiment. In practice, it would be silly to take GENIE into battle without using a fully-committed-to-svn version of the model, otherwise getting back to the same state in the future could be extremely difficult and messy even with the help of svn info and svn diff. If you're working with the GENIE trunk and not doing your own development work, it would be advisable to use a tuned tagged release of the model, which is set in stone and can always be reproduced.
-== creating and configuring GENIE experiments ==
+== Creating and configuring GENIE experiments ==
 It is assumed that you're adopting our proposed new genie/genie-experiments directory as the location for creating your compiled experiments (if not, you should be able to work out what needs changing in the details below, but don't come running to us when it all goes wrong...).

← Older revision		Revision as of 20:52, 9 June 2009
Line 101:		Line 101:
	=== build_experiment.sh ===		=== build_experiment.sh ===

−	~~<pre>~~	+	The build_experiment.sh script takes files in the file list e.g. genie-goldstein/data/input/worap2.psiles. build_experiment.sh
−
−	~~</pre>~~
−
−	The build_experiment.sh script takes files in the file list e.g. genie-goldstein/data/input/worap2.psiles. build_experiment.sh

@@ Line 69: / Line 69: @@
 The optional -n command-line argument will create an experiment without recompiling the model.
-The optional mapVarData and graphVarData files are specific to ALADDIN, where they are used to interpret ascii output files to extract parameters to be visualised in ALADDIN during the experiment. See [[GENIE_ALADDIN_varData_files]]
+The optional mapVarData and graphVarData files are specific to ALADDIN, where they are used to interpret ascii output files to extract parameters to be visualised in ALADDIN during the experiment. See [[GENIE_ALADDIN2_varData_files]]
 === You will need ===

@@ Line 65: / Line 65: @@
 <pre>
-./build_experiment -f files/<file list> -c configs/<config-file>
+./build_experiment -f files/<file list> -c configs/<config file> (-m <mavVarData file> -g<graphVarData file> -n)
 </pre>
-'''You will need'''
+=== You will need ===
 # a working version of the genie source code and its prerequisites
@@ Line 95: / Line 97: @@
 The first part of this is non-trivial - as well as knowing how many files of what type are required for a particular run/ setup, you also need to know which of a selection of files are most appropriate. This isn't obvious, or documented anywhere at the moment, but I hope in the future [[GENIE_input_files]] might contain what you need to know. For now, a good starting point is to look around in the genie/genie-<module-name>/data/input directories and see what you can see. To make life easier, there are a number of file lists provided for 'standard' model setups in genie-experiments/files. If you need to make a new one for a particular config / model setup, we'd be grateful if you'd add a filelist with a self-explanatory filename into the files directory and document it in [[GENIE_input_files]] if appropriate.
-These filelists are key to constructing a standalone experiment. The
+These filelists are key to constructing a standalone experiment and are passed to build_experiment.sh along with the experiment config file.