Polyglot

Mixed Language Programming: Mix up a quick and useful cocktail today!

Introduction

One of the key underling themes in this series of pragmatic programming workshops is getting things done, with a minimum of fuss and wasted effort, and this workshop on mixed language programming is no exception.

When we sit down to a keyboard, we do so with a particular goal in mind. We have a task, idea or experiment and the computer is our workbench. The languages we write programs with, along with the text editors, compilers, debuggers etc. are our tools. Now some tools are better than others (e.g. Subversion is an improvement upon CVS). Some we just may have a preference for (e.g. emacs vs. vi for editing files). None are perfect, however, and all have their pros and cons.

What's all this got to do with mixed language programming? you ask. Well, imagine a scenario:

You sit down to your workbench. You have your goal and you ask yourself whether somebody else has written some code which will do at least part of what you want. Perhaps they've bundled it up into a nice open-source library that you can use? That way you'll save truck-loads of time. A couple of web searches later, and bingo! You've found an ideal library for the job. There's only one snag. You like programming in Fortran, and the library is written in C. Scuppered! Well, perhaps not..

What are your options? You could use something not so suitable because it happens to be written in Fortran. Hmm, that doesn't sound so good. You could translate the library from C to Fortran. Hmm, that sounds like tedious and time-consuming work. Plus you'd need to understand the C, and perhaps a direct translation can't be made anyhow? This is looking like a dead-end too. It would be far better to leave the library as it is and to call the routines from your favoured language. Is that possible? Sure it is! Read on and find out how..

Another scenario is that C or Fortran do number crunching well, but aren't best suited to text processing or creating GUIs. For scripting languages, like python for example, the reverse is true. Wouldn't it be nice if we could combine their strengths in a single project. Well, you can!

In this workshop, we'll look at ways in which we can mix languages and in the process create a useful end product which plays to the strengths of it's components and gets you to where you want to be, without any laborious re-writes. In the first two examples we'll look at calling C code from Fortran and then calling Fortran from C. To get the code for examples, log into your preferred Linux machine and cut and paste the following into your terminal.

svn export https://svn.ggy.bris.ac.uk/subversion-open/polyglot/trunk ./polyglot

Making a Fortran wrapped C parcel

In the first example we'll call a C function from a Fortran program:

cd examples/example1

To compile the example, type:

make

and to run it, type:

call_c.exe

Tada! We've mixed our languages into a single executable and it works! Cool. Very cool. OK, so much for the magic, let's take a look inside the files. Open up call_c.f90. In this simple program, we have a character string that we pass to the C function--called cfunc--which in turn modifies the string and we can print the result. For those familiar with Fotran90, the main program is trivial enough. So, let's turn to the C code in func.c. Again this is a simple function and the syntax will be familiar to those who know some C. An interesting detail, however, is the name of the function. Note that we have called it cfunc_, i.e. with a trailing underscore. Why on earth have we done that? Well, we are anticipating the 'decoration' or name mangling that the compiler will apply.

We can look inside the object files using a utility called nm. First let's look inside call_c.o. On my machine I get:

> nm call_c.o
00000000 T MAIN__
         U _gfortran_set_std
         U _gfortran_st_write
         U _gfortran_st_write_done
         U _gfortran_transfer_character
         U cfunc_

These are the symbols created by the compiler in the object file. Now let's look inside func.o:

>  nm func.o
00000000 T cfunc_

Note the two cfunc_s? They match! If we had not pre-decorated the name of the function in func.c, we would not have got a match and we would have got a link-time error. However, we were smart and got ourselves a working executable--hurrah!

A word of caution, however. Different compilers 'mangle' subroutine and function names differently, i.e. dependening upon the mix of compilers, we can't always rely on a single trailing underscore decoration. You will need to use nm to determine the exact decoration your Fortran compiler expects and to design your code so that any changes are easily accomodated. One way to do this is to include a define preprocessor macro in you C code, such as:

#define CFUNC cfunc_
...
void CFUNC(int* string_size, char* string)

In that way, a change of decoration can be easily, and consistently made across the whole file/library.

Turning the Parcel Inside-Out

OK let's turn our mind to working the other way 'round and call a Fortran subroutine from a C program:

cd ../example2

We can compile and run in a similar way:

make
./call_fort.exe

If you look inside the two key files, sub.f90 and call_fort.c, you can see that we've still kept the program fairly simple, but we've introduced a few more variable types (integers, floating-point numbers and arrays) and syntactic refinements (use of preprocessor macros and 'const' and 'intent(in)' declarations).

The decoration and name-mangling will be familiar to you from the last example. Note the use of the define macro in call_fort.c:

#define FORT_SUB fort_sub_

The code comments in the last example touched upon the distinction between the pass-by-value default behaviour of C and the pass-by-reference adopted by Fortran. Thus, in order to dovetail the two languages, we must prevail upon our C code to use pass-by-reference. We can see this in the function prototype in call_fort.c:

extern void FORT_SUB(const int*, const int*, float[]);

Here we are declaring that we will use pointers-to variables, i.e. addresseses or references, as the arguments to the routine, denoted by the *s. The routine, of course, is written in Fortran, and so is expecting these references. By default, C would have taken a copy of the value of a variable and passed that. This is where the term pass-by-value comes from. (Note that a side effect of pass-by-value is that any changes are only local to a routine and so do not have an effect on the value of the variable used in the calling routine.) Note the the const modifiers match with the intent(in) attributes in sub.f90. This is an example of defensive programming. Normally, when we pass references, we run the risk that called routine could modify the values of the variables in a lasting way. This may not be what you would like, and so it would be unsafe. However, by using const and intent(in), we are saying from both sides that some values are not to be changed and so we protect ourselves.

In general with mixed langauge programming, we need to be careful about mathing the variable types we pass between the languages. For example, C typically uses 4 bytes to store a float. However, Fortran may use 4 or 8 bytes to store a real, depending upon the type of processor in the machine (32 ior 64 bit). We need to be careful to ensure a match, otherwise we could have all sorts errors in store realated to truncation or uninitialsed memory space. Thus in sub.f90, we have limited our variables to 4 bytes:

  integer(kind=4),intent(in)                     :: arg_in
  integer(kind=4),intent(in)                     :: array_size
  real(kind=4),dimension(array_size),intent(out) :: arg_out

An intersting point of difference to note between Fortran and C is that arrays are indexed differently. Firstly, Fortran gives the first element of an array the index 1, by default. C gives it an index of 0. Compare the two loops in the source code files. First for the calling C program:

  /* initialise array to ones */
  for(ii=0;ii<MAXSIZE;++ii) {
    from_fort[ii] = 1.0;
  }

and second for the Fortran subroutine:

  ! just set output to be equal to input
  do ii=1,array_size
     arg_out(ii) = real(arg_in)
  end do

Things diverge further for multi-dimensional arrays. This is because arrays in C are 'row-major' and arrays in Fortran are 'column major' in their ordering. Thus for a defualt 2-dimensional array, the cell (3,7) in C would map to (8,4) in Fortran. Oh the joys of mixed languages! Thankfully, since program units are typically self-contained and we would pass arrays in their entirity (actually just the memory address of the first cell), this difference in behaviour does not cause many headaches.

Linker Strife

If you have compiled your main program using a Fortran compiler, your C code with a C compiler, and would like to link the whole lot using a C compiler, you will get undefined reference to "main" errors. One way around this is to create a C main which immediately calls the Fortran program which has the symbol MAIN__:

/*main.c */

void MAIN__();

int main(int argc, char* argv[])
{
    MAIN__();
}

Fortran 2003 Improves Matters

It's important that we take care to match the types of the variables that we're using when exchanging between C and Fortran. We did this explicitly above, which was useful for emphasis, but quite likely to prove rather unportable from machine to machine. On project which largely addressed this issue of portability is the cfortran header project: [1].

However, support for the Fortran2003 standard is growing among compilers (e.g for gfortran-4.3 and newer) and with it, we receive a good deal of aid built in to the actual language. For example, the intrinsic module named ISO_C_BINDING contains the following parameterisations upon instrinsic Fortran types:

Type      Named constant      C type or types
INTEGER   C_INT               int
          C_SHORT             short int
          C_LONG              long int
          C_LONG_LONG         long long int
          C_SIGNED_CHAR       signed char, unsigned char
          C_SIZE_T            size_t
 ...
REAL      C_FLOAT             float
          C_DOUBLE            double
          C_LONG_DOUBLE       long double.
COMPLEX   C_FLOAT_COMPLEX     float _Complex
          C_DOUBLE_COMPLEX    double _Complex
          C_LONG_DOUBLE_COMPLEX long double _Complex
LOGICAL   C_BOOL              _Bool
CHARACTER C_CHAR              char

Using some appropriately specified types, we can re-write example1 in a much more transparent and portable manner:

cd ../example3
make
./call_c.exe

Looking at call_c.f90 we can see:

  ! make use of this intrinsic module
  use ISO_C_BINDING

near the top of the file. We then see our variable declarations, which make use of the newly available parameterisations:

  integer(C_INT),parameter                :: array_size = 5  ! parameter gets init
  character(C_CHAR),dimension(array_size) :: str = 'fffff'   ! a Fortran string
  real(C_DOUBLE),dimension(array_size)    :: data = 1.1      ! Fortran array
  integer(C_INT),allocatable,dimension(:) :: data2           ! can init to null
...

We see no change with regard to the C code.

Interpreting Fortran derived types as C structures

In example3 we saw the portable exchange of intrinsic Fortran types. This is all well and good, but perhaps you have tens of variables which you would like to pass as arguments to C function. The above approach would become cumbersome. A smart move would be to bundle up related variables into their own user-derived types and then pass them through to C.

Mapping user-derived types from Fortran to structures in C and vice versa used to be a somewhat fraught affair. Matters have been significantly improved with the advent of Fortran2003. However, we will see that full support is still not available and limits remain upon what can be done.

cd ../example4
make
./call_c.exe

Looking in call_c.f90 we see that this time we have declared a user-derived type, with a BIND(C) parameter:

  type, BIND(C) :: mystuff
     integer(C_INT)                           :: f_dataLen
     character(C_CHAR), dimension(array_size) :: f_data1
     real(C_DOUBLE),    dimension(array_size) :: f_data2
  end type mystuff

  type(mystuff) :: someStuff

This, we can happily to pass to C:

call cfunc(someStuff)

which is expecting a struct of the form:

#define DATALEN 5

/* using pointers here will result in a seg fault */
struct gubbins
{
  int    c_dataLen;
  char   c_charData[DATALEN];
  double c_realData[DATALEN];
};

All is fine for scalar variables, such as the c_dataLen, but you will notice that we couldn't use a C variable length array, or pointers in the struct. In turn, this has forced us to use C arrays of a hardwired length, which is rather unsatisfactory. If the arrays are specified with a mismatching length, you will get an instant bug, which we have tried to protect against with:

  /* Provide at least _some_ protection against subtle bugs */
  if (mystuff->c_dataLen != DATALEN) {
    fprintf(stderr, "Error: Assumed DATALEN does not match value in struct\n");
    exit(EXIT_FAILURE);

  }

Passing user-derived types to C and vice versa is still a very useful feature, and if using scalar variables suits your code, then you are in a good position. However, the foregoing does illustrate that while Fortran2003 significantly improves matters, it does not provide a complete solution to C-Fortran interoperability.

Calling Fortran from Python using F2PY

Another popular pairing is to use Python to create the interface, utilising it's strong support for text processing, GUI building and graphics, and to use a traditional programming language, such as Fortran or C/C++, to rapidly perform any number crunching.

In the examples below, we'll take a quick peek at the support that f2py (provided as part of the numpy python package) gives those who wish to call Fortran routines from their Python scripts.

cd ../example5
module add languages/python-2.7.2.0

Useful information on f2py can be found at the following URLs (indeed all of the examples below are derived from the linked documentation):

• http://cens.ioc.ee/projects/f2py2e/
• http://cens.ioc.ee/projects/f2py2e/usersguide/

A Quick Test

OK, let's just prove that we can quickly and simply call a Fortran subroutine from Python.

First, run f2py (we'll request the use of the gfortran compiler):

f2py --fcompiler=gnu95 -c -m hello hello.f

You'll see that the result in the creation of a shared object file, hello.so. We can test our labours using the following python script:

#!/bin/env python

import hello
print hello.__doc__
print hello.foo.__doc__
hello.foo(4)

where hello.so is the subject of the line import hello. This is the output when we run the script:

./test.py
This module 'hello' is auto-generated with f2py (version:2).
Functions:
  foo(a)
.
foo - Function signature:
  foo(a)
Required arguments:
  a : input int

 Hello from Fortran!
 a= 4

Fortran90 Constructs

OK, that's great. However, some will rightly ask whether f2py fully supports the Fortran90 dialect. In an attempt to answer that, consider the following examples:

1. moddata: This example introduces Fortran90 modules.
2. allocarr: This example adds in the use of a Fortran90 allocatable array.
3. type: This example demonstrates the use of a Fortran90 user derived type.
4. pointer: This shows that F2PY does not support Fortran90 pointers (although the Fortran code itself is legal).

Take some time to review the Fortran90 source code and the corresponding python test scripts.

Here's our first example. moddata.f90 includes a Fortran90 module. First we compile the source code and create an interface, all in a single invocation of f2py:

f2py --fcompiler=gnu95 -c -m moddata moddata.f90

Running the test script, moddata.py should yield:

./moddata.py
i - 'i'-scalar
x - 'i'-array(4)
a - 'f'-array(2,3)
foo - Function signature:
  foo()


 i=           5
 x=[           1           2           0           0 ]
 a=[
 [  1.00000000     ,   2.0000000     ,   3.0000000     ]
 [   4.0000000     ,   5.0000000     ,   6.0000000     ]
 ]

Our second example, allocarr.f90, uses an allocatable array and is prepared by typing:

f2py --fcompiler=gnu95 -c -m allocarr allocarr.f90

The test script, allocarr.py should yield:

./allocarr.py
b - 'f'-array(-1,-1), not allocated
foo - Function signature:
  foo()


 b is not allocated
 b=[
  1.00000000       2.0000000       3.0000000    
   4.0000000       5.0000000       6.0000000    
 ]
 b=[
  1.00000000       2.0000000       3.0000000    
   4.0000000       5.0000000       6.0000000    
   7.0000000       8.0000000       9.0000000    
 ]
 b is not allocated

Our third example, type.f90, shows the use of a user derived type:

f2py --fcompiler=gnu95 -c -m type type.f90

The corresponding test, type.py, pithily outputs:

./type.py
    1   2.0000000000000000        3.0000000000000000                4           5           6

Lastly, pointer.f90, attempts to use a Fortran90 pointer. Although the source code will compile, f2py does not support the use of pointers and so the test scripts produces an error:

./pointer.py
c - 'f'-array(-1,-1), not allocated
bar - Function signature:
  bar()


 c is not associated
Traceback (most recent call last):
  File "./pointer.py", line 10, in <module>
    pointer.mod.c = [[1,2,3],[4,5,6]]         # allocate/initialize c
SystemError: error return without exception set

Calling C from Python using SWIG

NB You will need SWIG installed on your machine for this example. SWIG is not installed on bluecrystal phases 1 & 2, but is installed on phase 3.

SWIG stands for the Simplified Wrapper and Interface Generator.

This example covers calling C and C++ from Python. However, SWIG can also be used to connect C and C++ code to other scripting languages, such as Perl, PHP, Tcl and Ruby.

See:

• http://www.swig.org/tutorial.html

First of all, let's move to the directory of examples:

cd ../exampleX
ls
example.c  example.i  header.h  Makefile  pair.h  pair.i  test.py

You will notice that in addition to the source code files, interface files--with file extension <file>.i--must also be written. This is in contrast to f2py, which automatically created the interface files. I have created a makefile to simplify the creation of the shared objects. Just type:

make

Then, in order to test the C and C++ wrapping, run:

./test.py
===wrapped C code===
example.fact(5) is: 120
example.my_mod(7,3) is: 1
example.get_time() is: Fri Aug 23 12:16:18 2013

===wrapped C++ code===
initialise a with: pair.pairii(3,4)
a.first is: 3
a.second is: 4
reset a.second = 16
a.second is: 16
initialise b with: pair.pairdi(3.5,8)
b.first is: 3.5
b.second is: 8

Calling C/C++ and Fortran from Matlab using MEX Files

• http://www.mathworks.co.uk/help/matlab/create-mex-files.html

Calling C/C++ and Fortran from R

• http://www.biostat.jhsph.edu/~rpeng/docs/interface.pdf
• http://users.stat.umn.edu/~geyer/rc/

Appendices

Making a Daisy Chain: Calling Fortran from C from Fortran

cd ../exampleQ

Here we pass a regular derived type through some C code and back out into some Fortran. We can see that the derived type emerges unscathed, since the C code treats it as an opaque pacel, through the use of a pointer to void. In Fortran:

  type station
     real                  :: frequency                   ! MHz
     character(len=maxstr) :: name                        ! str
     character(len=maxstr) :: broadcaster                 ! str
  end type station

  type(station),pointer,dimension(:)   :: stations    ! array of stations

  ...

  call cfunc( ... ,stations)

and in the C:

  void cfunc_( ..., void* stations)