Berkeley UPC - Unified Parallel C

Home

Downloads

Documentation

Bugs

Publications

Demos

Contact

Internal

Mixing UPC and C/C++/MPI/FORTRAN

As of release 2.0, Berkeley UPC supports creating applications which are a hybrid of UPC and C, C++, and/or FORTRAN. We also support applications which contain MPI (Message Passage Interface) calls. This allows you to rewrite performance-critical sections of an existing application in UPC, without needing to rewrite the entire application. It also lets you link a UPC application to existing MPI and/or C++/C/FORTRAN libraries.

The UPC Specification does not address language interoperability issues, and so the interfaces described here should be considered Berkeley UPC extensions.

General Concepts

UPC is a superset of ANSI/ISO C99. Thus, UPC code can refer to any functions/variables in C/C++/FORTRAN code, provided that they "look like C" to the UPC code. In the other direction, UPC functions and variables can be treated as C by other languages, so long as they do NOT contain any UPC-specific keywords (no 'shared', etc., in declarations).

NOTE: if -pthreads is used to generate UPC executables, differences emerge between C and UPC global variables, and function calls become the only interface from other languages into UPC. See the Programming Hybrid Applications with Pthreads section.

Bootstrapping the UPC Runtime from an external language

The 'main()' function can live either in UPC code, or in an object written in another language. If 'main' is in a UPC file, the UPC runtime is bootstrapped normally, and no special logic is needed at program exit.

If 'main()' does not live in UPC code, 'bupc_init()' or 'bupc_init_reentrant()' must be called at startup to bootstrap the Berkeley UPC runtime, and 'bupc_exit()' should be used at program exit. These functions are available by #including <bupc_extern.h>:

    #include <bupc_extern.h>

    int main(int argc, char** argv) 
    {
        int exitcode = 0;

        bupc_init(&argc, &argv);
        ... rest of program...
        bupc_exit(exitcode);
    }

The 'bupc_init_reentrant()' function must be used when '-pthreads' is used. It also supports non-pthreaded applications, and so it is preferred for portability.

The call to 'bupc_init()' (or 'bupc_init_reentrant()') should be the first statement in 'main()'. The semantics of any code appearing before it is implementation-defined (for example, it is undefined how many threads of control will run that code, or whether stdin/stdout/stderr are functional). Similarly, no code should follow a call to 'bupc_exit()': it is not defined if such code will even be reached (hint: it won't). See the comments in <bupc_extern.h> for more information.

If regular 'exit()' (or a return from main) is performed from non-UPC code, instead of 'bupc_exit()', the result is as if 'upc_global_exit()' had been called--i.e., program termination will occur immediately, for all threads, without the final UPC barrier that occurs during normal UPC program termination. Calling '_exit' is strongly discouraged--program behavior is undefined in this case, and even process cleanup is not guaranteed (i.e. zombie processes may be left behind).

The path needed to include <bupc_extern.h> is always available by calling 'upcc -print-include-dir'. Thus a C file that #includes <bupc_extern.h> can be portably compiled with

    gcc -I`upcc -print-include-dir` -c main.c

Finally, whenever main() is not declared within a .upc file, the call to link UPC and non-UPC objects together must use the '--extern-main' flag:

    upcc foo.upc 
    upcc --extern-main foo.o main.o

Hybrid C/UPC programs

Referring to C variables/functions in UPC code is quite simple. The standard mechanism of #including a header file containing C declarations/prototypes is all UPC needs to be able to use C symbols. Note there is one problematic corner case - if a C header included from UPC code uses a UPC keyword (eg shared) as the name of an function or variable, this will cause an error in the UPC compile.

To have C code refer to a UPC variable or call a UPC function, have the C file #include a header file that declares the UPC variables/functions (none of which may contain any UPC-specific constructs, like 'strict'), and then the C code can refer to them normally. You may use the '__UPC__' macro (always defined during UPC compilation) to hide UPC-specific constructs:

    #ifndef MY_UPC_HEADER_H
    #define MY_UPC_HEADER_H

    /* Variables/Routines exported to C routines */
    extern int foo;
    int get_bar();
    void set_bar(int newval);

    /* UPC-specific declarations */
    #ifdef __UPC__
      extern shared int bar;
      extern strict shared long myarray[THREADS * 2];
    #endif /* __UPC__ */

    #endif /* MY_UPC_HEADER_H */

You can compile C code with any C compiler that is ABI-compatible with the one used by Berkeley UPC (use 'upcc -version' to see which compiler upcc uses). Then pass both your C and UPC object files (and/or any C libraries you wish to use) to upcc to link:

    upcc -c foo.upc
    gcc -c bar.c
    upcc foo.o bar.o -lm
    upcrun -n 2 a.out

Hybrid C++/UPC programs

Having UPC and C++ code refer to each other's variables/routines works similarly to the UPC/C case, except that all declarations from UPC must appear as "C" declarations to the C++ compiler, and all declarations from C++ must appear as "C" to the UPC compiler. This means that when C++ compilation is in effect (i.e. the '__cplusplus' macro is defined), extern "C" must be used. For instance, to make the example UPC header given above for C work with either C/C++:

    #ifndef MY_UPC_HEADER_H
    #define MY_UPC_HEADER_H

    /* Variables/Routines exported to C/C++ routines */
    #ifdef __cplusplus
      extern "C" {
    #endif
        extern int foo;
        int get_bar();
        void set_bar(int newval);
    #ifdef __cplusplus
      } /* end "extern" */
    #endif

    /* UPC-specific declarations */
    #ifdef __UPC__
      extern shared int bar;
      extern strict shared long myarray[THREADS * 2];
    #endif /* __UPC__ */

    #endif /* MY_UPC_HEADER_H */

Similarly, a C++ header that is also #included by UPC code would need to use the same extern "C" wrapper around any symbols intended for use by UPC, and any C++-specific constructs would need to be hidden from UPC by putting them within an '#ifdef __cplusplus' block.

You can compile C++ code with any C++ compiler that is ABI-compatible with the C compiler used by Berkeley UPC. At link time, 'upcc' additionally needs to be pointed at the C++ linker that should be used for the final link step: use the '-link-with' flag for this:

    upcc -c foo.upc
    g++ -c bar.cc
    upcc -link-with=g++ foo.o bar.o -lsomeC++library
    upcrun -n 2 a.out

Exceptions:

When compiling with '-network=udp', upcc already uses a C++ compiler for linking, and you must use it (or another compatible C++ compiler) as the argument to '-link-with'. You should also use it to compile any C++ objects/libraries linked into the application. Use 'upcc -version -network=udp' to see which C++ compiler upcc uses for UDP.
When compiling with '-network=mpi', upcc must use a MPI-aware linker. You must thus use the '-link-with' flag to point upcc at a C++-capable MPI linker ('mpiCC' or 'mpic++' on many systems, but other names are also used: consult your system MPI documentation).

Hybrid FORTRAN/UPC programs

The conventions for interoperating between FORTRAN and C vary across different systems/compilers (some FORTRAN compilers transform all symbols to upper-case, or prepend an underscore to names, etc.). Berkeley UPC code can refer to FORTRAN symbols in whatever way regular C code can, given the back-end C compiler that upcc uses and the FORTRAN compiler used for FORTRAN code/libraries. As with C/C++, FORTRAN can not access any UPC variables/functions that use UPC-specific constructs like 'shared'.

As with C objects, you should generally be able to simply pass FORTRAN objects/libraries to the upcc linker:

    upcc -c foo.upc
    f77 -c bar.f
    upcc foo.o bar.o -lsomeFortranLibrary
    upcrun -n 2 a.out

Note: different systems use different methods for linking together C and FORTRAN. You may need to consult your system documentation. You may select a different linker for upcc to use via the '-link-with' flag, and/or pass any linker-specific flags needed via upcc's '-Wl,' flag. See the upcc man page.

MPI and UPC

Berkeley UPC now contains support for creating applications which use both UPC and MPI explicitly in user code (note that this is different from UPC programs which do not explicitly use MPI, but are compiled with 'upcc -network=mpi': such programs do not require any special treatment). MPI calls can appear within UPC files, and/or in non-UPC code that is linked into the application. Note however, that strict programming conventions must be adhered to when switching between MPI and UPC network communication, otherwise deadlock may result on some systems.

Note: at present, compiling mixed MPI/UPC applications requires that the Berkeley UPC runtime be configured and built with 'CC' and 'MPI_CC' set to a C MPI compiler (and 'CXX' set to a C++ MPI compiler, unless '--disable-udp' is passed). Such a runtime will always use an MPI compiler to compile UPC programs, even those which do not use MPI: you may thus wish to keep a separate runtime installation specifically for compiling MPI/UPC programs. Finally, MPI interoperability is not provided for all systems and network types, and '-pthreads' is not supported. See the 'INSTALL.TXT' document in the runtime for more information.

UPC files which contain calls to MPI functions must '#include <mpi.h>', and must be compiled with the '-uses-mpi' flag. If any objects in a UPC application contain calls to MPI, the flag must be passed to the upcc at link time:

    upcc -uses-mpi -c foo.upc
    mpicc -c bar.c
    upcc -uses-mpi foo.o bar.o

On some systems, '-lmpi' must also be passed to 'upcc' in order for MPI symbols to be resolved. If C++ MPI objects are linked into a UPC application, the '-link-with' flag must be additionaly used at link time to point upcc at a C++ MPI compiler to use for linking.

    upcc -uses-mpi -c foo.upc
    mpic++ -c bar.cc
    upcc -uses-mpi -link-with=mpic++ foo.o bar.o

When '-uses-mpi' is used to link a UPC application, the 'MPI_Init()' and 'MPI_Finalize()' calls are handled by the UPC runtime--these calls should not appear in client code. If 'main()' exists in a non-UPC file, calls to 'bupc_init()' and 'bupc_exit()' should replace any calls to the MPI init/finalize functions.

Certain UPC network types (notably 'mpi', 'ibv' and 'ofi') may use MPI under the covers. For this reason, non-UPC MPI objects should be compiled with the same MPI compiler family as is used by upcc. To see the MPI C compiler used by upcc, use 'upcc -print-mpicc'.

Both MPI and UPC cause network communication. At present, the network traffic generated by MPI is not coordinated with that generated by UPC code. As a result, it is quite easy to cause network deadlock when mixing MPI and UPC, unless the following protocol is strictly observed:

When the application starts, the first MPI or UPC code that results in network traffic from any thread should be considered to put the application in 'MPI' or 'UPC' mode, respectively.
When an application is in 'MPI' mode, and needs to switch to using UPC, it should collectively execute an 'MPI_Barrier()' as the last MPI call before causing any UPC-initiated communication. Once any UPC communication has occurred from any thread, the program should be considered to have switched to 'UPC' mode.
When an application is in 'UPC' mode, and an MPI call that may cause network traffic is needed, a collective call to 'upc_barrier' (or, equivalently, a 'upc_notify' followed by a 'upc_wait') should be executed as the last UPC communication before any MPI calls are made. Once any MPI functions have been called from any thread, the program should be considered to be in 'MPI' mode.

If this simple construct--UPC code must be followed by a UPC barrier, and MPI code must be followed by an MPI_Barrier--is followed, deadlock should not occur (Note: we are investigating support for automatically avoiding MPI/UPC deadlock, but unfortunately it is a difficult problem, and solutions would likely be specific to particular MPI implementations).

Combining multiple languages with UPC

The above methods for mixing UPC with C/C++/FORTRAN/MPI are not mutually exclusive, and so more than one of these languages/APIs can be mixed with UPC in the same application. For instance, a C/C++/FORTRAN/MPI/UPC application might be built and run with

    upcc -c upc.upc
    gcc -c c.c
    g++ -c c++.cc
    mpiCC -c mpi.cc
    f77 -c fortran.f
    upcc -uses-mpi -link-with=mpiCC upc.o c.o c++.o mpi.o fortran.o -lFortranLib -lC++lib -lClib
    upcrun -n 2 a.out

Passing pointers-to-shared between non-UPC and UPC code

The <bupc_extern.h> file also contains a number of convenience functions which allow non-UPC code to allocate shared memory as standard 'void *' pointers, which UPC code can then cast back to UPC shared pointers using the 'bupc_local_to_shared' function (a Berkeley UPC extension). This allows a convenient way for UPC code to provide non-UPC code with access to shared memory that has affinity to the executing thread, then convert it back to the original UPC type for UPC communication calls. See <bupc_extern.h> (located in the directory provided by 'upcc -print-include-dir') for details.

Programming hybrid applications with -pthreads

When 'upcc -pthreads' is used to create a pthreaded UPC application, global variables in UPC must be made thread-local (i.e., a copy of the variable is made for each UPC thread). This does not occur, however, for variables declared in other languages. All UPC threads in the same process will share the same copy of 'stdout', for instance. Generally, this is the desired behavior (it would be very detrimental for UPC to try to make multiple copies of 'stdout'). It is also the only feasible implementation strategy for pthreaded UPC, since the C library and other system libraries are generally not built by the UPC compiler, and thus C variables within them could not be made pthread-local even if this was desired.

The result of this is that when '-pthreads' is used, C/C++/FORTRAN code can not refer to UPC global variables. However, UPC functions which have a C interface (no UPC-specific constructs) can still be called. Note, though, that UPC functions should only be called from pthreads that were created by the UPC runtime: pthreads created by user calls to 'pthread_create' cannot safely call UPC routines.

When '-pthreads' is used in a hybrid application, any non-UPC code linked into the application must be thread-safe (i.e. capable of being referenced by multiple threads at the same time), if you plan to call it concurrently from more than one UPC thread.

Many MPI libraries are not safe to use with pthreads, and those that are often require special initialization. At present Berkeley UPC does not support hybrid MPI/UPC applications with '-pthreads', but support may be added for reentrant MPI libraries in the future.

See the Using Pthreads section of the UPC User's Manual for more details on '-pthreads' usage.