Berkeley UPC - Unified Parallel C

Contents

• UPC standards and APIs supported by this version of Berkeley UPC

• A Sample UPC Program

• Compiling UPC programs with 'upcc'

• Creating libraries of UPC code

• Running UPC programs with 'upcrun'

• Using pthreaded UPC Programs

• Debugging UPC Programs

• Analyzing UPC Programs with 'upc_trace'

• Gathering application statistics

• Profiling UPC Programs with 'upcc -pg' and 'gprof'

• Berkeley-specific extensions to the UPC Language
  • Non-blocking memcpy functions

  • Non-contiguous memcpy functions

  • Point-to-point synchronization functions

  • Value-based collectives convenience interface (bupc_collectivev.h)

  • The 'bupc_all_reduce_all' function family

  • The 'bupc_dump_shared' function

  • The 'bupc_ptradd' function

  • The 'bupc_poll' function

  • The 'bupc_assert_type' built-in

  • High-precision wall-clock timer support

  • Runtime thread layout query for hierarchical systems

  • Castability of pointers-to-shared

  • Atomic Memory Operations

  • Collective deallocation functions

  • The 'bupc_system' function

• Known bugs and limitations

• Platform-specific issues

UPC standards and APIs supported by this version of Berkeley UPC

This version of Berkeley UPC includes:

• Full support for the UPC Language Specifications, Version 1.3, a specification which is itself a superset of ISO/IEC 9899:1999 (aka. "C99").
  That document specifies the UPC language constructs, as well as the following core libraries:
  • UPC Utilities <upc.h>
  • UPC Standard Typed <upc_types.h>
  These libraries are fully implemented in BUPC and well-tuned for most platform configurations.

• A complete implementation of the UPC Required library, as specified in UPC Required Library Specifications, Version 1.3.
  That document specifies the following required libraries:
  • UPC Collective Utilities <upc_collective.h>
  • High-Performance Wall-Clock Timers <upc_tick.h>
  These libraries are fully implemented in BUPC and well-tuned for most platform configurations.

• A complete implementation of the UPC Optional library, as specified in UPC Optional Library Specifications, Version 1.3.
  That document specifies the following optional libraries:
  • UPC Non-Blocking Transfer Operations <upc_nb.h>:
    BUPC provides a complete and high-performance implementation.
  • Castability Functions <upc_castable.h>:
    BUPC provides a complete and high-performance implementation.
  • UPC Atomic Memory Operations <upc_atomic.h>:
    As of version 2019.4.0, BUPC provides a complete and high-performance implementation. For details see Atomic Memory Operations.
    The runtime source distribution also includes a reference implementation, located in upc-tests/UPC-Atomic-RefImp, which is functionally spec-compliant but is not designed to be highly performant (atomicity is achieved via UPC locks).
  • UPC Parallel I/O <upc_io.h>:
    BUPC provides a reference implementation, which is functionally spec-compliant but is not designed to be highly performant (all I/O is channeled through a single node).

• A number of Berkeley-specific extensions to the UPC Language.
  Some of these extensions have been adopted into the official specifications since their inception, and are now deprecated in favor of the standardized versions (as noted in each section).

The three UPC specifications referenced above are also available for convenience as a combined document:
    UPC Language and Library Specifications, Version 1.3
    UPC Consortium, Lawrence Berkeley National Lab Tech Report LBNL-6623E, Nov 2013.

A Sample UPC Program

    #include <upc_relaxed.h>
    #include <stdio.h>

    int main() {
      printf("Hello from thread %i/%i\n", MYTHREAD, THREADS);
      upc_barrier;
      return 0;
    }

For more involved examples of UPC code, see the UPC Language Tutorials - from the UPC Language Community website (archived) and the 'upc-examples' directory in of the Berkeley UPC runtime distribution. The Official UPC Specifications are a useful reference, and contains a description of the standard libraries.

Compiling UPC programs with 'upcc'

    upcc -o light particle.upc wave.c -lgrottymath

upcc recognizes most commonly used C compiler flags (-D, -I, etc.). It also uses a number of its own flags for the choice of network API your program will run over, for compiling your UPC code for a static number of threads, and other UPC-specific options. See the upcc man page for details.

Choosing a network API for your UPC executable

Note that you can only compile for a given network type if your Berkeley UPC runtime was configured to support it at build/installation time. To see which APIs are supported in your installation, and to see which is used by default, use 'upcc --version'.

Compiling for a fixed number of UPC threads

An executable compiled for a fixed number of UPC threads will fail at startup if you try to run it with a different number of threads. However, fixing the number of threads allows optimization on certain operations (such as shared pointer arithmetic), especially when the number of threads is a power of 2.

Overriding global upcc.conf settings in a user configuration file

UPC Standard and Berkeley-specific preprocessor macros

Using a remote UPC-to-C translator

A remote translator can be contacted via either the HTTP or SSH protocols. To use HTTP, the 'upcc.cgi' CGI script (located in the 'contrib' directory of the runtime distribution) must be installed and configured with a web server on the remote host. Simply set the 'translator' parameter in your user configuration file (or the global 'upcc.conf') to the URL for the CGI script. To use SSH, you must be able to login to the remote host using SSH, and the 'translator' parameter must be set to 'remote_host:/path/to/translator'. You will want to use key-based authentication, and 'ssh-agent' to avoid entering your password each time you compile. See our SSH Agent Tutorial.

When using an HTTP-based remote translator, upcc also includes support for use of an HTTP proxy. Set the 'http_proxy' parameter in your user configuration file (or the global 'upcc.conf') to the proxy URL. The upcc front end does not currently support HTTPS or SOCKS proxies, nor HTTP proxies that require authentication (HTTP error 407).

Creating libraries of UPC code

If you wish to create a reusable set of compiled code, you must currently keep the files in *.o format. So, instead of the traditional C format, where you'd create 'libmyupc.a', and then link with something like

    upcc myprogram.o -L/libpath -lmyupc

    upcc myprogram.o /libpath/libmyupc/*.o 

Running UPC programs

Berkeley UPC executables should be run the same way as any other parallel program on your system that uses the same underyling network API. So, for instance, a program compiled with '--network=mpi' is run on many systems via 'mpirun -np <number of processes> a.out'. Other systems may use other invocations, such as 'prun' or 'poe', especially when API's other than MPI are used. Consult your system's documentation for details.

Using 'upcrun'

    upcrun -n 4 parboil

An additional benefit of using upcrun is that it provides consistent support for propagating environment variables to all threads of your UPC program. If you use upcrun, any environment variable beginning with either 'UPC_' or 'GASNET_' is guaranteed to be propagated to all threads. (Support for propagating all environment variables is planned). If you do not use upcrun, environment propagation will only work to the extent that the parallel job launcher you use provides it normally.

You can see how upcrun thinks your job should be run without actually running it by passing the upcrun '-t' flag. Also, 'upcrun -i <executable>' will provide information about a Berkeley UPC executable, such as the network API that it was built against, and the number of fixed threads (if any) that it was compiled for.

See 'upcrun --help' or the upcrun man page for more information.

Setting the amount of shared memory available to your applications

The default amount of shared memory to reserve per UPC thread on a system is chosen at configure time (see INSTALL.TXT for details), but you can override that value for a particular application at either compile time or at job startup. Generally this is only needed if you observe that your application is running out of either shared or regular C memory.

To embed a different default amount of shared memory into your application, simply pass 'upcc -shared-heap=144MB' for instance (to get 144 megabytes per UPC thread). You can also use 'GB' for gigabyte amounts (if neither 'MB' nor 'GB' is used, megabytes are assumed). To override the embedded default amount of shared memory at application startup, set the UPC_SHARED_HEAP_SIZE environment variable to whatever value you want ('2GB', etc.), or pass '-shared-heap' to upcrun.

While it is tempting to simply grab an extremely large shared memory segment, be aware that this is not always a good idea, or even possible. Since the shared address space range cannot be used for regular malloc allocations, creating too large of a shared space can cause the amount of regular heap memory available to your application to become small (causing malloc to eventually return NULL when you request more memory). Also, the shared memory space is reserved via an mmap() call, and while this does not generally cause any physical memory pages to be allocated, certain operating systems (for instance, Linux) will not allow more memory to be reserved by applications then the OS can guarantee is available, and so allocating a shared region larger than the physical memory (plus swap space) may fail.

The default amount of shared memory per UPC thread can be changed system-wide by modifying the 'shared_heap' parameter in the installation's upcc.conf file. You can override the system-wide default for your own applications by setting shared_heap in your user configuration file.

Using pthreaded Berkeley UPC programs

The '-pthreads' flag must be passed consistently at all stages of compilation and linking. Also, when pthreads are used, upcc needs to delay much of the compilation of your code until link time, so if you split code generation into separate compilation and linking steps (i.e., 'upcc -c foo.upc', followed by 'upcc foo.o bar.o'), you need to pass any macro and/or include path directives (ex: '-DFOO=bar -I/usr/local/include') to upcc for both the compilation and link commands.

Any C libraries that your code links against must be thread-safe in order to be used with -pthreads. If one or more of your libraries is not thread-safe, you must compile without pthreads, and run separate processes on the same node to exploit an SMP system. In the non-pthreads case support for shared memory communication among UPC processes on an SMP node is available on many systems via the "PSHM" feature (See "INTRA-NODE SHARED MEMORY SUPPORT" in INSTALL.TXT).

When you link an application with '-pthreads', a subdirectory named <executable_name>_pthread-link will be created in the current directory. This directory exists in order to speed up further linking commands of the same program. If you link the same application again with the same object file names, and none of the global static unshared variables in your program have changed name or size, recompilation of all the files in your application can be avoided, which can make a significant difference in build time for programs with many source files. You may delete the temporary directory at any time without any side effects (other than possibly longer link times). One can prevent this optimization with the -nolink-cache flag to upcc.

Unless otherwise specified, pthreaded UPC applications use a default number of pthreads per process (run 'upcc --version' to see the default for your system. This number is set in the upcc.conf configuration file, and can be changed there (or in your user configuration file). It can also be overridden in several ways. Compiling with 'upcc -pthreads=<NUMBER>' changes the default number of pthreads per UPC process for an executable to NUMBER. If the 'UPC_PTHREADS_PER_PROC' environment variable is set to a nonzero integer when you run a UPC program, it will override any default value. Finally, upcrun is smart about pthreads in several ways. First, if you run a pthreaded parallel job with 'upcrun -n <NUMBER> ...', the number of processes actually launched will be divided by the number of pthreads, so that exactly NUMBER UPC threads are used. Second, if the PSHM feature is disabled and you use -network=smp (generating an executable that will run only a single process), upcrun -n NUMBER will automatically set the number of pthreads to NUMBER.

Debugging Berkeley UPC programs

You can use a regular C debugger and get usable debugging support. Berkeley UPC provides several mechanisms for attaching a regular C debugger to one or more of your UPC application's threads at various points during execution. While this does not provide a fully normal debugging environment (the debugger will show the C code emitted by our translator, rather than your UPC code), it can still allow you to see program stack traces and other important information. This can be very useful if you wish to submit a helpful bug report to us. See Attaching a regular C debugger to Berkeley UPC programs for details.

Berkeley UPC also supports automatically generating backtraces if a fatal error occurs in your program. This will allow you to see a stack trace of the function calls that your program was in at the time it crashed. To use auto-backtracing, run with upcrun -backtrace or set GASNET_BACKTRACE=1 in your environment. The level of backtracing support available depends on the back-end C compiler and operating system, and so not all systems are equally functional, and some systems will not provide backtraces. See gasnet/README for more information on backtracing.

Analyzing UPC Programs with 'upc_trace'

Examining tracing information is one of the best ways to go about optimizing your UPC program. It provides a way for you to see which lines of your code are generating the most network traffic (and the size of the network messages used). From this you may be able to determine how to either avoid some of this traffic, or change your code to use fewer, larger messages (for instance, by replacing sets of individual reads/writes with bulk memory movement calls like 'upc_memget()', etc.), which is typically more efficient. Examining barrier wait times can also let you know if your computations are imbalanced across threads, and/or if you could profit by using split-phase barriers, moving computation in between 'upc_notify' and 'upc_wait'.

How to use 'upc_trace'

1. Tracing must be enabled in order to work. By default, tracing is enabled for debug compilations (i.e. if 'upcc -g' is used), but not otherwise (as it incurs some overhead). If you wish to also trace non-debug executables, you must rebuild your UPC runtime system and pass '--with-multiconf=+opt_trace' to configure, then build your application with 'upcc -trace'.

2. You must run your application with 'upcrun -trace ...' or 'upcrun -tracefile TRACE_FILE_NAME ...'. Either of these flags causes your UPC executable to dump out tracing information while it executes. The '-trace' flag causes one file per UPC thread to be generated, with the name 'upc_trace-a.out..-N', where 'a.out' is the name of your executable, and 'N' is the UPC thread's number. The '-tracefile NAME' option lets you specify your own name for the tracing file(s): if the name contains a '%' character, one trace file per thread is generated, with the '%' replaced with the UPC thread's number. Otherwise, all threads will write to the same file.

   Note that running with tracing may slow down your application considerably: the exact amount depends on your filesystem, and the ratio of communication/computation in your program. If you are only interested in a subset of trace information, consider setting 'GASNET_TRACEMASK' as described below.

3. After your application has completed, you may run 'upc_trace' on one or more of the trace files generated by your program run:
   1. Running 'upc_trace' on a trace file generated by a single UPC thread shows the information only for that thread. If you pass multiple files from the same application run, the information for the various threads is coalesced, so passing in all the tracefiles generated by a run allows you to see information for the entire application.
   2. There are a number of flags to 'upc_trace' which control what kinds of information is reported, and how it is sorted. See 'upc_trace --help' or the upc_trace man page for details.
   3. Note that upc_trace may take a while to run, especially on large tracefiles. Consider setting GASNET_TRACEMASK and/or GASNET_TRACELOCAL (described below) to streamline the trace file's contents to include only those events you're interested in analyzing.
   4. If you compile with 'upcc -opt', it is possible that the UPC-to-C translator has coalesced some of the network operations in your program, in order to get better network performance. This means that 'upc_trace' may not report communication for certain lines of your program, and other lines may seem to be getting/putting more data than they should.
   The trace files are in a human-readable text format, and are amenable to generic text-file processing tools such as grep, perl, etc. This means you can also elect to perform your own analyses on the trace file if the upc_trace tool doesn't provide the information you are looking for.

Controlling what gets logged in the trace file by setting GASNET_TRACEMASK

To trace only a subset of these features, set the 'GASNET_TRACEMASK' environment variable to a string containing the ID's of the features you wish to trace. Note that the "N" and "H" flags must always be among those set for 'upc_trace' to work (if you are intending to manually examine the trace file, they do not need to be set).

So, for instance, if you are trying to perform an analysis that does not require get/put information, you are highly advised to set 'GASNET_TRACEMASK' to "BHN" and 'GASNET_TRACELOCAL' to "no" (or "0"). This will turn off tracing for all get and put operations. Since gets/puts are typically the majority of items in a full trace file, this will probably result in much faster program execution, a much smaller trace file, and faster analysis by 'upc_trace'.

Controlling tracing during runtime

    extern void         bupc_trace_setmask (const char *newmask);
    extern const char * bupc_trace_getmask (void);
    extern int          bupc_trace_gettracelocal (void);
    extern void         bupc_trace_settracelocal (int val);
    void                bupc_trace_printf ((const char *msg, ...));

   Ex: 
     bupc_trace_setmask("PGHN");   // trace everything
     bupc_trace_settracelocal(1);  // include local puts and gets 
     // do something...
     bupc_trace_setmask("");      // stop tracing

The 'bupc_trace_printf' utility outputs a message into the trace file, if it exists. Note that two sets of parentheses are required when invoking this operation, in order to allow it to compile away completely for non-tracing builds.

  Ex:   double A[4] = ...; 
        int i = ...;
        bupc_trace_printf(("the value of A[%i] is: %f", i, A[i]));

Gathering application statistics

To generate statistics, simply set the 'GASNET_STATSFILE' environment variable to a file name, into which statistics will be written at the end of your program's run. (Note: by default, only debug executables support statistics generation, as it incurs a performance penalty: if you wish to have non-debug UPC executables generate statistics, you must rebuild your UPC runtime system, passing '--with-multiconf=+opt_trace' to configure, then build your application with 'upcc -trace'.) You may generate both stats and tracing info for the same program run if you wish.

Just as with tracing, you may set a mask to control what types of events are included in the statistics, by setting the 'GASNET_STATSMASK' environment variable, and/or by calling the following functions:

    extern void         bupc_stats_setmask (const char *newmask);
    extern const char * bupc_stats_getmask (void);

Profiling UPC Programs with 'upcc -pg' and 'gprof'

    upcc -pg foo.c
    upcrun -n 2 a.out
    gprof a.out gmon.out.0/gmon.out gmon.out.1/gmon.out | less

Berkeley-specific extensions to the UPC Language

Non-blocking memcpy functions (partially deprecated)

NOTICE: A large portion of this Berkeley-specific extension is now officially deprecated in favor of the standardized version adopted into the official UPC specification. See below for details.

The full interface is described in sections 2 through 4 of our Proposal for Extending the UPC Memory Copy Library Functions. See that document for details on the functions and their usage.

NOTICE: The following interfaces have been adopted in the UPC Optional Library Specifications, Version 1.3, with semantics compatible to a subset of those given in the document referenced above, and are available in Berkeley UPC beginning with the 2.18 release. Users are strongly encouraged to develop new codes using the standardized interfaces and to migrate existing code to them. The corresponding portion of the Berkeley-specific non-blocking interfaces (specifically, those operating on contiguous data) are now officially deprecated, and the 'bupc_' prefixed equivalents of those functions will be removed in a future version.

    #define     __UPC_NB__  1  // predefined feature macro
    #include    <upc_nb.h>     // defines the following:

    // Explicit-handle non-blocking operations and synchronization:
    typedef      ... upc_handle_t;
    #define      UPC_COMPLETE_HANDLE ...
    upc_handle_t upc_memcpy_nb(shared void * restrict dst,
                               shared const void * restrict src,
                               size_t n);
    upc_handle_t upc_memget_nb(void * restrict dst,
                               shared const void * restrict src,
                               size_t n);
    upc_handle_t upc_memput_nb(shared void * restrict dst,
                               const void * restrict src,
                               size_t n);
    upc_handle_t upc_memset_nb(shared void *dst, int c, size_t n);
    int          upc_sync_attempt(upc_handle_t handle);
    void         upc_sync(upc_handle_t handle);

    // Implicit-handle non-blocking operations and synchronization:
    void upc_memcpy_nbi(shared void * restrict dst,
                        shared const void * restrict src,
                        size_t n);
    void upc_memget_nbi(void * restrict dst,
                        shared const void * restrict src,
                        size_t n);
    void upc_memput_nbi(shared void * restrict dst,
                        const void * restrict src,
                        size_t n);
    void upc_memset_nbi(shared void *dst, int c, size_t n);
    int  upc_synci_attempt(void);
    void upc_synci(void);

NOTE: The types upc_handle_t and bupc_handle_t are interchangable. One may freely mix the standard library calls in upc_nb.h with the Berkeley-specifc interfaces for contiguous non-blocking memcpy.

Non-contiguous memcpy functions

The full interface is described in sections 5 and 6 of our Proposal for Extending the UPC Memory Copy Library Functions. See that document for details on the functions and their usage.

NOTE: The types upc_handle_t and bupc_handle_t are interchangable. One may freely mix the standard library calls in upc_nb.h with the Berkeley-specifc interfaces for non-contiguous memcpy.

Point-to-point synchronization functions

The full interface is described in our Proposal for Extending the UPC Libraries with Explicit Point-to-Point Synchronization Support. See that document for details on the functions and their usage.

Value-based collectives convenience interface (bupc_collectivev.h)

The 'bupc_all_reduce_all' function family

The 'bupc_dump_shared' function

    int bupc_dump_shared(shared const void *ptr, char *buf, int maxlen);

    "<address=0x1234 (addrfield=0x1234), thread=4, phase=1>" 

Both bupc_dump_shared() and BUPC_DUMP_MIN_LENGTH are visible when any of the standard UPC headers (upc.h, upc_relaxed.h, or upc_strict.h) are #included.

The 'bupc_ptradd' function

The 'bupc_ptradd()' function provides support for performing pointer-to-shared arithmetic with variable blocksize, which need not be a compile-time constant.

  shared void * bupc_ptradd(shared void *p, size_t blockelems, size_t elemsz, ptrdiff_t elemincr);
    - 'p': the base pointer
    - 'blockelems': the block size (number of elements in a block)
    - 'elemsz': the element size (usually sizeof(*p))
    - 'elemincr': the positive or negative offset from the base pointer

The following call:

    bupc_ptradd(p, blockelems, sizeof(T), elemincr);

    shared [blockelems] T *q = p;
    q += elemincr;

  int blockelems = ...; // choose some arbitrary block size

  // allocate an array of doubles with that blocksize
  shared void *myarr = upc_all_alloc(..., blockelems*sizeof(double)); 

  // access element 14
  double d = *(shared double *)bupc_ptradd(myarr, blockelems, sizeof(double), 14);

It's worth noting that in some cases bupc_ptradd() may be less efficient than regular pointer-to-shared addition, because the compile-time constant blocksize of the pointer referent type generally makes the latter more amenable to compiler optimization of the addition operation and surrounding code. This is especially true in the case of indefinitely-blocked or cyclically-blocked pointers-to-shared. However, the potential cost may be worth the added convenience in non-performance-critical code.

The 'bupc_poll' function

You will normally not need to call this function, as the runtime will automagically perform checks for incoming network requests whenever your UPC code causes network activity to be performed, and this usually occurs fairly frequently in a UPC application. However, if you writing your own 'spin lock' style synchronization, you may need to use this function to avoid deadlock. Here is an example:

    shared strict int flag[THREADS];

    ...

    if (MYTHREAD % 2) {
        while (flag[MYTHREAD] == 0)
            bupc_poll();
    } else {
        ... some calculation ...
        flag[MYTHREAD + 1] = 1;
    }

If a program contains computationally intensive sections in which no remote accesses are performed for a long time, it is also possible that performance may be improved by intermittently calling bupc_poll, particularly if other threads are likely to be performing communication (eg. remote accesses, lock synchronization, shared memory allocation, etc.) during this time.

The 'bupc_assert_type' built-in (Berkeley UPC translator only)

1. 'expr' = any arbitrary (legal) UPC expression
2. 'type' = any legal C/UPC type

If 'expr' has a static type which is identical to 'type', does nothing. Otherwise, prints a non-fatal warning containing the line number and a description of the two differing types.

High-precision wall-clock timer support (deprecated)

NOTICE: This Berkeley-specific extension is now officially deprecated in favor of the standardized version adopted into the official UPC specification. See the end of this section for details.

    typedef     ... bupc_tick_t; /* 64-bit integral type */
    #define     BUPC_TICK_MAX ...
    #define     BUPC_TICK_MIN ...
    bupc_tick_t bupc_ticks_now (void);
    uint64_t    bupc_ticks_to_us (bupc_tick_t ticks);
    uint64_t    bupc_ticks_to_ns (bupc_tick_t ticks);
    double      bupc_ticks_granularityus (void); 
    double      bupc_ticks_overheadus (void);

The 'bupc_tick_t' type represents an integral quantity of abstract timer ticks, whose ratio to real time is system-dependent and thread-dependent. bupc_ticks_now() returns the current value of the tick timer for the calling thread, using the fastest mechanism available. bupc_ticks_to_us() and bupc_ticks_to_ns() convert a difference in bupc_tick_t values obtained by the calling thread into microseconds or nanoseconds, respectively. The bupc_ticks_to_{us,ns}() conversion calls can be significantly more expensive than the bupc_ticks_now() tick query, so for timing short intervals it's recommended to keep timing results in units of ticks until final output. BUPC_TICK_MAX and BUPC_TICK_MIN provide tick values which are respectively larger and smaller than any possible tick value. bupc_ticks_granularityus() and bupc_ticks_overheadus() respectively report the estimated microsecond granularity (minimum time between distinct ticks) and microsecond overhead (time it takes to read a single tick value, not including conversion) for the timer facility.

Example:

  bupc_tick_t start = bupc_ticks_now();
    compute_foo(); /* do something that needs to be timed */
  bupc_tick_t end = bupc_ticks_now();

  printf("Time was: %d microseconds\n", (int)bupc_ticks_to_us(end-start));

  printf("Timer granularity: <= %.3f us, overhead: ~ %.3f us\n",
       bupc_tick_granularityus(), bupc_tick_overheadus());
  printf("Estimated error: +- %.3f %%\n",
      100.0*(bupc_tick_granularityus()+bupc_tick_overheadus()) /
            bupc_ticks_to_us(end-start));

NOTICE: The following interfaces have been adopted in the UPC Required Library Specifications, Version 1.3, with similar semantics to those as described above, and are available in Berkeley UPC beginning with the 2.16 release. Users are strongly encouraged to develop new codes using the standardized interfaces and to migrate existing code to them. The Berkeley-specific 'bupc_' prefixed variants are now officially deprecated in favor of the standardized variant, and will be removed in a future version.

    #define     __UPC_TICK__  1  // predefined feature macro
    #include    <upc_tick.h>     // defines the following:
    typedef     ... upc_tick_t;
    #define     UPC_TICK_MAX ...
    #define     UPC_TICK_MIN ...
    upc_tick_t  upc_ticks_now (void);
    uint64_t    upc_ticks_to_ns (upc_tick_t ticks);

Runtime thread layout query for hierarchical systems

    unsigned int bupc_thread_distance(int threadX, int threadY); 
    #define     BUPC_THREADS_SAME     ...
    #define     BUPC_THREADS_VERYNEAR ...
    #define     BUPC_THREADS_NEAR     ...
    #define     BUPC_THREADS_FAR      ...
    #define     BUPC_THREADS_VERYFAR  ...

• BUPC_THREADS_SAME (must be defined to 0) implies threadX == threadY
• BUPC_THREADS_VERYNEAR implies threadX has the closest possible distance (fastest access) to threadY's memory without being the same actual thread
• BUPC_THREADS_NEAR implies distance not less than BUPC_THREADS_VERYNEAR, but not more than BUPC_THREADS_FAR
• BUPC_THREADS_FAR implies distance not less than BUPC_THREADS_NEAR, but not more than BUPC_THREADS_VERYFAR
• BUPC_THREADS_VERYFAR implies threadX has the farthest possible distance (slowest access) to threadY's memory

These constants have implementation-defined integral values which are monotonically increasing in the order given above. Implementations may add further intermediate level with values between BUPC_THREADS_VERYNEAR and BUPC_THREADS_VERYFAR (with no corresponding define) to represent deeper hierarchies, so users should test against the constants using <= or >= instead of ==.

The intent of the interface is for users to not rely on the physical significance of any particular level and simply test the differences to discover which threads are relatively closer than others. Implementations are encouraged to document the physical significance of the various levels whenever possible (see below), however any code based on assuming exactly N levels of hierarchy or a fixed significance for a particular level will probably not be performance portable to different implementations or machines.

The relation is symmettric, ie: bupc_thread_distance(X,Y) == bupc_thread_distance(Y,X)
but the relation is not transitive, ie: bupc_thread_distance(X,Y) == A && bupc_thread_distance(Y,Z) == A does NOT imply bupc_thread_distance(X,Z) == A

Furthermore, the value of bupc_thread_distance(X,Y) is guaranteed to be unchanged over the span of a single program execution, and the same value is returned regardless of the thread invoking the query.

Currently the significance of the BUPC_THREADS_* constants is as follows:

Castability of pointers-to-shared

Converting a pointer to "remote" shared data into a pointer-to-local (deprecated)

NOTICE: This Berkeley-specific extension is now officially deprecated in favor of the standardized version adopted into the official UPC specification. See the end of this section for details.

    int bupc_castable(shared void *ptr);
    int bupc_thread_castable(unsigned int threadnum);
    void * bupc_cast(shared void *ptr);

This family of functions implements a UPC language extension propsed by Brian Wibecan of HP. Their purpose is to allow a UPC programmer to take advantage of UPC implementations in which some or all of the shared data with affinity to a given UPC thread can be directly addressed by other UPC threads using a pointer-to-local.

We use the term 'castable' to denote that the UPC implementation is able to represent a given pointer-to-shared using a pointer-to-local on a given thread. Any pointer-to-shared with affinity to a thread is guaranteed (by the language spec) to castable by that same thread. However, in general shared storage with affinity to one thread is not castable by other threads. Depending on the UPC implementation, it is possible that for a given pair of threads either all, none, or only some of the shared address space with affinity to the first may be castable by the second.

bupc_castable() takes a pointer-to-shared as argument and returns non-zero if and only if the argument is castable by the calling thread. It is guaranteed that a call to bupc_castable() with an argument having affinity to the calling thread will always return non-zero.

bupc_thread_castable() takes a UPC thread number as argument and returns non-zero if and only if every pointer-to-shared with affinity to the argument thread is castable by the calling thread. It is guaranteed that bupc_thread_castable(MYTHREAD) is always non-zero.

bupc_cast() takes a pointer-to-shared as argument and returns a pointer-to-local. The returned pointer may be used to reference the same object as the argument only if the argument pointer is castable by the calling thread, as may determined by bupc_castable() or bupc_thread_castable(). Otherwise the returned pointer is NULL.

NOTICE: The following interfaces have been adopted in the UPC Optional Library Specifications, Version 1.3, with similar semantics to those as described above, and are available in Berkeley UPC beginning with the 2.16.2 release. Users are strongly encouraged to develop new codes using the standardized interfaces and to migrate existing code to them. The Berkeley-specific 'bupc_' prefixed variants are now officially deprecated in favor of the standardized variant, and will be removed in a future version.

    void *upc_cast(const shared void *ptr);
    upc_thread_info_t upc_thread_info(size_t threadnum);

Converting a pointer-to-local referencing shared data into a pointer-to-shared

In addition to the functions described in the previous section, Berkeley UPC 2.14.2 and later implement an 'inverse cast' function:

    shared void * bupc_inverse_cast(void *ptr);

Atomic Memory Operations

The UPC Optional Library Specifications, Version 1.3 adds a standardized UPC Atomic Memory Operations library in <upc_atomic.h>. This standardized atomics interface is fully implemented in this version of Berkeley UPC. The former Berkeley-specific atomics extensions (in the 'bupc_atomic*' function family), were subsumed by the 1.3 standardized atomics interface, and (after a lengthy deprecation period) that proprietary function family has now been removed. Berkeley UPC provides several new extensions to the 1.3 specified atomics interface, described below.

Atomic Domain creation hints

Berkeley UPC defines several extended hint values for the upc_atomichint_t hints argument to upc_all_atomicdomain_alloc() to control the behavior of the created atomic domain:

• UPC_ATOMIC_HINT_FAVOR_FAR - this hint requests the created atomic domain use algorithms that favor improved performance characteristics of atomic operations on locations in shared memory whose affinity is "far" from the initiating thread, such as those requiring communication over an interconnect. For example, this requests that atomic memory operations be offloaded to network hardware where such support is available and implemented. However, this may come at the expense of performance for atomic operations on "nearby" shared memory locations. For example, network atomic offload hardware is often not coherent with CPU-based atomic memory instructions, and thus they cannot safely be mixed on the same locations.
• UPC_ATOMIC_HINT_FAVOR_NEAR - this hint requests the created atomic domain use algorithms that favor improved performance characteristics of atomic operations on locations in shared memory whose affinity is "near" the initiating thread, for example with affinity to the initiating thread itself, or a thread sharing the same physical memory system. This may come at the expense of performance for atomic operations on distant shared memory locations.

Atomic domain hint values are guaranteed to be macros, thus the recommended portable means to use the hints described above is to protect their use with #ifdef, for example:

  #ifdef UPC_ATOMIC_HINT_FAVOR_FAR
    upc_atomichint_t hint = UPC_ATOMIC_HINT_FAVOR_FAR;
  #else
    upc_atomichint_t hint = 0;
  #endif

  // create an atomic domain for atomic get, set and increment operations on 64-bit unsigned integers,
  // using network offload hardware where possible
  upc_atomicdomain_t *my_ad = upc_all_atomicdomain_alloc(UPC_UINT64, UPC_SET|UPC_GET|UPC_INC, hint);

  // perform an atomic fetch-increment on A[i]
  uint64_t result;
  upc_atomic_relaxed(my_ad, &result, UPC_INC, &A[i], 0, 0);

Collective deallocation functions (deprecated)

NOTICE: This Berkeley-specific extension is now officially deprecated in favor of the standardized version adopted into the official UPC specification. See the end of this section for details.

    void bupc_all_free(shared void *ptr);
    void bupc_all_lock_free(upc_lock_t *lockptr);

These two functions implement collective alternatives to the standard functions upc_free() and upc_lock_free(), as a convenience to the programmer. Both functions must be called collectively by all threads with the same argument. The object referenced by the argument is guaranteed to remain valid until all threads have entered the collective deallocation call, but the function does not otherwise guarantee any synchronization or strict reference. In all other respects the semantics of these functions and constraints on their usage are identical to their non-collective variants.

NOTICE: The following interfaces have been adopted in the UPC Language Specifications, Version 1.3, with the same semantics as those as described above, and are available in Berkeley UPC beginning with the 2.16 release. Users are strongly encouraged to develop new codes using the standardized interfaces and to migrate existing code to them. The Berkeley-specific 'bupc_' prefixed variants are now officially deprecated in favor of the standardized variant, and will be removed in a future version.

    void upc_all_free(shared void *ptr);
    void upc_all_lock_free(upc_lock_t *lockptr);

The 'bupc_system' function

Berkeley UPC provides 'bupc_system()' as a drop-in replacement for C99 'system()'. This implementation is designed to avoid adverse interactions between fork() and RMDA-capable networking libraries.

Known bugs and limitations

Implicit library calls may modify errno

The value of errno is zero at program startup, but is never set to zero by any library function. The value of errno may be set to nonzero by a library function call whether or not there is an error, provided the use of errno is not documented in the description of the function in this International Standard.

The problem in the context of Berkeley UPC and its source-to-source translation is that there is one copy of errno per UPC thread which is shared by both the generated code representing translated UPC code, and all the runtime libraries running underneath it (including UPCR, GASNet, vendor network libs, etc.). Furthermore, many actions in UPC which do not qualify as library calls at UPC level (e.g. dereferencing a pointer-to-shared) result in library calls within the generated code. Consequently, the value of errno set by a failed library call invoked at the UPC source level may be subsequently overwritten by any of these implicit library calls.

While one could imagine the Berkeley UPC compiler and runtime taking action to preserve the value of errno across all the implicit library calls, doing so would adversely affect performance and we do not currently take this approach. This means that a UPC user who wants to inspect the value of errno after a failed library call they make must do so immediately - not just before the next UPC-level library call, but also before taking any action that might possibly invoke implicit library calls in the generated source code.

Basically, the only 100% safe way for a UPC program to read errno when using Berkeley UPC is to copy it into a local variable immediately after the failed library call returns. This is the "recommended practice" for using errno with Berkeley UPC.

Preprocessor macros defined in UPC files must not affect .h files

    #define NDEBUG
    #include <assert.h>

There is a simple workaround: if you need to define a macro that affects the behavior of #included files, define it on the command line to upcc:

    upcc -DNDEBUG myprogam.upc

Behavior of the 'getenv/setenv' functions

Berkeley UPC guarantees that 'getenv' allows retrieval of certain environment variable values that were present when the job was launched. At present this function is only guaranteed to retrieve these value for all threads if the environment variable's name begins with 'UPC_' or 'GASNET_'. On some platforms all environment variables seen by the job launcher may be propagated, but it is not portable to rely on this.

The 'setenv' and 'unsetenv' functions are not guaranteed to work in a Berkeley UPC runtime environment, and should be avoided.

Correctness when using GCC 4.x (x<3) as the C compiler

1. Try compiling your code in debug mode (ie with 'upcc -g'). If the problem persists, then this issue is *not* the culprit.
2. Try compiling your code using the flag 'upcc -Wc,-fno-strict-aliasing'. If the problem persists, then this issue is *not* the culprit.
3. Run your code several times. If the problem is intermittent, then this issue is probably not the culprit (the optimizer bug is deterministic).

If you still believe you are encountering this issue, there are several recommended workarounds:

1. Configure BUPC to use a different backend C compiler. If you have a non-gcc vendor C compiler available, this may actually be a better choice for performance anyhow. Failing that, using gcc >= 4.3 (or gcc 3.x) should also resolve the issue, as the bug is only believed to be present in gcc 4.0.x through 4.2.x.
2. Build the affected modules using the flag 'upcc -Wc,-fno-strict-aliasing'. This makes the gcc 4.x optimizer more conservative, and also inhibits the illegal optimization.
3. Reconfigure BUPC using 'configure --enable-conservative-local-copy'. This globally activates a more conservative implementation of shared-local accesses that also prevents the illegal optimization.

GUPC+UPCR with -pthreads

    
    UPC Runtime error: pthread_create: Invalid argument

Other known limitations/bugs

• The Berkeley UPC-to-C translator currently does not perform any optimizations on the network traffic of UPC programs--all network accesses are treated as 'strict', and are blocking. Note: we now have some experimental optimizations implemented, but they are not considered production-ready, and so are not enabled by default. Pass the '-opt' flag to upcc to turn on these translator optimizations.

• Shared memory optimizations for inter-thread communcations are used on SMP systems only if you use the '-pthreads' flag to upcc (which requires the use of only thread-safe libraries), or if the runtime has been configured with --enable-pshm (currently the default only on Linux). Support for shared memory communication among (non-pthreaded) UPC processes on an SMP node is available via the "PSHM" feature beginning in the 2.10.0 release (See "INTRA-NODE SHARED MEMORY SUPPORT" in INSTALL.TXT).

• Programs which #include complex.h, and/or tgmath.h do not work on certain platforms.

• Inline assembly statements are not supported within UPC code. However, UPC applications are permitted to #include C-mode header files containing inline assembly statements, written in the syntax appropriate for the backend native C compiler and platform. (A C-mode header file is any header which contains no UPC-specific constructs and includes no files containing UPC-specific constructs.)

• UPC programs are not permitted to register signal handlers (i.e. via signal(2) or sigaction(2)), use the pthread library (ie. pthread_create(3), pthread mutexes), fork child processes (via fork(2)/clone(2)), call setjmp/longjmp, or use System V shared memory constructs.

• UPC programs are recommended to use bupc_system(), rather than the C99 function system().

• UPC programs are recommended to terminate by calling exit(), abort(), upc_global_exit() or returning from main. Calling _exit() could result in the creation of orphaned processes.

• UNIX file descriptors (obtained via open(), fopen(), socket() etc) may not be safely shared across UPC threads.

• 'stdin' may not work on some platforms - programs are recommended to obtain their input via the command line or file system. 'stdout' and 'stderr' are multiplexed on a line-by-line basis to the console, but not ordered in any way between UPC threads (i.e., a single thread's output lines will appear in order, but may be arbitrarily interleaved with the output from other threads)

• The variable-argument list access macro va_arg may be used with pointer-to-shared types, but the pointer type passed to va_arg must match the type of the actual argument (specifically, the shared blocksize must match) and no typedefs may be used in the argument to va_arg.

• Berkeley UPC does not currently support C99-style variable-length arrays (eg a stack temporary array with bounds that are not a compile-time constant). This is planned to be fixed in a future version.

• As documented under Using pthreaded UPC Programs, all C functions called by UPC code must be thread-safe when compiling with the -pthreads flag. Unfortunately, this precludes use of 'getopt()' and related functions for parsing command-line arguments (because it may write to 'argv' when permuting options, and because it uses global state variables). There are two suggested work-arounds for this issue:
  • Perform argument parsing on THREAD 0 and broadcast the results.
  • Serialize the 'getopt()' calls and additionally ensure that the global variable 'optind' is reset to 1 before each call.

• The default flags for optimization by the back-end compiler are typically the most aggressive that do not produce incorrect results on our test suite. This can, in some cases, mean that ``unsafe'' optimizations have been enabled. Currently known instances:
  • IBM XLC: Arithmetic operations involving NaNs and Infinities may not produce correct results by default. Pass '-Wc,-qstrict=ieeefp' to upcc to enforce IEEE-compilant arithmetic.

• Many other bugs with particular code constructs, and/or when using specific architectures/compilers/networks. See our Bugzilla bug database for the grisly details.

Platform-specific issues

Running into Maximum size limits on pinning-based networks

Feedback

Thank you for using Berkeley UPC!

This page last modified on Friday, 28-Oct-2022 15:49:55 PDT