Introduction to OpenMP¶

OpenMP, short for Open Multi-Processing, was introduced in the 1990s as a standard API to support parallel programming on shared memory systems. It emerged in response to the growing need for simplifying parallel computing across multiple cores, which were becoming more common in high-performance computing systems. The initial OpenMP standard supported parallelization in Fortran and was later extended to support C/C++. OpenMP enabled programmers to leverage multiple processors in a shared-memory environment without drastically modifying their code.

Recent Developments¶

Recent updates to OpenMP have introduced several features to enhance compatibility with modern hardware architectures, especially those leveraging SIMD (Single Instruction, Multiple Data) and heterogeneous computing (such as GPUs). Modern versions of OpenMP now support:

• SIMD constructs: Allow for fine-grained parallelization and data-level parallelism, making it easier to apply OpenMP to SIMD hardware.
• Task-based parallelism: Supports more dynamic and asynchronous parallelism, improving efficiency for complex workflows.
• Support for accelerators: New directives like target enable offloading to accelerators such as GPUs.
• Memory management constructs: Enable better control over memory placement and data sharing, critical in multi-node or GPU environments.

Shared Memory Architecture in OpenMP¶

OpenMP is designed for shared memory architectures, where multiple processing cores can access the same main memory space. In this environment, threads can read from and write to the same memory locations, which enables efficient parallel processing without needing explicit data transfers between threads. However, to ensure data consistency and prevent race conditions, OpenMP provides data-sharing directives (e.g., shared, private) and synchronization constructs (e.g., critical, atomic).

In shared memory, OpenMP’s parallel regions allow threads to execute code blocks concurrently. Here’s a brief overview of the most commonly used data-sharing attributes:

• Shared: Variables are accessible to all threads. Each thread can read and write to the same memory location.
• Private: Each thread has its own instance of the variable, with no sharing across threads.
• Reduction: Combines values from multiple threads (e.g., sum or product) in a safe, consistent manner.

For example, consider the following OpenMP code snippet:

#pragma omp parallel for
for (int i = 0; i < N; ++i) {
    C[i] = A[i] + B[i];
}

In this example, A, B, and C are shared across threads by default, as they are accessed in a read-only or write-only manner by each thread.

Applying OpenMP to SIMD Architectures¶

SIMD architectures perform the same operation on multiple data points simultaneously, ideal for applications with repetitive, uniform operations (e.g., matrix manipulations, vector additions). OpenMP’s support for SIMD architectures enables it to run efficiently on CPUs and accelerators with vector processors. OpenMP provides the #pragma omp simd directive, allowing developers to request SIMD parallelization explicitly.

Consider this OpenMP example for a simple vector addition:

for (int i = 0; i < N; ++i) {
    C[i] = A[i] + B[i];
}

SIMD Optimization with OpenMP¶

When we apply OpenMP to parallelize this loop, we enable work division among multiple threads:

#pragma omp parallel for
for (int i = 0; i < N; ++i) {
    C[i] = A[i] + B[i];
}

This #pragma omp parallel for directive distributes loop iterations across threads, making it well-suited for SIMD architectures. Here, each thread processes its portion in a vectorized fashion, effectively leveraging SIMD processing units.

Data Sharing and Memory Considerations¶

SIMD architectures work efficiently when memory accesses are streamlined. OpenMP’s memory management and data-sharing directives, such as #pragma omp simd, enable programmers to control data sharing among threads to minimize contention and maximize memory throughput.

Utilizing OpenMP in C/C++ and Fortran¶

OpenMP’s simplicity and language interoperability make it highly useful for C, C++, and Fortran. Each language can utilize OpenMP with minimal syntax adjustments but similar directives and shared memory model semantics.

OpenMP in C/C++¶

In C and C++, OpenMP is typically used to parallelize for loops and computationally intensive sections of code. The API can be invoked with #include <omp.h>, and parallel regions are created using #pragma omp parallel. C/C++ developers can also benefit from OpenMP’s reduction clause for operations like summing an array, where each thread accumulates partial results safely:

#include <omp.h>

int main() {
    int N = 1000;
    double sum = 0.0;
    double A[N];

    #pragma omp parallel for reduction(+:sum)
    for (int i = 0; i < N; ++i) {
        sum += A[i];
    }
    return 0;
}

This example shows a parallelized summation where each thread accumulates a part of the sum independently, and OpenMP’s reduction clause efficiently combines these results.

OpenMP in Fortran¶

OpenMP in Fortran uses similar directives but adapts to Fortran syntax. Fortran users benefit from OpenMP because it aligns well with array-based computations common in scientific computing. In addition, adding use omp_lib module provides access to OpenMP functions and constants.

program vector_addition
    use omp_lib
    integer :: i, N
    parameter (N=1000)
    real :: A(N), B(N), C(N)

    !$omp parallel do
    do i = 1, N
        C(i) = A(i) + B(i)
    end do
    !$omp end parallel do
end program vector_addition

Here, the !$omp parallel do directive parallelizes the loop, instructing the compiler to distribute iterations among available threads.

Summary¶

• #pragma omp parallel: Spawns a team of threads that execute the code within the parallel region.
• #pragma omp for: Distributes loop iterations among threads in the parallel team.
• collapse(n): Collapses n nested loops into a single loop, enabling parallel execution across all loops.
• reduction: Specifies how to combine results from each thread safely.
• simd: Optimizes a loop to run with SIMD instructions if supported by the hardware.
• shared and private: Specify whether variables are shared across threads or private to each thread.