FlexiBLAS

Guidance for using FlexiBLAS on ARC HPC systems, including backend selection, compiling and linking applications, checking runtime linkage, and setting thread counts for Slurm jobs.

What FlexiBLAS Does

FlexiBLAS is a wrapper library for BLAS and LAPACK. Applications link against FlexiBLAS, and FlexiBLAS routes the mathematical calls to a selected backend implementation.

This allows a user or software maintainer to change the active BLAS/LAPACK backend without rebuilding the application. For example, the same application can use OpenBLAS, Intel MKL, BLIS, or the Netlib reference implementation, depending on what is available in the FlexiBLAS configuration.

Application
  -> FlexiBLAS
      -> selected BLAS/LAPACK backend

The main benefit is flexibility. A single build can be tested with multiple math libraries, which can be useful for performance comparisons, numerical checks, or software support.

Loading FlexiBLAS

To load FlexiBLAS, use the module command:

$ module load flexiblas

To load a specific version:

$ module load flexiblas/3.5.0

To check which modules are currently loaded:

$ module list

To inspect what the module sets in your environment:

$ module show flexiblas

Available Backends

To list the FlexiBLAS backends available in your environment:

$ flexiblas list

For 3.5.0, the output will show

NETLIB
MKLSERIAL
OPENBLASOPENMP
NVBLAS13
NVBLAS12

The exact backend list depends on how FlexiBLAS was installed and which math libraries are available on the system.

Current Backend

To check the current FlexiBLAS backend:

$ flexiblas current

If the backend is not what you expected, check your loaded modules and FlexiBLAS configuration files.

Selecting a Backend

The active backend can be selected with the FlexiBLAS command line tool or through configuration, depending on the system setup.

To set the backend for the current user:

$ flexiblas set OPENBLASOPENMP

To switch to another available backend:

$ flexiblas set NETLIB

After changing the backend, verify the result:

$ flexiblas current

Backend names are case-sensitive on some systems. Use the exact name reported by flexiblas list.

You can also set the FLEXIBLAS environment variable before calling a program as follows:

FLEXIBLAS=MKLSERIAL /path/to/my/program

Compiling and Linking

Applications that use BLAS or LAPACK can be linked against FlexiBLAS instead of linking directly against a specific backend.

Example C Program

Create a file named dgemm_test.c:

#include <stdio.h>

extern void dgemm_(char *transa, char *transb,
                   int *m, int *n, int *k,
                   double *alpha,
                   double *a, int *lda,
                   double *b, int *ldb,
                   double *beta,
                   double *c, int *ldc);

int main(void) {
    int m = 2, n = 2, k = 2;
    int lda = 2, ldb = 2, ldc = 2;
    double alpha = 1.0, beta = 0.0;
    char trans = 'N';

    double A[4] = {1.0, 3.0, 2.0, 4.0};
    double B[4] = {5.0, 7.0, 6.0, 8.0};
    double C[4] = {0.0, 0.0, 0.0, 0.0};

    dgemm_(&trans, &trans, &m, &n, &k,
           &alpha, A, &lda, B, &ldb,
           &beta, C, &ldc);

    printf("C = [%f %f; %f %f]\n", C[0], C[2], C[1], C[3]);
    return 0;
}

Compile and link with FlexiBLAS:

$ module load flexiblas
$ gcc dgemm_test.c -o dgemm_test -lflexiblas

Run the executable:

$ ./dgemm_test

Using pkg-config

If the FlexiBLAS module provides a pkg-config file, you may be able to compile with:

$ gcc dgemm_test.c -o dgemm_test $(pkg-config --cflags --libs flexiblas)

If pkg-config does not find FlexiBLAS, use the compiler and linker flags provided by the local module environment.

Checking Linkage

To check whether an executable is linked against FlexiBLAS:

$ ldd ./dgemm_test | grep -i flexiblas

The output should include libflexiblas. For more detailed symbol information, use:

$ objdump -T ./dgemm_test | grep -i dgemm

To inspect dynamic library binding during runtime, use:

$ LD_DEBUG=libs ./dgemm_test

For very verbose symbol binding output:

$ LD_DEBUG=bindings ./dgemm_test

Thread Settings

Many BLAS backends are multithreaded. On a shared HPC system, match the number of backend threads to the number of CPU cores requested from Slurm.

For OpenBLAS:

$ export OPENBLAS_NUM_THREADS=$SLURM_CPUS_PER_TASK

For Intel MKL:

$ export MKL_NUM_THREADS=$SLURM_CPUS_PER_TASK

For OpenMP-based backends:

$ export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

Using more threads than requested can oversubscribe the node and reduce performance for both your job and other jobs on the system.

Example Slurm Job

#!/bin/bash
#SBATCH --job-name=flexiblas-test
#SBATCH --cpus-per-task=8
#SBATCH --mem=8G
#SBATCH --time=00:10:00

module load flexiblas

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OPENBLAS_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MKL_NUM_THREADS=$SLURM_CPUS_PER_TASK

flexiblas list
flexiblas current

./dgemm_test

Troubleshooting

When reporting a FlexiBLAS issue, include the loaded modules, the active backend, and the way the application was linked.

$ module list
$ which flexiblas
$ flexiblas list
$ flexiblas current
$ ldd ./dgemm_test | grep -i "blas\|lapack\|flexiblas"

Also include relevant thread settings:

$ echo $OMP_NUM_THREADS
$ echo $OPENBLAS_NUM_THREADS
$ echo $MKL_NUM_THREADS