Lapack and Similar Maths Libraries

The NAG maths libraries are described separately at file:///usr/local/shared/nag/NAGdoc/index.html (your browser may be too paranoid to follow that link when it is clicked on, so you may need to cut and paste it.)

The rest of this page describes Lapack, BLAS and FFTW3 libraries, and, at the bottom ScaLAPACK and ARPACK.

Intel Compilers

Lapack, BLAS and FFTW3

Serial: -mkl=sequential
Threaded: -mkl

This option should appear as though it were a library, but should be as above, and not -lmkl. It uses Intel's Maths Kernel Library, which is a superset of Lapack, BLAS and FFTW3.

AMD's ACML libraries are also installed for this compiler, and can be used as
Serial: -lacml
Threaded: -openmp and -lacml_mp
They are expected to be significantly slower than Intel's libraries, but they may have different bugs. The threaded version may require the stack limit to be raised significantly.

gfortran

BLAS

-lblas

Lapack

-llapack -lblas

These use the Lapack and BLAS libraries as shipped with Ubuntu. These are based on OpenBLAS, and are reasonably well optimised.

Open64

Lapack and BLAS

Serial: -lacml
Threaded: -mp and -lacml_mp

This uses AMD's ACML library.

Benchmark Comparison

Runs on a 3.1GHz quad core Haswell processor, using dgsev on a 5,000x5,000 matrix.

MKL, threaded	154GFLOPS
MKL, serial	39GFLOPS
-llapack	3GFLOPS
ACML 5.3.1	fails

Runs on a 3.2GHz quad core Sandy Bridge processor, using dgsev on a 5,000x5,000 matrix.

MKL, threaded	79GFLOPS
ACML, threaded	32GFLOPS
MKL, serial	22GFLOPS
ACML, serial	9GFLOPS
-llapack	3GFLOPS

As can be seen, the correct choice of library can make a large difference to performance.

Licences

ACML and MKL are not free in the Gnu sense. No statement is made here as to whether it is permitted to redistribute binaries linked against these libraries.

ScaLAPACK

ScaLAPACK is a version of Lapack for distributed memory computers. It is of little use in TCM, for generally the threaded versions of Lapack will be faster on single multicore computers, and are much easier to use. However, for convenience there are some versions of ScaLAPACK in /usr/local/shared/ScaLAPACK. Note that one must use the same compiler and MPI version that these were built with, and one will also need to link with Lapack.

ARPACK

ARPACK is a well-known library for finding eigenvalues of sparse matrices. For those who want a centrally-installed copy, something can be found in /rscratch/Apps/ARPACK/, with some sparse compilation instructions in the README.txt files in the subdirectories thereof.

MKL by hand

If you really, really, must name MKL libraries explicitly, which I do not recommend, the canonical place to look is:

/usr/local/shared/mkl

As an example:

pc52:~/bench/linpack$ /usr/local/shared/mkl/tools/mkl_link_tool -c gnu_f -libs

       Intel(R) Math Kernel Library (Intel(R) MKL) Link Tool v4.0
       ==========================================================

Output
======

Linking line:
 -L$(MKLROOT)/lib/intel64 -Wl,--no-as-needed  -lmkl_gf_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

pc52:~/bench/linpack$ gfortran dgesv5k_t.f -L/usr/local/shared/mkl/lib -Wl,-rpath,/usr/local/shared/mkl/lib \
  -Wl,--no-as-needed  -lmkl_gf_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm
pc52:~/bench/linpack$ ./a.out
 Please send the results of this run to:

[...]

In order to make this work more easily, links to libiomp5 are included in the above MKL directory, as is a link to the omp.h include file.