using MKL in cython fails with undefined symbol in shared object from MKL, but direct execution is fine - cython

Question

I'm writting a physics simulation in C using Intel Math Kernel Library (MKL) and want to call it directly from python code using cython.
The cython compilation itself works (program runs error free if MKL is not included in the example) and if I compile my C code directly in gcc with
gcc -O3 -Wall -m64 -I"${MKLROOT}/include" bar.c -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl
it works well. The compiler flags are generated by the MKL Link line Advisor.
But If I now try to compile the same code with cython, I'm getting the error message
INTEL MKL ERROR: /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_avx2.so.1: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so.1 or libmkl_def.so.1.
I've also tried to compile my program (without the main) into a shared library (.so) to get the MKL linking done outside of cython and then just link against that, but after fiddling that library into LD_LIBRARY_PATH I just got the same behavior again.
Any idea how I can get the linking correct?
I can swap out the used cblas_ function for a different one (e.g. cblas_drot and some more tried) and get the same error.
I've read quite some other questions (many concerning MKL within anaconda, my MKL is installed manually in /opt as the path above shows), including this which tries to use MKL in java and got the same error.
I can reproduce the same results regarding the nm statements (undefined in libmkl_avx2.so.1, but defined in libmkl_gnu_thread.so), but I failed to apply the answer from that question to my problem. If I try to add -lmkl_gnu_thread in the below shown setup.py script, I get different unfulfilled dependencies, fixed by also including -fopenmp to get back to the old error...
further information and used files
LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/compiler/2021.1.1/linux/lib:/opt/intel/oneapi/compiler/2021.1.1/linux/lib/x64:/opt/intel/oneapi/compiler/2021.1.1/linux/lib/emu:/opt/intel/oneapi/compiler/2021.1.1/linux/compiler/lib/intel64_lin:/opt/intel/oneapi/compiler/2021.1.1/linux/compiler/lib:/opt/intel/oneapi/tbb/2021.1.1/env/../lib/intel64/gcc4.8 (set by the mkl setvars script, obtained by echo $LD_LIBRARY_PATH)
does python maybe do sth weird with this variable?
readelf -d $MKLROOT/lib/intel64/libmkl_avx2.so.1 | grep NEEDED 0x0000000000000001 (NEEDED) returns Shared library: [libdl.so.2], indicating that it claims to only depend on libdl (I don't find the question where I got that command from, but it was also to this topic, just the .1 was missing in the filename)
used files
setup.py
(contains a second version as comamnd, can be run like this)
from Cython.Distutils import build_ext
#from Cython.Build import cythonize
from distutils.extension import Extension
from distutils.core import setup
import numpy
extensions = [
Extension("foo", ["foo.pyx"],
include_dirs=[numpy.get_include()],
extra_compile_args=["-Wall", "-m64", "-I\"${MKLROOT}/include\""],
extra_link_args=["-fopenmp", "-L${MKLROOT}/lib/intel64", "-Wl,--no-as-needed", "-lmkl_gnu_thread", "-lmkl_intel_lp64",
"-lmkl_sequential", "-lmkl_core", "-lpthread", "-lm", "-ldl"])
]
for e in extensions:
e.cython_directives = {'language_level': "3"} #all are Python-3
# produce the same behaviour, first:
setup(ext_modules=cythonize(extensions))
#second
#setup(ext_modules=cythonize(extensions),
# cmdclass = {'build_ext':build_ext})
used with
python setup.py build_ext --inplace
EDIT another version with better usage of keywords, according to comments. It gives the same error.
from Cython.Build import cythonize
from distutils.extension import Extension
from distutils.core import setup
import numpy
extensions = [
Extension("foo", ["foo.pyx"],
include_dirs=[numpy.get_include(), "\"${MKLROOT}/include\""],
libraries=["mkl_intel_lp64", "mkl_sequential", "mkl_core", "pthread", "m", "dl"],
library_dirs=["${MKLROOT}/lib/intel64"],
extra_compile_args=["-Wall", "-m64"],
extra_link_args=["-Wl,--no-as-needed", ])
]
for e in extensions:
e.cython_directives = {'language_level': "3"} #all are Python-3
setup(ext_modules=cythonize(extensions))
foo.pyx
cimport numpy as np
import numpy as np
import ctypes
cdef extern from "bar.c":
void double_elements(int n, double* vec_y)
def func(np.ndarray[np.double_t, ndim=1] y not None):
double_elements(<int> y.size//2, <double*> <size_t> y.__array_interface__['data'][0])
return y
bar.c
#include <mkl.h>
#include <stdio.h>
void double_elements(int n,
double* x) {
cblas_dscal(n, 2., x, 1);
}
#ifndef PY_VERSION_HEX // compile the main only, if not using cython
int main() {
double x[2] = {1., 2.};
double_elements(2, x);
printf("%g %g\n", x[0], x[1]);
return 0;
}
#endif
run.py
(used for testing, quite boring and only calls the function)
import numpy as np
import foo
x = np.array([1., 2.])
y = foo.func(x)
print(x)
print(y)

Answer 1

I've solved the problem by using static linking of MKL instead. Because I also failed to apply all the commands from static linking (it uses groups) in the setup.py script, I've switched to compiling it entirely on my own with
gcc -O3 -Wall -m64 -I"${MKLROOT}/include" -c bar.c -o build/bar.o
cythonize foo.pyx
# exchange /path/to/numpy/ by value given by numpy.get_include()
gcc -Wall -O2 -fstack-protector-strong -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -fPIC -I/path/to/numpy/core/include -I/usr/include/python3.8 -c foo.c -o build/foo.o -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION
gcc -shared build/foo.o build/bar.o -o foo.cpython-38-x86_64-linux-gnu.so -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_sequential.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl
This corresponds to the compile steps done by the setup script (without checking the change dates) and with some compile flags removed (it used many duplicates).