Prevent double compilation of c files in cython - cython

Question

I am writing a wrapper over c libriary and this lib has file with almost all functions, let say, all_funcs.c. This file in turn requires compilation of lots of another c files
I have created all_funcs.pyx, where I wraped all functions, but I also want to create a submodule, that has access to functions from all_funcs.c. What works for now is adding all c-files to both Extensions in setup.py, however each c-file compiles twice: first for all_funcs.pyx and second for submodule extension.
Are there any ways to provide common sourse files to each Extension?
Example of current setup.py:
ext_helpers = Extension(name=SRC_DIR + '.wrapper.utils.helpers',
sources=[SRC_DIR + '/wrapper/utils/helpers.pyx'] + source_files_paths,
include_dirs=[SRC_DIR + '/include/'])
ext_all_funcs = Extension(name=SRC_DIR + '.wrapper.all_funcs',
sources=[SRC_DIR + '/wrapper/all_funcs.pyx'] + source_files_paths,
include_dirs=[SRC_DIR + '/include/'])
EXTENSIONS = [
ext_helpers,
ext_all_funcs,
]
if __name__ == "__main__":
setup(
packages=PACKAGES,
zip_safe=False,
name='some_name',
ext_modules=cythonize(EXTENSIONS, language_level=3)
)
source_files_paths - the list with common c source files

Answer 1

Note: this answer only explains how to avoid multiple compilation of c/cpp-files using libraries-argument of setup-function. It doesn't however explain how to avoid possible problems due to ODR-violation - for that see this SO-post.
Adding libraries-argument to setup will trigger build_clib prior to building of ext_modules (when running setup.py build or setup.py install commands), the resulting static library will also be automatically passed to the linker, when extensions are linked.
For your setup.py, this means:
from setuptools import setup, find_packages, Extension
...
#common c files compiled to a static library:
mylib = ('mylib', {'sources': source_files_paths}) # possible further settings
# no common c-files (taken care of in mylib):
ext_helpers = Extension(name=SRC_DIR + '.wrapper.utils.helpers',
sources=[SRC_DIR + '/wrapper/utils/helpers.pyx'],
include_dirs=[SRC_DIR + '/include/'])
# no common c-files (taken care of in mylib):
ext_all_funcs = Extension(name=SRC_DIR + '.wrapper.all_funcs',
sources=[SRC_DIR + '/wrapper/all_funcs.pyx'],
include_dirs=[SRC_DIR + '/include/'])
EXTENSIONS = [
ext_helpers,
ext_all_funcs,
]
if __name__ == "__main__":
setup(
packages=find_packages(where=SRC_DIR),
zip_safe=False,
name='some_name',
ext_modules=cythonize(EXTENSIONS, language_level=3),
# will be build as static libraries and automatically passed to linker:
libraries = [mylib]
)
To build the extensions inplace one should invoke:
python setupy.py build_clib build_ext --inplace
as build_ext alone is not enough: we need the static libraries to build before they can be used in extensions.