Why does adding an __init__.py change Cython build_ext --inplace behavior? - cython

Question

This is a toy version of a problem I have. I'm trying to use setuptools to compile some Cython code in place, as part of a larger project.
My topdir is test. It contains:
hello_world.pyx
def say_hi():
print('meh')
and setup.py
from setuptools import setup
from Cython.Build import cythonize
setup(
ext_modules=cythonize("hello.pyx"),
)
If I build these modules in place with python3 setup.py build_ext --inplace, everything works as expected.
But if add an empty __init__.py to my topdir, I get the following error:
copying build/lib.linux-x86_64-3.8/Code/test/hello.cpython-38-x86_64-linux-gnu.so -> Code/test
error: could not create 'Code/test/hello.cpython-38-x86_64-linux-gnu.so': No such file or directory
Why is this happening? And is there a standard way of fixing it? I would like for these compiled Cython modules to live alongside some other C/Python code which is still in development.

Answer 1

what is happening?
The complete description of what is happening can be found in this GitHub issue: https://github.com/pypa/setuptools/issues/3343#issuecomment-1143527672
TL;DR: When you add a __init__.py file in your top folder, it interferes with the way Cython computes the extension's name. This results in an incorrect name (incompatible with your project layout due to a missing folder) and causes setuptools to crash.
And is there a standard way of fixing it?
If you really want to keep the __init__.py file under your top-level project folder[^1], you can explicitly use Extension objects with the proper/correct name instead of relying on the automatic naming algorithm used by Cython:
from setuptools import Extension, setup
from Cython.Build import cythonize
extensions = [Extension("hello", ["hello.pyx"])]
setup(
ext_modules=cythonize(extensions),
)
[^1] Please note that this is not common practice in the Python packaging ecosystem.