I noticed that Arch always picks up new igraph releases exceptionally fast. Arch Linux - igraph 0.9.0-1 (x86_64) 0.9.0 appeared within hours of release. It also appears that Arch links python-igraph to an external igraph (instead of using a private version). At least pyhon-igraph is listed as a dependent of igraphhere. If so, this quick update has most certainly broken python-igraph in Arch Linux. @iosonofabio can you check if it’s really broken, and if yes, talk to someone at Arch Linux?
This is why I am so strongly against linking high-level interfaces to the C core dynamically.
I’d like to propose that:
By default, the Python interface should link to a guaranteed-to-match the C core statically. @tamas is this already the case? Linking to an external C core dynamically should be an opt-in feature, to be enabled by distro/package maintainers on their own responsibility. They then need to ensure that the two packages always match.
For package management systems that use a rolling release model it is just not feasible not to decouple the high-level igraph interfaces from the C core, otherwise things will break. We don’t have facilities to allow multiple C core versions on the same systems. Thus the issue affects basically all package management systems that are not tied to a Linux distro (MacPorts, Homebrew, vcpkg, etc.) as well as a bunch of Linux distros (notably Arch).
Yes, if you run pip install python-igraph, the source package now contains a vendored copy of a matching release of igraph and the compilation will use that. You need to walk the extra mile to prevent that by downloading the tarball, removing the vendored copy and ensuring that pkg-config finds your desired copy of igraph.
It also seems that the dependencies are left to auto-detection:
This can lead to inconsistent behaviour, e.g. pick up different BLAS/LAPACK implementations depending on what’s installed on the system at build time. For a Linux distro, it’s much better to specify all these explicitly: igraph Reference Manual
First of all, I upgraded my arch to have igraph-0.9.0-1, copy-pasted the igraph_empty.c example, and compiled and linked against the archlinux shared lib. All works fine.
Second, python-igraph is currently at 0.8.3-2 in arch. Again, it seems to work just fine from a superficial level, or is there a particular function you would like to test? The files do not include a static library for the C core, so I think it’s linked against the shared library (presumably 0.9.0).
Third, @szhorvat you seem unfamiliar with the breathing life of a rolling release system. AFAIK arch doesn’t mind too much if things break once in a while because an upstream dev (like us) relies on an old version of <insert your dep here>, in this case the igraph C core. The key determinant of a responsive action is how long this state of suffering is expected to persist. If we expect a new python-igraph release within a few days, I’d say we have three choices:
Do nothing: the imminent release of python-igraph will fix it anyway.
Raise a request against the python-igraph package such that it requires igraph<0.9.0.
Carefully coordinate the release of each igraph and python-igraph API-breaking version with downstream package maintainers, so they are in the know about breaking dep chains beforehand.
I’d go for option 1. Honestly, 2 involves negotiation and takes probably longer than just releasing python-igraph. 3 takes coordination with who knows how many downstream sinks (each distro, homebrew, etc).
Instead of spending time on negotiating with downstream devs, @tamas please let me know if you would like my help to release python-igraph and I’d be happy to help. ATM the release is triggered by the curl script onto github actions → github releases, so one would have to download/reupload those files onto Pypi manually. Let me know if you would like me to do that.
Cheers,
Fabio
edit: @szhorvat perhaps I missed one point. Arch packages are binaries, i.e. they use whatever arpack was on the build server at the time of building. They don’t compile on my laptop. The reason specific versions of arpack and the likes are not present in most arch packages is the risk that every single app dev wants their own super-specific version of arpack because they are too lazy to test their app against the current version. This leads to some inconsistencies, but maintaining five different arpack packages is just too much work for arch devs, so they rather encourage app devs like us to make sure we are compatible with a reasonably modern version of the deps.
A new version of python-igraph 0.9.0 is basically ready for release, I just need to find a time slot to write the changelog. I plan to do it on Monday.
[~]$ ipython
Python 3.9.1 (default, Feb 6 2021, 06:49:13)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import igraph as ig
In [2]: g = ig.Graph.Full(5)
In [3]: g.betweenness()
Out[3]: [0.0, 0.0, 0.0, 0.0, 0.0]
In [4]: g.closeness()
Out[4]: [1.0, 1.0, 1.0, 1.0, 1.0]
In [5]: ig.__version__
Out[5]: '0.8.3'
In [6]:
[~]$ y -Qs igraph
local/igraph 0.9.0-1
A library for creating and manipulating (un)directed graphs.
local/python-igraph 0.8.3-2
Python bindings for the igraph library
[~]$
Seems to work? Well it does not crash, and I think the results are correct? (graph is full so shortest paths never bother passing through you - also distance to everyone is 1, so closeness is also 1).
If it may help, I checked the size of the .so interface file for python-igraph:
-rwxr-xr-x 1 root root 4.3M Nov 13 03:55 igraph/_igraph.cpython-39-x86_64-linux-gnu.so
Is 4.3 MB just the interface or do you think it’s plugging in the whole static igraph? Looks big to me.
@tamas: sounds fabulous! If you push on Monday, arch will have the update a few hours later, nothing to fear.
ugh, and it’s even bundling glpk, lapack and arpack. Not good.
It’s fine (and possible—even preferable) to link to those dynamically from python-igraph. In fact, for 0.9 at least, that should happen by default.
What we need in order for things not to break (without us having to be too constrained) is that python-igraph should link statically to igraph (not to link statically to igraph’s further dependencies).
@iosonofabio Did Arch stop updating igraph or does 0.9.3 just not show on this page for some reason? In the past they have always been the first to update. Arch Linux - igraph 0.9.2-1 (x86_64)
The reason why we link statically to igraph and bundle everything is because we also need to build Python wheels for publication on PyPI, and wheels should not have any additional dependencies (otherwise people will complain why pip3 install python-igraph does not just Do The Right Thing™.
So it looks like the distros should be able to customize how python-igraph is built if they want to ship python-igraph as a package. I’m not sure whether bundling python-igraph as a package has any real benefits now that we have properly built Python wheels – as far as I know, no other packages depend on python-igraph (it is a leaf in the dependency graph), and for most people typing pip3 install python-igraph is just as easy as installing from a package manager, with the additional benefit that it also works for Python virtualenvs.