Linking python-igraph to an external C/igraph and (too) early igraph ugprades

@iosonofabio You’re an Arch Linux user, right?

I noticed that Arch always picks up new igraph releases exceptionally fast. Arch Linux - igraph 0.9.0-1 (x86_64) 0.9.0 appeared within hours of release. It also appears that Arch links python-igraph to an external igraph (instead of using a private version). At least pyhon-igraph is listed as a dependent of igraph here. If so, this quick update has most certainly broken python-igraph in Arch Linux. @iosonofabio can you check if it’s really broken, and if yes, talk to someone at Arch Linux?

This is why I am so strongly against linking high-level interfaces to the C core dynamically.

I’d like to propose that:

  • By default, the Python interface should link to a guaranteed-to-match the C core statically. @tamas is this already the case? Linking to an external C core dynamically should be an opt-in feature, to be enabled by distro/package maintainers on their own responsibility. They then need to ensure that the two packages always match.
  • These issues should be clearly documented. We do now have a guide for package maintainers.

For package management systems that use a rolling release model it is just not feasible not to decouple the high-level igraph interfaces from the C core, otherwise things will break. We don’t have facilities to allow multiple C core versions on the same systems. Thus the issue affects basically all package management systems that are not tied to a Linux distro (MacPorts, Homebrew, vcpkg, etc.) as well as a bunch of Linux distros (notably Arch).

Opinions?

Yes, if you run pip install python-igraph, the source package now contains a vendored copy of a matching release of igraph and the compilation will use that. You need to walk the extra mile to prevent that by downloading the tarball, removing the vendored copy and ensuring that pkg-config finds your desired copy of igraph.

Then I guess the question is: does the Arch version have its own copy (and the stated dependency on igraph is incorrect), or is it really broken now?

It also seems that the dependencies are left to auto-detection:

This can lead to inconsistent behaviour, e.g. pick up different BLAS/LAPACK implementations depending on what’s installed on the system at build time. For a Linux distro, it’s much better to specify all these explicitly: igraph Reference Manual

Hi guys,

First of all, I upgraded my arch to have igraph-0.9.0-1, copy-pasted the igraph_empty.c example, and compiled and linked against the archlinux shared lib. All works fine.

Second, python-igraph is currently at 0.8.3-2 in arch. Again, it seems to work just fine from a superficial level, or is there a particular function you would like to test? The files do not include a static library for the C core, so I think it’s linked against the shared library (presumably 0.9.0).

Third, @szhorvat you seem unfamiliar with the breathing life of a rolling release system. AFAIK arch doesn’t mind too much if things break once in a while because an upstream dev (like us) relies on an old version of <insert your dep here>, in this case the igraph C core. The key determinant of a responsive action is how long this state of suffering is expected to persist. If we expect a new python-igraph release within a few days, I’d say we have three choices:

  1. Do nothing: the imminent release of python-igraph will fix it anyway.
  2. Raise a request against the python-igraph package such that it requires igraph<0.9.0.
  3. Carefully coordinate the release of each igraph and python-igraph API-breaking version with downstream package maintainers, so they are in the know about breaking dep chains beforehand.

I’d go for option 1. Honestly, 2 involves negotiation and takes probably longer than just releasing python-igraph. 3 takes coordination with who knows how many downstream sinks (each distro, homebrew, etc).

Instead of spending time on negotiating with downstream devs, @tamas please let me know if you would like my help to release python-igraph and I’d be happy to help. ATM the release is triggered by the curl script onto github actions → github releases, so one would have to download/reupload those files onto Pypi manually. Let me know if you would like me to do that.

Cheers,
Fabio

edit: @szhorvat perhaps I missed one point. Arch packages are binaries, i.e. they use whatever arpack was on the build server at the time of building. They don’t compile on my laptop. The reason specific versions of arpack and the likes are not present in most arch packages is the risk that every single app dev wants their own super-specific version of arpack because they are too lazy to test their app against the current version. This leads to some inconsistencies, but maintaining five different arpack packages is just too much work for arch devs, so they rather encourage app devs like us to make sure we are compatible with a reasonably modern version of the deps.

Try a closeness or betweenness calculation. If it’s really linked to 0.9, both will fail.

A new version of python-igraph 0.9.0 is basically ready for release, I just need to find a time slot to write the changelog. I plan to do it on Monday.

@szhorvat

[~]$ ipython
Python 3.9.1 (default, Feb  6 2021, 06:49:13) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import igraph as ig

In [2]: g = ig.Graph.Full(5)

In [3]: g.betweenness()
Out[3]: [0.0, 0.0, 0.0, 0.0, 0.0]

In [4]: g.closeness()
Out[4]: [1.0, 1.0, 1.0, 1.0, 1.0]

In [5]: ig.__version__
Out[5]: '0.8.3'

In [6]:                                                                                                                                          
[~]$ y -Qs igraph
local/igraph 0.9.0-1
    A library for creating and manipulating (un)directed graphs.
local/python-igraph 0.8.3-2
    Python bindings for the igraph library
[~]$ 

Seems to work? Well it does not crash, and I think the results are correct? (graph is full so shortest paths never bother passing through you - also distance to everyone is 1, so closeness is also 1).

If it may help, I checked the size of the .so interface file for python-igraph:

-rwxr-xr-x 1 root root 4.3M Nov 13 03:55 igraph/_igraph.cpython-39-x86_64-linux-gnu.so

Is 4.3 MB just the interface or do you think it’s plugging in the whole static igraph? Looks big to me.

@tamas: sounds fabulous! If you push on Monday, arch will have the update a few hours later, nothing to fear.

Cheers,
Fabio

Then they’re linking statically without realizing it (as it’s the default). The igraph package is not truly a dependency.

If you want, you could check with ldd what it’s linking to, but this is roughly the expected size if it links statically.

I would encourage them to keep linking statically, just remove the igraph dependency.

Agreed, I’ll open a bug for Arch requesting they drop the dependency.

1 Like

https://bugs.archlinux.org/task/69732?string=igraph&project=5&search_name=&type[0]=&sev[0]=&pri[0]=&due[0]=&reported[0]=&cat[0]=&status[0]=open&percent[0]=&opened=&dev=&closed=&duedatefrom=&duedateto=&changedfrom=&changedto=&openedfrom=&openedto=&closedfrom=&closedto=

From comments on the bug report:

ugh, and it’s even bundling glpk, lapack and arpack. Not good.

It’s fine (and possible—even preferable) to link to those dynamically from python-igraph. In fact, for 0.9 at least, that should happen by default.

What we need in order for things not to break (without us having to be too constrained) is that python-igraph should link statically to igraph (not to link statically to igraph’s further dependencies).