Create a Graph in C-API, export for use in Python

1

Hello igraph users,

I’ve been using igraph effectively for visualizing neutrons in a fission chain
(see this tweet of mine https://twitter.com/abnowack/status/1574508829647118337)

I would like to speed this process up. Currently on the C side, I create an igraph_t object from a function such as “igraph_t MyDataAnalysis(MyData *data)”

To expose it to Python I use pybind11 to create a python dictionaries of the vertexes and edges. In a python script, this is wrapped and an igraph.IGraph object is created from the dictionaries.

Instead I would like to operate on the igraph_t object created from my C code directly. My goal is to expose a wrapper around igraph_t to Python, which allows me to use the IGraph python interface as well as additional compiled functions I’ve written

Is this possible? Is there an example? I’m unsure of how the “plumbing” of exposing the igraph_t to act as igraph.Graph would work.

The reason is I am hoping to wrap the igraph_t object for calculating some quantities, and I’d like to create scripts using that on the python side.

2

Hello,

I’m not sure I quite understand your use-case; do you construct your graphs on the C side first and do you want to wrap them in an “official” igraph.Graph object on the Python side?

The constructor on the Python side has a hidden __ptr argument that you can use to create a Graph object backed by an already-constructed igraph_t struct, but note that the constructed Graph object will then take ownership of the graph (you should not call igraph_destroy() on it any more). It’s not tested much as it is not being used any more in the core of the library, but it’s still there, so I’d start with this. Start reading the code of the constructor from here to understand what’s going on behind the scenes.

There’s also the other way round: if you have an igraph.Graph object on the Python side, it has a _raw_pointer hidden property that gives you the address of the wrapped igraph_t object, which you can then cast into an igraph_t* on the C side.

3

Yes, I am constructing a graph on the C side, and want to be able to see them as igraph.Graph in Python. Going the other way as you described will also be useful.

Using the __ptr argument and _raw_pointer property sounds promising, I will try those out!

Thanks

4

I’m able to pass around an igraph_t* to and from Python/C, and I can instantiate an igraph.Graph() from it using the __ptr method. I can call some functions, such as vcount(), ecount() etc, however calling other functions such as print() or add_vertex crashes.

I’m not sure why this is, I’m still stumbling around but posting in case there’s a hint about what’s going on.

This is the relevant portion on the C/C++ side, which uses pybind11.

    m.def("test_c_to_python", []()
          {
            igraph_integer_t num_vertices = 102;
            igraph_integer_t num_edges = 101;
            igraph_real_t diameter;
            igraph_t *g = reinterpret_cast<igraph_t*>(igraph_malloc(sizeof(igraph_t)));

            igraph_rng_seed(igraph_rng_default(), 42);

            igraph_erdos_renyi_game(g, IGRAPH_ERDOS_RENYI_GNM,
                                    num_vertices, num_edges,
                                    IGRAPH_UNDIRECTED, IGRAPH_NO_LOOPS);

            // print igraph vertices and edges
            std::cout << "test_c_to_python" << std::endl;
            std::cout << "vcount: " << igraph_vcount(g) << std::endl;
            std::cout << "ecount: " << igraph_ecount(g) << std::endl;

            auto c = py::capsule(g, "__igraph_t", 0);

            return c;
        });

And using it in Python,

>>> import fermiac, igraph             
>>> 
>>> a = fermiac.test_c_to_python()
test_c_to_python
vcount: 102
ecount: 101
>>>
>>> ga = igraph.Graph(__ptr=a)
>>>
>>> ga.vcount()
102
>>>
>>> ga.ecount()
101
>>>
>>> print(ga)

Python then terminates without an error message.

5

After some digging it looks like the attr member of my created igraph_t object is uninitialized, and that is causing at least the print statement to crash.

So my question now is simpler.

What initialization steps of an igraph_t object are expected to have been done, when setting to an igraph.Graph object in Python?

I need some kind of vertex attributes, would I need edge and graph attributes? What functions should I be using?

6

Spent more time reading about attributes and handlers. My understanding is that the C and Python interfaces have their own separate handlers for attributes, and they are not compatible.

I’m not sure it’s possible to do what I had planned. To create an igraph_t object in C, with attributes and all, and be able to pass the pointer to this object to Python and igraph.Graph(__ptr) and use the igraph_t as an igraph.Graph object.

I’m also assuming it’s not possible to go the other way, to pass an igraph.Graph object from Python into a C function accepting an igraph_t and perform operations with attributes as well.

7

Yes, at some point you need to decide whether you want to initialize the attribute pointer (void* attr) in a way that I think I would at least need to export a function from the Python interface that initializes the attribute handlers for an already-constructed graph in a way that is suitable for the Python interface. Would that solve the problem for you (at least partially)?

8

I think that would partially solve my issue, yes. Ideally I’d like an igraph.Graph object to handle an igraph_t with an initialized attribute pointer. This can be done using a function from the Python interface, as long as I can still manipulate them within C.

9

I was going to create a new thread, however, I found this one. And it’s identical to my question at least the end of it.
First of all is there any updates to this? Can we use the python attributes in C or vice-versa even if its just for the native C-type’s like int or float?

Assuming it is still not possible - I see the attributes.h and attributes.c in the _igraph folder in the python-igraph repo here attributes.c. So my question is, can I use this instead of the c-igraph in my C code to get the attribute values?

If even that’s not possible, can you point me towards the code/doc where it shows how certain python side functions such as strength, distance that use edge attribute pass it to its C equivalent to get the results? So that I can try to do that, thanks!

10

No, there aren’t. The attribute handler API is undergoing a rewrite now in preparation for version 1.0 of the C core, but this use-case is not a priority at the moment. If I understand correctly, this can potentially be solved by calling into Python from C and asking Python to construct a Graph object, then extracting the igraph_t object wrapped in the Graph object and modifying that. Basically, whenever you want to modify the attributes from C, you need to go through Python, but it’s doable.

If this is not suitable for you, please describe your use-case in more detail so we can understandi t better.

So my question is, can I use this instead of the c-igraph in my C code to get the attribute values?

I don’t think so. Maybe. The Python attribute handler starts by constructing three Python dictionaries to hold the graph, vertex and edge attributes, using PyDict_New(). PyDict_New() might assume that there is a Python interpreter running in the context of the process and/or it might want to reach out to the Python garbage collector, so trying to strip out the Python attribute handler from the context of the Python interpreter where it is assumed to live is risky at least. I wouldn’t pursue this direction if I were you.

can you point me towards the code/doc where it shows how certain python side functions such as strength, distance that use edge attribute pass it to its C equivalent to get the results?

The implementations of these functions are in src/_igraph/graphobject.c in the source code of the Python interface. Try looking up igraphmodule_Graph_strength() for instance. The weights argument is converted by calling igraphmodule_attrib_to_vector_t(), which is a generic conversion function in src/_igraph/convert.c. All the other conversion functions are also there.

11

Thanks for your detailed reply and sorry for my delayed response.

Simplified version of my use case:

  1. Program starts in Python - creates a Python igraph.
  2. Calls my c-function in a .so file using Python ctypes, which performs custom graph operations (with edge weights) and returns the result to python.

I want to do these operations on c to get better performance in general but also to utilize multi-threading easily. So, the problem with the first option and the last option is that I have to depend on CPython and PyDicts/PyObjects to get the edge weight values in c. I want to avoid using CPython in my c-side of the code and not mess with GIL lock or reference counting.

My current working solution is somewhat similar to the one in igraphmodule_Graph_strength. But rather than using CPython to get values from Pydict, I copy the edge weights and other additional information about the graph (a minimized CSR) in Python to numpy arrays and use the numpy’s builtin ndpointers to send the array pointers to use in my c-function. This works and I don’t have to worry about ref counting or anything, but it essentially doubles the memory usage and also I am not able to use the c-igraph modules from c.

As you said this is not a priority and I get it. But I want to make the following suggestion/request anyway so that if possible you can consider it in the future.

  • From my understanding the core data that is shared between Python and c versions of igraphs is an unweighted graph.
  • So my suggestion is: What if there is also a separate class for weighted graphs with the weights (int or float) stored in c?
  • This would mostly solve my issue as I currently only need edge weights on my C-code, but on a broader note, I would assume a weighted graph is a common enough use case that will benefit many. And as an added benefit, for functions such as strength, and distances there is no need to copy/retrieve the weights again and again.