Ways to make an Edge object pickle-able?

Usage
1

I am trying to parallelize an operation that works on all edges of a graph using a ProcessPoolExecutor from concurrent.futures. The problem is that igraph.Edge objects are not pickle-able.

Is there a way to obtain a copy of an Edge object that is picklable for such a purpose?

2

I guess the problem is that igraph uses a c struct for its graph representation. So that seems hard to serialize by python. Do you want to change the edges using your ProcessPoolExecutor, or do you just want to read them?
Maybe it would also help if you tell us your end goal, maybe there’s some other way around it.

3

No, I just need read-only access to the edges, I’m not changing them. You can see the threaded version of what I want to do here: textnets/network.py at 4846dbe1d3b8593dff41f0f0d47cba5d0511bcfd · jboynyc/textnets · GitHub

4

Maybe you can split up the tasks by just using the edge id’s, and then for each id look up their information like source, target and strength from the graph?

def _edge_significance(edge_id):
    edge = g.es[edge_id]
....

edges = range(0, len(g.es))
with concurrent.futures.ProcessPoolExecutor() as executor:
    out = executor.map(_edge_significance, edges)

seems to work.

Else you’d need to go over all the edges and take their information out, like their vertices and weights and then use your executor on that, but I’m not sure if that’s worth it for performance.

I’m not an expert, but I don’t really see another way, because the unpickleability seems pretty essential to the working of igraph. But if the above solutions don’t work, I can try to ask others :slightly_smiling_face:

1 Like
5

Edge IDs instead of Edge objects! Very elegant. Thanks so much.

1 Like