Introduction
The purpose of the Graph Streaming API is to build a unified framework for streaming graph objects. Gephi’s data structure and visualization engine has been built with the idea that a graph is not static and might change continuously. By connecting Gephi with external data-sources, we leverage its power to visualize and monitor complex systems or enterprise data in real-time. Moreover, the idea of streaming graph data goes beyond Gephi, and a unified and standardized API could bring interoperability with other available tools for graph and network analysis, as they could start to interoperate with other tools in a distributed and cooperative fashion.
Plugin Owner's Notes
With the increasing level of connectivity and cooperation between systems, for a system that aim to be interoperable, it is imperative to comply with the available standards. Graph objects are abstractions that can represent a wide range of real-world structures, from computer networks to human interactions, and there are a lot of standards to exchange graph data in different formats, from text-based formats to xml-based formats. But the real-world structures are constantly changing, and the current formats are not suitable to exchange such type of dynamic data.
A lot of well-established systems already stream data to its users using a streaming API. Twitter for example defined a Streaming API to allow near-realtime access to its data. They are using two different formats: XML and JSON, but JSON is strongly encouraged over XML, as JSON is more compact and parsing is greatly simplified.
We are not the first to implement a Graph Streaming API, and another very interesting experience is the GraphStream Java Library. It is composed of an API that gives a way to add edges and nodes in a graph and make them evolve. The graphs are composed of nodes and edges that can appear, disappear or be modified, and these operations are called events. The sequence of operations that occur in a graph is seen as a stream of events.
So, as other people already had successful experiences with graph streaming, why not start our work based on these experiences? That’s what we are doing, and beyond finding these experiences very useful, we are also trying to be compatible with the available work. This Gephi Graph Streaming release is using two formats: JSON for flexibility, and a text-based format, based in the GraphStream implementation.
To illustrate how simple it will be to connect to a master, this video shows Gephi connecting to a master and visualizing the received graph data in real time. The graph in this demo is a part of the Amazon.com library, where the nodes represent books and the edges represent their similarities. For each book, a node is added, the similar books are explored, adding the similar ones as nodes and the similarity as an edge.
The Graph Streaming specification goes beyond the simple fact that a client can pull data from a master: in fact, clients can interact with the master pushing data to it, in a REST architecture. The same data format used by the master to send graph events to the clients is used by clients to interact with the master.
But what about connecting two different Gephi instances together? One instance will be master, and the other client. Using the Graph Streaming API, a change in a graph at the master’s workspace should cause a change in the client’s workspace, and a change at the client’s workspace will cause it to send requests to the master to update its graph accordingly. Both instances working in a distributed mode. In fact, different people could work in a distributed mode to construct a graph: it’s the Collaborative Graph Construction.
Sources
Source code is hosted on Launchpad.
30 March 2011 at 6:00 pm | Permalink
Current specifications: http://wiki.gephi.org/index.php/Specification_-_GSoC_Graph_Streaming_API
8 October 2011 at 9:07 pm | Permalink
Dear Sebastien, this is really cool stuff!
I would like to know, however, how to use this feature from the Gephi toolkit. I mean this: suppose I do have a java app which produces real-time graph data, I would like to
1) turn my app into a server which broadcast those streaming data
and maybe
2) have another client app which visualizes the graph stream, using the streaming api plugin
If I understand correctly, one can import the plugin into the toolkit, but how to carry out 1) and possibly 2)?
In the current javadoc for the toolkit there is no web exporter, and it is not clear to me in which format I have to export the streaming data so that a client as in 2) or even a gephi server turned into a streaming client can use them
Thanks
6 January 2012 at 8:53 pm | Permalink
What version of your Graph Streaming plugin works with GEPHI .8 and/or .7
10 January 2012 at 4:48 am | Permalink
I would like to work with this plugin. I’d especially like to try to do collaborative graph construction. Would someone please post or e-mail some instructions on how to set it (including server components) up?
I’ll happily write up and contribute comprehensive instructions… once I get it up and running.
Thank you.
10 January 2012 at 11:40 am | Permalink
Hello Mark, to test the GraphStreaming plugin to do collaborative graph construction you can use two Gephi instances.
First, install the plugin in both instances. In the first instance, you configure it as master (go to the Streaming tab and click Server/Start). After, you start the second instance and connect it to the first instance: go to the Streaming tab, add a new connection and paste the URL of your master instance (normally it should be http://your-master-ip:8080/workspace0) in the “Client” / “Source URL” text box and click “Connect”.
There are examples of connecting Gephi to a Python application in the following github repository: https://github.com/panisson/pygephi_graphstreaming
The current plugin version works with Gephi 0.8-beta.
If you have problems contact me.
10 January 2012 at 12:21 pm | Permalink
To Polymathicus,
You can use the Graph Streaming API to connect from the toolkit, without a user interface:
ProjectController projectController = Lookup.getDefault().lookup(ProjectController.class);
Project project = projectController.getCurrentProject();
Workspace workspace = projectController.getCurrentWorkspace();
// Get the graph instance
GraphController graphController = Lookup.getDefault().lookup(GraphController.class);
GraphModel graphModel = graphController.getModel();
Graph graph = graphModel.getHierarchicalMixedGraph();
// Connect to stream using the Streaming API
StreamingController controller = Lookup.getDefault().lookup(StreamingController.class);
StreamingEndpoint endpoint = new StreamingEndpoint();
endpoint.setUrl(new URL(“http://localhost:8080/workspace0″));
endpoint.setStreamType(controller.getStreamType(“JSON”));
StreamingConnection connection = controller.connect(endpoint, graph);
connection.asynchProcess();
And you can also start a master using the API:
StreamingServer server = Lookup.getDefault().lookup(StreamingServer.class);
ServerControllerFactory controllerFactory = Lookup.getDefault().lookup(ServerControllerFactory.class);
ServerController serverController = controllerFactory.createServerController(graph);
String context = “/mycontext”;
server.register(serverController, context);
Using this code, the Gephi master will be accessible in the following url:
http://your_ip_here:8080/mycontext