[Gephi-plugins] Plugin for realtime graph generation based on web browsing

T B phreakocious at gmail.com
Fri Dec 10 16:21:01 CET 2010


Sure.. The largest nodes represent clients.  If you are the only one
pointing at the proxy, you'll only have one client node.  The smallest nodes
represent resources (aka URLs or URIs) (html/css, images, etc.)

When your browser makes a request, a URI node is created for each resource,
and an edge drawn from the client to that resource.  Also, if the request
specifies a Referring page, a URI node is created for the referer and an
edge drawn from the referer to that resource.  This allows the linking
structure between pages to be revealed.

Host nodes are created for the hosts that serve the resources, with the
appropriate edges, and finally, domain nodes tie the hosts together.

Each domain gets a color (up to 32) and the other nodes should get the
domain color.

NBM updated with some more bug fixes:
http://phreakocious.net/dev/net-phreakocious-httpgraph.nbm

Enjoiez!

On Fri, Dec 10, 2010 at 3:23 AM, Mathieu Bastian
<mathieu.bastian at gmail.com>wrote:

> Cool I tested it and its working good :-)
>
> Some questions,
> - could you explain what are the differents nodes, some of them have URLs,
> some not
> - what represent a link between two nodes
> - does the node color mean something?
>
>
> On Wed, Dec 8, 2010 at 8:25 PM, T B <phreakocious at gmail.com> wrote:
>
>> Switched from LittleProxy to Membrane:
>> http://www.membrane-soa.org/soap-monitor/  Haven't had any issues..
>>
>> Launchpad updated, new NBM at
>> http://phreakocious.net/dev/net-phreakocious-httpgraph.nbm
>>
>> Looking forward to hearing any feedback.
>>
>>
>>
>> On Tue, Dec 7, 2010 at 11:28 PM, T B <phreakocious at gmail.com> wrote:
>>
>>> Thanks, it has been stretching my Java abilities a bit.  I wasn't able to
>>> rewrite it with Import as you suggest, but was able to get something mostly
>>> functional from the advice.  This is still a Generator, but it stays alive
>>> in the background updating the workspace until the app is closed.  All you
>>> have to do is (install plugin), pick New Project, File/Generate/HttpGraph,
>>> set your browser's proxy for 127.0.0.0:8088 (or whatever port), and
>>> browse the web.
>>>
>>> Some issues:
>>> Strange nodes with incomplete information
>>> Nodes without edges??
>>> Highly complex web sites (Google maps, for example) will bring everything
>>> to a grinding halt
>>>
>>> Code available at: lp:~phreakocious/gephi/httpgraph
>>> NBM at: http://phreakocious.net/dev/net-phreakocious-httpgraph.nbm
>>>
>>> Merci beaucoup!
>>>
>>>
>>>
>>> On Mon, Dec 6, 2010 at 11:43 PM, Mathieu Bastian <
>>> mathieu.bastian at gmail.com> wrote:
>>>
>>>> That is cool stuff!
>>>>
>>>> Yeah Generator is not the best place for doing that. I would recommend
>>>> to use Import directly, but woithout UI, to discard displaying the import
>>>> report. As you were trying to do this with Generator you are probably
>>>> already familiar with the container. Here is how to create, fill and append
>>>> a container into the current workspace.
>>>>
>>>> //Get Currennt workspace
>>>> ProjectController pc =
>>>> Lookup.getDefault().lookup(ProjectController.class);
>>>> Workspace workspace = pc.getCurrentWorkspace();
>>>>
>>>> //Get container
>>>> Container container =
>>>> Lookup.getDefault().lookup(ContainerFactory.class).newContainer();
>>>>
>>>> //Append container to current graph
>>>> ImportController importController =
>>>> Lookup.getDefault().lookup(ImportController.class);
>>>> importController.process(container, new AppendProcessor(), workspace);
>>>>
>>>> Mathieu
>>>>
>>>> On Mon, Dec 6, 2010 at 7:24 PM, T B <phreakocious at gmail.com> wrote:
>>>>
>>>>> A bit more information..
>>>>>
>>>>> Adding a 20 second sleep() to the generate() function results in 20
>>>>> seconds of data collected, so this seems like the wrong place for real time
>>>>> updating. =)  Advice would be really helpful at this point.
>>>>>
>>>>> Here's an example from my Perl/Streaming solution of some gephi.organd
>>>>> wiki.gephi.org browsing:
>>>>> http://phreakocious.net/dev/httpgraph.gephi
>>>>> [image: httpgraph.png]
>>>>>
>>>>>
>>>>> On Mon, Dec 6, 2010 at 1:29 PM, T B <phreakocious at gmail.com> wrote:
>>>>>
>>>>>> Gephi devs,
>>>>>>
>>>>>> The idea is to stream data about your web browsing directly into Gephi
>>>>>> in real-time.
>>>>>>
>>>>>> After successfully proving out the concept with a Perl script and the
>>>>>> Streaming plugin, I'm trying to implement it in Java.  I've embedded
>>>>>> LittleProxy (http://www.littleshoot.org/littleproxy/) and extended a
>>>>>> few classes to collect the necessary data and now need to create the
>>>>>> appropriate nodes/edges on the graph.
>>>>>>
>>>>>> Currently trying to do this as a Generator, but running into some
>>>>>> problems.  Because this would run indefinitely in the background, the
>>>>>> controlling thread that updates the graph needs to be around whenever new
>>>>>> updates are available, but this doesn't seem to be the way that the
>>>>>> Generator class works.  Does it make more sense to do this as an Import?
>>>>>>
>>>>>> I'd appreciate any feedback you might have as I'm really close to
>>>>>> having this done!
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> gephi-plugins mailing list
>>>>> gephi-plugins at lists.gephi.org
>>>>> http://gephi.org/mailman/listinfo/gephi-plugins
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> gephi-plugins mailing list
>>>> gephi-plugins at lists.gephi.org
>>>> http://gephi.org/mailman/listinfo/gephi-plugins
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> gephi-plugins mailing list
> gephi-plugins at lists.gephi.org
> http://gephi.org/mailman/listinfo/gephi-plugins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gephi.org/pipermail/gephi-plugins/attachments/20101210/036192c6/attachment.html>


More information about the gephi-plugins mailing list