broadcast variable in spargel

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

broadcast variable in spargel

Martin Neumann
Hej,

I'm using the latest trunk version of Flink and the new JavaAPI.
I'm writing some graph algorithms that need to know the number of nodes in
the graph. To get the number of nodes I run a short count job and then get
a DataSet<Long> that I need to give as input to the other calculations.

It works for normal operations:

nodeList.map(new VertexInit()).withBroadcastSet(numNodes, "#Nodes");


However when I call the Spargelcode I don't get the option to do
.withBroadcastSet

initNodeList.runOperation(VertexCentricIteration.withPlainEdges(edgeList,
new VertexRankUpdater(10, 0.85), new RankMessenger(), 100));


How do I use broadcast variables in Spargel?


cheers Martin
Reply | Threaded
Open this post in threaded view
|

Re: broadcast variable in spargel

Stephan Ewen
Hi Martin!

There is an option to use BC variables in Spargel:

The VertexCentricIteration has two methods "
addBroadcastSetForMessagingFunction"and"addBroadcastSetForUpdateFunction".

Stephan



On Mon, Jul 14, 2014 at 1:23 PM, Martin Neumann <[hidden email]>
wrote:

> Hej,
>
> I'm using the latest trunk version of Flink and the new JavaAPI.
> I'm writing some graph algorithms that need to know the number of nodes in
> the graph. To get the number of nodes I run a short count job and then get
> a DataSet<Long> that I need to give as input to the other calculations.
>
> It works for normal operations:
>
> nodeList.map(new VertexInit()).withBroadcastSet(numNodes, "#Nodes");
>
>
> However when I call the Spargelcode I don't get the option to do
> .withBroadcastSet
>
> initNodeList.runOperation(VertexCentricIteration.withPlainEdges(edgeList,
> new VertexRankUpdater(10, 0.85), new RankMessenger(), 100));
>
>
> How do I use broadcast variables in Spargel?
>
>
> cheers Martin
>