Flink Streaming - sser-defined data source

[normanSp]

Hey,
In the given streaming examples the test data is always (as far as I
understand) generated in the Flink application.
I have an application that generates test data and sends it to a
specific port.
My question: can I specify a StreamExecutionEnvironment that listens on
that port?
In the docs for streaming
(https://github.com/mbalassi/incubator-flink/blob/streaming-new/docs/streaming_guide.md#basics)
in the Kafka, Flume, ... section it is possible to listen on a specific
port. But how can I define a custom data source on a port?

greetz norman

[Gyula Fóra]

Hey,
You can add any user defined source functionality by using the .addSource
method of the environment.

You can pass a SourceFunction implementation that does whatever you want.
Take a look at the kafka or other sourcefunction for implementation ideas.

Regards,
Gyula
2014.09.20. 9:49 ezt írta ("Norman Spangenberg" <
[hidden email]>):

> Hey,
> In the given streaming examples the test data is always (as far as I
> understand) generated in the Flink application.
> I have an application that generates test data and sends it to a specific
> port.
> My question: can I specify a StreamExecutionEnvironment that listens on
> that port?
> In the docs for streaming (https://github.com/mbalassi/
> incubator-flink/blob/streaming-new/docs/streaming_guide.md#basics) in the
> Kafka, Flume, ... section it is possible to listen on a specific port. But
> how can I define a custom data source on a port?
>
> greetz norman
>
>
>

[Márton Balassi-3]

Hey,

Thanks for checking out the streaming package by the way. :)
As for streaming the examples need an update to match the current state of
the standard of the core API - that is coming soon and is one of the
reasons while I have not merged the pull request yet.

Would you expect another pre-implemented source function then? Like the
socketTextStream (
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala#L275-281)
that Spark Streaming provides?


On Sat, Sep 20, 2014 at 9:48 AM, Norman Spangenberg <
[hidden email]> wrote:

> Hey,
> In the given streaming examples the test data is always (as far as I
> understand) generated in the Flink application.
> I have an application that generates test data and sends it to a specific
> port.
> My question: can I specify a StreamExecutionEnvironment that listens on
> that port?
> In the docs for streaming (https://github.com/mbalassi/
> incubator-flink/blob/streaming-new/docs/streaming_guide.md#basics) in the
> Kafka, Flume, ... section it is possible to listen on a specific port. But
> how can I define a custom data source on a port?
>
> greetz norman
>
>
>