Access flink-conf.yaml data

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Access flink-conf.yaml data

Dulaj Viduranga
Hi,
Can someone help me on how to access the flink-conf.yaml configuration values inside the flink sources? Are these readily available as a map somewhere?

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Access flink-conf.yaml data

Chiwan Park
I think that you can use `org.apache.flink.configuration.GlobalConfiguration` to obtain configuration object.

Regards.
Chiwan Park (Sent with iPhone)


> On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]> wrote:
>
> Hi,
> Can someone help me on how to access the flink-conf.yaml configuration values inside the flink sources? Are these readily available as a map somewhere?
>
> Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: Access flink-conf.yaml data

Stephan Ewen
Hey Dulaj!

As Chiwan said, the GlobalConfiguration object is used to load them
initially.

You can always use that to access the values (it works as a singleton
internally) - but we are starting to move away from singletons, as they
make test setups and embedding more difficult.
In the JobManager and TaskManager setup, we pass a Configuration object
around, which has all the values from the global configuration.

Stephan



On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> wrote:

> I think that you can use
> `org.apache.flink.configuration.GlobalConfiguration` to obtain
> configuration object.
>
> Regards.
> Chiwan Park (Sent with iPhone)
>
>
> > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]>
> wrote:
> >
> > Hi,
> > Can someone help me on how to access the flink-conf.yaml configuration
> values inside the flink sources? Are these readily available as a map
> somewhere?
> >
> > Thanks.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Access flink-conf.yaml data

Kirschnick, Johannes
In reply to this post by Dulaj Viduranga
Hi Stephan,

I just came across the same problem in accessing the constants and in particular setting custom properties.
In particular I noticed that the Minicluster started in the Local Environment cannot easily be customized as it does not take into account any custom environment variables - no way to pass them.
I tried to fix that locally and suggested a pull request - does that make sense?
https://github.com/apache/flink/pull/448


Johannes

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Stephan Ewen
Gesendet: Dienstag, 3. März 2015 10:03
An: [hidden email]
Betreff: Re: Access flink-conf.yaml data

Hey Dulaj!

As Chiwan said, the GlobalConfiguration object is used to load them initially.

You can always use that to access the values (it works as a singleton
internally) - but we are starting to move away from singletons, as they make test setups and embedding more difficult.
In the JobManager and TaskManager setup, we pass a Configuration object around, which has all the values from the global configuration.

Stephan



On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> wrote:

> I think that you can use
> `org.apache.flink.configuration.GlobalConfiguration` to obtain
> configuration object.
>
> Regards.
> Chiwan Park (Sent with iPhone)
>
>
> > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]>
> wrote:
> >
> > Hi,
> > Can someone help me on how to access the flink-conf.yaml
> > configuration
> values inside the flink sources? Are these readily available as a map
> somewhere?
> >
> > Thanks.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Access flink-conf.yaml data

Robert Metzger
Hi Johannes,

This change will allow users to pass a custom configuration to the
LocalExecutor: https://github.com/apache/flink/pull/427.
Is that what you're looking for?

On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes <
[hidden email]> wrote:

> Hi Stephan,
>
> I just came across the same problem in accessing the constants and in
> particular setting custom properties.
> In particular I noticed that the Minicluster started in the Local
> Environment cannot easily be customized as it does not take into account
> any custom environment variables - no way to pass them.
> I tried to fix that locally and suggested a pull request - does that make
> sense?
> https://github.com/apache/flink/pull/448
>
>
> Johannes
>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von
> Stephan Ewen
> Gesendet: Dienstag, 3. März 2015 10:03
> An: [hidden email]
> Betreff: Re: Access flink-conf.yaml data
>
> Hey Dulaj!
>
> As Chiwan said, the GlobalConfiguration object is used to load them
> initially.
>
> You can always use that to access the values (it works as a singleton
> internally) - but we are starting to move away from singletons, as they
> make test setups and embedding more difficult.
> In the JobManager and TaskManager setup, we pass a Configuration object
> around, which has all the values from the global configuration.
>
> Stephan
>
>
>
> On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> wrote:
>
> > I think that you can use
> > `org.apache.flink.configuration.GlobalConfiguration` to obtain
> > configuration object.
> >
> > Regards.
> > Chiwan Park (Sent with iPhone)
> >
> >
> > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]>
> > wrote:
> > >
> > > Hi,
> > > Can someone help me on how to access the flink-conf.yaml
> > > configuration
> > values inside the flink sources? Are these readily available as a map
> > somewhere?
> > >
> > > Thanks.
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Access flink-conf.yaml data

Stephan Ewen
I think that #427 is a good way to do this. It bypasses the singleton
GlobalConfiguration that I personally hope to get rid of.

On Wed, Mar 4, 2015 at 11:49 AM, Robert Metzger <[hidden email]> wrote:

> Hi Johannes,
>
> This change will allow users to pass a custom configuration to the
> LocalExecutor: https://github.com/apache/flink/pull/427.
> Is that what you're looking for?
>
> On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes <
> [hidden email]> wrote:
>
> > Hi Stephan,
> >
> > I just came across the same problem in accessing the constants and in
> > particular setting custom properties.
> > In particular I noticed that the Minicluster started in the Local
> > Environment cannot easily be customized as it does not take into account
> > any custom environment variables - no way to pass them.
> > I tried to fix that locally and suggested a pull request - does that make
> > sense?
> > https://github.com/apache/flink/pull/448
> >
> >
> > Johannes
> >
> > -----Ursprüngliche Nachricht-----
> > Von: [hidden email] [mailto:[hidden email]] Im Auftrag von
> > Stephan Ewen
> > Gesendet: Dienstag, 3. März 2015 10:03
> > An: [hidden email]
> > Betreff: Re: Access flink-conf.yaml data
> >
> > Hey Dulaj!
> >
> > As Chiwan said, the GlobalConfiguration object is used to load them
> > initially.
> >
> > You can always use that to access the values (it works as a singleton
> > internally) - but we are starting to move away from singletons, as they
> > make test setups and embedding more difficult.
> > In the JobManager and TaskManager setup, we pass a Configuration object
> > around, which has all the values from the global configuration.
> >
> > Stephan
> >
> >
> >
> > On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]>
> wrote:
> >
> > > I think that you can use
> > > `org.apache.flink.configuration.GlobalConfiguration` to obtain
> > > configuration object.
> > >
> > > Regards.
> > > Chiwan Park (Sent with iPhone)
> > >
> > >
> > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]>
> > > wrote:
> > > >
> > > > Hi,
> > > > Can someone help me on how to access the flink-conf.yaml
> > > > configuration
> > > values inside the flink sources? Are these readily available as a map
> > > somewhere?
> > > >
> > > > Thanks.
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Access flink-conf.yaml data

Kirschnick, Johannes
In reply to this post by Dulaj Viduranga
Good point #427 looks more complete - will wait for the merge and just use my workaround locally in the meantime

Johannes

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Stephan Ewen
Gesendet: Mittwoch, 4. März 2015 12:42
An: [hidden email]
Betreff: Re: Access flink-conf.yaml data

I think that #427 is a good way to do this. It bypasses the singleton GlobalConfiguration that I personally hope to get rid of.

On Wed, Mar 4, 2015 at 11:49 AM, Robert Metzger <[hidden email]> wrote:

> Hi Johannes,
>
> This change will allow users to pass a custom configuration to the
> LocalExecutor: https://github.com/apache/flink/pull/427.
> Is that what you're looking for?
>
> On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes <
> [hidden email]> wrote:
>
> > Hi Stephan,
> >
> > I just came across the same problem in accessing the constants and
> > in particular setting custom properties.
> > In particular I noticed that the Minicluster started in the Local
> > Environment cannot easily be customized as it does not take into
> > account any custom environment variables - no way to pass them.
> > I tried to fix that locally and suggested a pull request - does that
> > make sense?
> > https://github.com/apache/flink/pull/448
> >
> >
> > Johannes
> >
> > -----Ursprüngliche Nachricht-----
> > Von: [hidden email] [mailto:[hidden email]] Im Auftrag
> > von Stephan Ewen
> > Gesendet: Dienstag, 3. März 2015 10:03
> > An: [hidden email]
> > Betreff: Re: Access flink-conf.yaml data
> >
> > Hey Dulaj!
> >
> > As Chiwan said, the GlobalConfiguration object is used to load them
> > initially.
> >
> > You can always use that to access the values (it works as a
> > singleton
> > internally) - but we are starting to move away from singletons, as
> > they make test setups and embedding more difficult.
> > In the JobManager and TaskManager setup, we pass a Configuration
> > object around, which has all the values from the global configuration.
> >
> > Stephan
> >
> >
> >
> > On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]>
> wrote:
> >
> > > I think that you can use
> > > `org.apache.flink.configuration.GlobalConfiguration` to obtain
> > > configuration object.
> > >
> > > Regards.
> > > Chiwan Park (Sent with iPhone)
> > >
> > >
> > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga
> > > > <[hidden email]>
> > > wrote:
> > > >
> > > > Hi,
> > > > Can someone help me on how to access the flink-conf.yaml
> > > > configuration
> > > values inside the flink sources? Are these readily available as a
> > > map somewhere?
> > > >
> > > > Thanks.
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Access flink-conf.yaml data

Robert Metzger
Cool.
I'll merge #427 as soon as https://github.com/apache/flink/pull/410 is in
master.

On Wed, Mar 4, 2015 at 12:48 PM, Kirschnick, Johannes <
[hidden email]> wrote:

> Good point #427 looks more complete - will wait for the merge and just use
> my workaround locally in the meantime
>
> Johannes
>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von
> Stephan Ewen
> Gesendet: Mittwoch, 4. März 2015 12:42
> An: [hidden email]
> Betreff: Re: Access flink-conf.yaml data
>
> I think that #427 is a good way to do this. It bypasses the singleton
> GlobalConfiguration that I personally hope to get rid of.
>
> On Wed, Mar 4, 2015 at 11:49 AM, Robert Metzger <[hidden email]>
> wrote:
>
> > Hi Johannes,
> >
> > This change will allow users to pass a custom configuration to the
> > LocalExecutor: https://github.com/apache/flink/pull/427.
> > Is that what you're looking for?
> >
> > On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes <
> > [hidden email]> wrote:
> >
> > > Hi Stephan,
> > >
> > > I just came across the same problem in accessing the constants and
> > > in particular setting custom properties.
> > > In particular I noticed that the Minicluster started in the Local
> > > Environment cannot easily be customized as it does not take into
> > > account any custom environment variables - no way to pass them.
> > > I tried to fix that locally and suggested a pull request - does that
> > > make sense?
> > > https://github.com/apache/flink/pull/448
> > >
> > >
> > > Johannes
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: [hidden email] [mailto:[hidden email]] Im Auftrag
> > > von Stephan Ewen
> > > Gesendet: Dienstag, 3. März 2015 10:03
> > > An: [hidden email]
> > > Betreff: Re: Access flink-conf.yaml data
> > >
> > > Hey Dulaj!
> > >
> > > As Chiwan said, the GlobalConfiguration object is used to load them
> > > initially.
> > >
> > > You can always use that to access the values (it works as a
> > > singleton
> > > internally) - but we are starting to move away from singletons, as
> > > they make test setups and embedding more difficult.
> > > In the JobManager and TaskManager setup, we pass a Configuration
> > > object around, which has all the values from the global configuration.
> > >
> > > Stephan
> > >
> > >
> > >
> > > On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]>
> > wrote:
> > >
> > > > I think that you can use
> > > > `org.apache.flink.configuration.GlobalConfiguration` to obtain
> > > > configuration object.
> > > >
> > > > Regards.
> > > > Chiwan Park (Sent with iPhone)
> > > >
> > > >
> > > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga
> > > > > <[hidden email]>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > > Can someone help me on how to access the flink-conf.yaml
> > > > > configuration
> > > > values inside the flink sources? Are these readily available as a
> > > > map somewhere?
> > > > >
> > > > > Thanks.
> > > >
> > > >
> > >
> >
>