Hi,
Can someone help me on how to access the flink-conf.yaml configuration values inside the flink sources? Are these readily available as a map somewhere? Thanks. |
I think that you can use `org.apache.flink.configuration.GlobalConfiguration` to obtain configuration object.
Regards. Chiwan Park (Sent with iPhone) > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]> wrote: > > Hi, > Can someone help me on how to access the flink-conf.yaml configuration values inside the flink sources? Are these readily available as a map somewhere? > > Thanks. |
Hey Dulaj!
As Chiwan said, the GlobalConfiguration object is used to load them initially. You can always use that to access the values (it works as a singleton internally) - but we are starting to move away from singletons, as they make test setups and embedding more difficult. In the JobManager and TaskManager setup, we pass a Configuration object around, which has all the values from the global configuration. Stephan On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> wrote: > I think that you can use > `org.apache.flink.configuration.GlobalConfiguration` to obtain > configuration object. > > Regards. > Chiwan Park (Sent with iPhone) > > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]> > wrote: > > > > Hi, > > Can someone help me on how to access the flink-conf.yaml configuration > values inside the flink sources? Are these readily available as a map > somewhere? > > > > Thanks. > > |
In reply to this post by Dulaj Viduranga
Hi Stephan,
I just came across the same problem in accessing the constants and in particular setting custom properties. In particular I noticed that the Minicluster started in the Local Environment cannot easily be customized as it does not take into account any custom environment variables - no way to pass them. I tried to fix that locally and suggested a pull request - does that make sense? https://github.com/apache/flink/pull/448 Johannes -----Ursprüngliche Nachricht----- Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Stephan Ewen Gesendet: Dienstag, 3. März 2015 10:03 An: [hidden email] Betreff: Re: Access flink-conf.yaml data Hey Dulaj! As Chiwan said, the GlobalConfiguration object is used to load them initially. You can always use that to access the values (it works as a singleton internally) - but we are starting to move away from singletons, as they make test setups and embedding more difficult. In the JobManager and TaskManager setup, we pass a Configuration object around, which has all the values from the global configuration. Stephan On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> wrote: > I think that you can use > `org.apache.flink.configuration.GlobalConfiguration` to obtain > configuration object. > > Regards. > Chiwan Park (Sent with iPhone) > > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]> > wrote: > > > > Hi, > > Can someone help me on how to access the flink-conf.yaml > > configuration > values inside the flink sources? Are these readily available as a map > somewhere? > > > > Thanks. > > |
Hi Johannes,
This change will allow users to pass a custom configuration to the LocalExecutor: https://github.com/apache/flink/pull/427. Is that what you're looking for? On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes < [hidden email]> wrote: > Hi Stephan, > > I just came across the same problem in accessing the constants and in > particular setting custom properties. > In particular I noticed that the Minicluster started in the Local > Environment cannot easily be customized as it does not take into account > any custom environment variables - no way to pass them. > I tried to fix that locally and suggested a pull request - does that make > sense? > https://github.com/apache/flink/pull/448 > > > Johannes > > -----Ursprüngliche Nachricht----- > Von: [hidden email] [mailto:[hidden email]] Im Auftrag von > Stephan Ewen > Gesendet: Dienstag, 3. März 2015 10:03 > An: [hidden email] > Betreff: Re: Access flink-conf.yaml data > > Hey Dulaj! > > As Chiwan said, the GlobalConfiguration object is used to load them > initially. > > You can always use that to access the values (it works as a singleton > internally) - but we are starting to move away from singletons, as they > make test setups and embedding more difficult. > In the JobManager and TaskManager setup, we pass a Configuration object > around, which has all the values from the global configuration. > > Stephan > > > > On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> wrote: > > > I think that you can use > > `org.apache.flink.configuration.GlobalConfiguration` to obtain > > configuration object. > > > > Regards. > > Chiwan Park (Sent with iPhone) > > > > > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]> > > wrote: > > > > > > Hi, > > > Can someone help me on how to access the flink-conf.yaml > > > configuration > > values inside the flink sources? Are these readily available as a map > > somewhere? > > > > > > Thanks. > > > > > |
I think that #427 is a good way to do this. It bypasses the singleton
GlobalConfiguration that I personally hope to get rid of. On Wed, Mar 4, 2015 at 11:49 AM, Robert Metzger <[hidden email]> wrote: > Hi Johannes, > > This change will allow users to pass a custom configuration to the > LocalExecutor: https://github.com/apache/flink/pull/427. > Is that what you're looking for? > > On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes < > [hidden email]> wrote: > > > Hi Stephan, > > > > I just came across the same problem in accessing the constants and in > > particular setting custom properties. > > In particular I noticed that the Minicluster started in the Local > > Environment cannot easily be customized as it does not take into account > > any custom environment variables - no way to pass them. > > I tried to fix that locally and suggested a pull request - does that make > > sense? > > https://github.com/apache/flink/pull/448 > > > > > > Johannes > > > > -----Ursprüngliche Nachricht----- > > Von: [hidden email] [mailto:[hidden email]] Im Auftrag von > > Stephan Ewen > > Gesendet: Dienstag, 3. März 2015 10:03 > > An: [hidden email] > > Betreff: Re: Access flink-conf.yaml data > > > > Hey Dulaj! > > > > As Chiwan said, the GlobalConfiguration object is used to load them > > initially. > > > > You can always use that to access the values (it works as a singleton > > internally) - but we are starting to move away from singletons, as they > > make test setups and embedding more difficult. > > In the JobManager and TaskManager setup, we pass a Configuration object > > around, which has all the values from the global configuration. > > > > Stephan > > > > > > > > On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> > wrote: > > > > > I think that you can use > > > `org.apache.flink.configuration.GlobalConfiguration` to obtain > > > configuration object. > > > > > > Regards. > > > Chiwan Park (Sent with iPhone) > > > > > > > > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga <[hidden email]> > > > wrote: > > > > > > > > Hi, > > > > Can someone help me on how to access the flink-conf.yaml > > > > configuration > > > values inside the flink sources? Are these readily available as a map > > > somewhere? > > > > > > > > Thanks. > > > > > > > > > |
In reply to this post by Dulaj Viduranga
Good point #427 looks more complete - will wait for the merge and just use my workaround locally in the meantime
Johannes -----Ursprüngliche Nachricht----- Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Stephan Ewen Gesendet: Mittwoch, 4. März 2015 12:42 An: [hidden email] Betreff: Re: Access flink-conf.yaml data I think that #427 is a good way to do this. It bypasses the singleton GlobalConfiguration that I personally hope to get rid of. On Wed, Mar 4, 2015 at 11:49 AM, Robert Metzger <[hidden email]> wrote: > Hi Johannes, > > This change will allow users to pass a custom configuration to the > LocalExecutor: https://github.com/apache/flink/pull/427. > Is that what you're looking for? > > On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes < > [hidden email]> wrote: > > > Hi Stephan, > > > > I just came across the same problem in accessing the constants and > > in particular setting custom properties. > > In particular I noticed that the Minicluster started in the Local > > Environment cannot easily be customized as it does not take into > > account any custom environment variables - no way to pass them. > > I tried to fix that locally and suggested a pull request - does that > > make sense? > > https://github.com/apache/flink/pull/448 > > > > > > Johannes > > > > -----Ursprüngliche Nachricht----- > > Von: [hidden email] [mailto:[hidden email]] Im Auftrag > > von Stephan Ewen > > Gesendet: Dienstag, 3. März 2015 10:03 > > An: [hidden email] > > Betreff: Re: Access flink-conf.yaml data > > > > Hey Dulaj! > > > > As Chiwan said, the GlobalConfiguration object is used to load them > > initially. > > > > You can always use that to access the values (it works as a > > singleton > > internally) - but we are starting to move away from singletons, as > > they make test setups and embedding more difficult. > > In the JobManager and TaskManager setup, we pass a Configuration > > object around, which has all the values from the global configuration. > > > > Stephan > > > > > > > > On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> > wrote: > > > > > I think that you can use > > > `org.apache.flink.configuration.GlobalConfiguration` to obtain > > > configuration object. > > > > > > Regards. > > > Chiwan Park (Sent with iPhone) > > > > > > > > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga > > > > <[hidden email]> > > > wrote: > > > > > > > > Hi, > > > > Can someone help me on how to access the flink-conf.yaml > > > > configuration > > > values inside the flink sources? Are these readily available as a > > > map somewhere? > > > > > > > > Thanks. > > > > > > > > > |
Cool.
I'll merge #427 as soon as https://github.com/apache/flink/pull/410 is in master. On Wed, Mar 4, 2015 at 12:48 PM, Kirschnick, Johannes < [hidden email]> wrote: > Good point #427 looks more complete - will wait for the merge and just use > my workaround locally in the meantime > > Johannes > > -----Ursprüngliche Nachricht----- > Von: [hidden email] [mailto:[hidden email]] Im Auftrag von > Stephan Ewen > Gesendet: Mittwoch, 4. März 2015 12:42 > An: [hidden email] > Betreff: Re: Access flink-conf.yaml data > > I think that #427 is a good way to do this. It bypasses the singleton > GlobalConfiguration that I personally hope to get rid of. > > On Wed, Mar 4, 2015 at 11:49 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi Johannes, > > > > This change will allow users to pass a custom configuration to the > > LocalExecutor: https://github.com/apache/flink/pull/427. > > Is that what you're looking for? > > > > On Wed, Mar 4, 2015 at 11:46 AM, Kirschnick, Johannes < > > [hidden email]> wrote: > > > > > Hi Stephan, > > > > > > I just came across the same problem in accessing the constants and > > > in particular setting custom properties. > > > In particular I noticed that the Minicluster started in the Local > > > Environment cannot easily be customized as it does not take into > > > account any custom environment variables - no way to pass them. > > > I tried to fix that locally and suggested a pull request - does that > > > make sense? > > > https://github.com/apache/flink/pull/448 > > > > > > > > > Johannes > > > > > > -----Ursprüngliche Nachricht----- > > > Von: [hidden email] [mailto:[hidden email]] Im Auftrag > > > von Stephan Ewen > > > Gesendet: Dienstag, 3. März 2015 10:03 > > > An: [hidden email] > > > Betreff: Re: Access flink-conf.yaml data > > > > > > Hey Dulaj! > > > > > > As Chiwan said, the GlobalConfiguration object is used to load them > > > initially. > > > > > > You can always use that to access the values (it works as a > > > singleton > > > internally) - but we are starting to move away from singletons, as > > > they make test setups and embedding more difficult. > > > In the JobManager and TaskManager setup, we pass a Configuration > > > object around, which has all the values from the global configuration. > > > > > > Stephan > > > > > > > > > > > > On Tue, Mar 3, 2015 at 6:08 AM, Chiwan Park <[hidden email]> > > wrote: > > > > > > > I think that you can use > > > > `org.apache.flink.configuration.GlobalConfiguration` to obtain > > > > configuration object. > > > > > > > > Regards. > > > > Chiwan Park (Sent with iPhone) > > > > > > > > > > > > > On Mar 3, 2015, at 12:17 PM, Dulaj Viduranga > > > > > <[hidden email]> > > > > wrote: > > > > > > > > > > Hi, > > > > > Can someone help me on how to access the flink-conf.yaml > > > > > configuration > > > > values inside the flink sources? Are these readily available as a > > > > map somewhere? > > > > > > > > > > Thanks. > > > > > > > > > > > > > > |
Free forum by Nabble | Edit this page |