Hi,
does anybody know how I can increase the stack size of a yarn session? Best Regards, Hilmi -- ================================================================== Hilmi Yildirim, M.Sc. Researcher DFKI GmbH Intelligente Analytik für Massendaten DFKI Projektbüro Berlin Alt-Moabit 91c D-10559 Berlin Phone: +49 30 23895 1814 E-Mail: [hidden email] ------------------------------------------------------------- Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) Dr. Walter Olthoff Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes Amtsgericht Kaiserslautern, HRB 2313 ------------------------------------------------------------- |
Hi Hilmi,
Flink on YARN is respecting the env.java.opts configuration parameter from the flink-conf.yaml. 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: > Hi, > does anybody know how I can increase the stack size of a yarn session? > > Best Regards, > Hilmi > > -- > ================================================================== > Hilmi Yildirim, M.Sc. > Researcher > > DFKI GmbH > Intelligente Analytik für Massendaten > DFKI Projektbüro Berlin > Alt-Moabit 91c > D-10559 Berlin > Phone: +49 30 23895 1814 > > E-Mail: [hidden email] > > ------------------------------------------------------------- > Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH > Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern > > Geschaeftsfuehrung: > Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) > Dr. Walter Olthoff > > Vorsitzender des Aufsichtsrats: > Prof. Dr. h.c. Hans A. Aukes > > Amtsgericht Kaiserslautern, HRB 2313 > ------------------------------------------------------------- > > |
Hi,
I added the following line to my flink-conf.yaml: env.java.opts: "-Xss1024m" But it does not work. Does this line mean that every single slot of each task manager uses a stack size of 1 GB? Best Regards, Hilmi Am 22.01.2016 um 11:46 schrieb Robert Metzger: > Hi Hilmi, > > Flink on YARN is respecting the env.java.opts configuration parameter from > the flink-conf.yaml. > > 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: > >> Hi, >> does anybody know how I can increase the stack size of a yarn session? >> >> Best Regards, >> Hilmi >> >> -- >> ================================================================== >> Hilmi Yildirim, M.Sc. >> Researcher >> >> DFKI GmbH >> Intelligente Analytik für Massendaten >> DFKI Projektbüro Berlin >> Alt-Moabit 91c >> D-10559 Berlin >> Phone: +49 30 23895 1814 >> >> E-Mail: [hidden email] >> >> ------------------------------------------------------------- >> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >> >> Geschaeftsfuehrung: >> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >> Dr. Walter Olthoff >> >> Vorsitzender des Aufsichtsrats: >> Prof. Dr. h.c. Hans A. Aukes >> >> Amtsgericht Kaiserslautern, HRB 2313 >> ------------------------------------------------------------- >> >> |
In reply to this post by Robert Metzger
Hi,
I added the following line to my flink-conf.yaml: env.java.opts: "-Xss1024m" But it does not work. Does this line mean that every single slot of each task manager uses a stack size of 1 GB? Best Regards, Hilmi Am 22.01.2016 um 11:46 schrieb Robert Metzger: > Hi Hilmi, > > Flink on YARN is respecting the env.java.opts configuration parameter from > the flink-conf.yaml. > > 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: > >> Hi, >> does anybody know how I can increase the stack size of a yarn session? >> >> Best Regards, >> Hilmi >> >> -- >> ================================================================== >> Hilmi Yildirim, M.Sc. >> Researcher >> >> DFKI GmbH >> Intelligente Analytik für Massendaten >> DFKI Projektbüro Berlin >> Alt-Moabit 91c >> D-10559 Berlin >> Phone: +49 30 23895 1814 >> >> E-Mail: [hidden email] >> >> ------------------------------------------------------------- >> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >> >> Geschaeftsfuehrung: >> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >> Dr. Walter Olthoff >> >> Vorsitzender des Aufsichtsrats: >> Prof. Dr. h.c. Hans A. Aukes >> >> Amtsgericht Kaiserslautern, HRB 2313 >> ------------------------------------------------------------- >> >> -- ================================================================== Hilmi Yildirim, M.Sc. Researcher DFKI GmbH Intelligente Analytik für Massendaten DFKI Projektbüro Berlin Alt-Moabit 91c D-10559 Berlin Phone: +49 30 23895 1814 E-Mail: [hidden email] ------------------------------------------------------------- Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) Dr. Walter Olthoff Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes Amtsgericht Kaiserslautern, HRB 2313 ------------------------------------------------------------- |
Hi Hilmi,
it means that each JVM the YARN session is starting (JobManager and TaskManagers) is initialized with that parameter. Can you check the logs of the TaskManager to see if the option has been passed properly? On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <[hidden email]> wrote: > Hi, > I added the following line to my flink-conf.yaml: > > env.java.opts: "-Xss1024m" > > But it does not work. Does this line mean that every single slot of each > task manager uses a stack size of 1 GB? > > Best Regards, > Hilmi > > Am 22.01.2016 um 11:46 schrieb Robert Metzger: > >> Hi Hilmi, >> >> Flink on YARN is respecting the env.java.opts configuration parameter from >> the flink-conf.yaml. >> >> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: >> >> Hi, >>> does anybody know how I can increase the stack size of a yarn session? >>> >>> Best Regards, >>> Hilmi >>> >>> -- >>> ================================================================== >>> Hilmi Yildirim, M.Sc. >>> Researcher >>> >>> DFKI GmbH >>> Intelligente Analytik für Massendaten >>> DFKI Projektbüro Berlin >>> Alt-Moabit 91c >>> D-10559 Berlin >>> Phone: +49 30 23895 1814 >>> >>> E-Mail: [hidden email] >>> >>> ------------------------------------------------------------- >>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >>> >>> Geschaeftsfuehrung: >>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >>> Dr. Walter Olthoff >>> >>> Vorsitzender des Aufsichtsrats: >>> Prof. Dr. h.c. Hans A. Aukes >>> >>> Amtsgericht Kaiserslautern, HRB 2313 >>> ------------------------------------------------------------- >>> >>> >>> > > -- > ================================================================== > Hilmi Yildirim, M.Sc. > Researcher > > DFKI GmbH > Intelligente Analytik für Massendaten > DFKI Projektbüro Berlin > Alt-Moabit 91c > D-10559 Berlin > Phone: +49 30 23895 1814 > > E-Mail: [hidden email] > > ------------------------------------------------------------- > Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH > Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern > > Geschaeftsfuehrung: > Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) > Dr. Walter Olthoff > > Vorsitzender des Aufsichtsrats: > Prof. Dr. h.c. Hans A. Aukes > > Amtsgericht Kaiserslautern, HRB 2313 > ------------------------------------------------------------- > > |
Hi Robert,
in the logs I found this 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner - JVM Options: 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Xms3072m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Xmx3072m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -XX:MaxDirectMemorySize=3072m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Xss1024m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Dlogback.configurationFile=file:logback.xml 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Dlog4j.configuration=file:log4j.properties It seems that the argument is passed to the Taskmanager. Or am I wrong? Best Regards, Hilmi Am 22.01.2016 um 12:25 schrieb Robert Metzger: > Hi Hilmi, > > it means that each JVM the YARN session is starting (JobManager and > TaskManagers) is initialized with that parameter. > > Can you check the logs of the TaskManager to see if the option has been > passed properly? > > On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <[hidden email]> > wrote: > >> Hi, >> I added the following line to my flink-conf.yaml: >> >> env.java.opts: "-Xss1024m" >> >> But it does not work. Does this line mean that every single slot of each >> task manager uses a stack size of 1 GB? >> >> Best Regards, >> Hilmi >> >> Am 22.01.2016 um 11:46 schrieb Robert Metzger: >> >>> Hi Hilmi, >>> >>> Flink on YARN is respecting the env.java.opts configuration parameter from >>> the flink-conf.yaml. >>> >>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: >>> >>> Hi, >>>> does anybody know how I can increase the stack size of a yarn session? >>>> >>>> Best Regards, >>>> Hilmi >>>> >>>> -- >>>> ================================================================== >>>> Hilmi Yildirim, M.Sc. >>>> Researcher >>>> >>>> DFKI GmbH >>>> Intelligente Analytik für Massendaten >>>> DFKI Projektbüro Berlin >>>> Alt-Moabit 91c >>>> D-10559 Berlin >>>> Phone: +49 30 23895 1814 >>>> >>>> E-Mail: [hidden email] >>>> >>>> ------------------------------------------------------------- >>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >>>> >>>> Geschaeftsfuehrung: >>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >>>> Dr. Walter Olthoff >>>> >>>> Vorsitzender des Aufsichtsrats: >>>> Prof. Dr. h.c. Hans A. Aukes >>>> >>>> Amtsgericht Kaiserslautern, HRB 2313 >>>> ------------------------------------------------------------- >>>> >>>> >>>> >> -- >> ================================================================== >> Hilmi Yildirim, M.Sc. >> Researcher >> >> DFKI GmbH >> Intelligente Analytik für Massendaten >> DFKI Projektbüro Berlin >> Alt-Moabit 91c >> D-10559 Berlin >> Phone: +49 30 23895 1814 >> >> E-Mail: [hidden email] >> >> ------------------------------------------------------------- >> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >> >> Geschaeftsfuehrung: >> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >> Dr. Walter Olthoff >> >> Vorsitzender des Aufsichtsrats: >> Prof. Dr. h.c. Hans A. Aukes >> >> Amtsgericht Kaiserslautern, HRB 2313 >> ------------------------------------------------------------- >> >> |
In reply to this post by Robert Metzger
Hi Robert,
in the logs I found this 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner - JVM Options: 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Xms3072m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Xmx3072m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -XX:MaxDirectMemorySize=3072m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Xss1024m 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Dlogback.configurationFile=file:logback.xml 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner - -Dlog4j.configuration=file:log4j.properties It seems that the argument is passed to the Taskmanager. Or am I wrong? Best Regards, Hilmi Am 22.01.2016 um 12:25 schrieb Robert Metzger: > Hi Hilmi, > > it means that each JVM the YARN session is starting (JobManager and > TaskManagers) is initialized with that parameter. > > Can you check the logs of the TaskManager to see if the option has been > passed properly? > > On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <[hidden email]> > wrote: > >> Hi, >> I added the following line to my flink-conf.yaml: >> >> env.java.opts: "-Xss1024m" >> >> But it does not work. Does this line mean that every single slot of each >> task manager uses a stack size of 1 GB? >> >> Best Regards, >> Hilmi >> >> Am 22.01.2016 um 11:46 schrieb Robert Metzger: >> >>> Hi Hilmi, >>> >>> Flink on YARN is respecting the env.java.opts configuration parameter from >>> the flink-conf.yaml. >>> >>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: >>> >>> Hi, >>>> does anybody know how I can increase the stack size of a yarn session? >>>> >>>> Best Regards, >>>> Hilmi >>>> >>>> -- >>>> ================================================================== >>>> Hilmi Yildirim, M.Sc. >>>> Researcher >>>> >>>> DFKI GmbH >>>> Intelligente Analytik für Massendaten >>>> DFKI Projektbüro Berlin >>>> Alt-Moabit 91c >>>> D-10559 Berlin >>>> Phone: +49 30 23895 1814 >>>> >>>> E-Mail: [hidden email] >>>> >>>> ------------------------------------------------------------- >>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >>>> >>>> Geschaeftsfuehrung: >>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >>>> Dr. Walter Olthoff >>>> >>>> Vorsitzender des Aufsichtsrats: >>>> Prof. Dr. h.c. Hans A. Aukes >>>> >>>> Amtsgericht Kaiserslautern, HRB 2313 >>>> ------------------------------------------------------------- >>>> >>>> >>>> >> -- >> ================================================================== >> Hilmi Yildirim, M.Sc. >> Researcher >> >> DFKI GmbH >> Intelligente Analytik für Massendaten >> DFKI Projektbüro Berlin >> Alt-Moabit 91c >> D-10559 Berlin >> Phone: +49 30 23895 1814 >> >> E-Mail: [hidden email] >> >> ------------------------------------------------------------- >> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >> >> Geschaeftsfuehrung: >> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >> Dr. Walter Olthoff >> >> Vorsitzender des Aufsichtsrats: >> Prof. Dr. h.c. Hans A. Aukes >> >> Amtsgericht Kaiserslautern, HRB 2313 >> ------------------------------------------------------------- >> >> -- ================================================================== Hilmi Yildirim, M.Sc. Researcher DFKI GmbH Intelligente Analytik für Massendaten DFKI Projektbüro Berlin Alt-Moabit 91c D-10559 Berlin Phone: +49 30 23895 1814 E-Mail: [hidden email] ------------------------------------------------------------- Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) Dr. Walter Olthoff Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes Amtsgericht Kaiserslautern, HRB 2313 ------------------------------------------------------------- |
In reply to this post by Hilmi Yildirim-2
Hi,
yes, looks like the argument is passed correctly. By the way: Why are you sending your answers always twice to the mailing list? On Fri, Jan 22, 2016 at 2:17 PM, Hilmi Yildirim < [hidden email]> wrote: > Hi Robert, > in the logs I found this > > 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - JVM Options: > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - -Xms3072m > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - -Xmx3072m > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - -XX:MaxDirectMemorySize=3072m > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - -Xss1024m > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - > -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - -Dlogback.configurationFile=file:logback.xml > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - -Dlog4j.configuration=file:log4j.properties > > > It seems that the argument is passed to the Taskmanager. Or am I wrong? > > Best Regards, > Hilmi > > > > Am 22.01.2016 um 12:25 schrieb Robert Metzger: > >> Hi Hilmi, >> >> it means that each JVM the YARN session is starting (JobManager and >> TaskManagers) is initialized with that parameter. >> >> Can you check the logs of the TaskManager to see if the option has been >> passed properly? >> >> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <[hidden email]> >> wrote: >> >> Hi, >>> I added the following line to my flink-conf.yaml: >>> >>> env.java.opts: "-Xss1024m" >>> >>> But it does not work. Does this line mean that every single slot of each >>> task manager uses a stack size of 1 GB? >>> >>> Best Regards, >>> Hilmi >>> >>> Am 22.01.2016 um 11:46 schrieb Robert Metzger: >>> >>> Hi Hilmi, >>>> >>>> Flink on YARN is respecting the env.java.opts configuration parameter >>>> from >>>> the flink-conf.yaml. >>>> >>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: >>>> >>>> Hi, >>>> >>>>> does anybody know how I can increase the stack size of a yarn session? >>>>> >>>>> Best Regards, >>>>> Hilmi >>>>> >>>>> -- >>>>> ================================================================== >>>>> Hilmi Yildirim, M.Sc. >>>>> Researcher >>>>> >>>>> DFKI GmbH >>>>> Intelligente Analytik für Massendaten >>>>> DFKI Projektbüro Berlin >>>>> Alt-Moabit 91c >>>>> D-10559 Berlin >>>>> Phone: +49 30 23895 1814 >>>>> >>>>> E-Mail: [hidden email] >>>>> >>>>> ------------------------------------------------------------- >>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >>>>> >>>>> Geschaeftsfuehrung: >>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >>>>> Dr. Walter Olthoff >>>>> >>>>> Vorsitzender des Aufsichtsrats: >>>>> Prof. Dr. h.c. Hans A. Aukes >>>>> >>>>> Amtsgericht Kaiserslautern, HRB 2313 >>>>> ------------------------------------------------------------- >>>>> >>>>> >>>>> >>>>> -- >>> ================================================================== >>> Hilmi Yildirim, M.Sc. >>> Researcher >>> >>> DFKI GmbH >>> Intelligente Analytik für Massendaten >>> DFKI Projektbüro Berlin >>> Alt-Moabit 91c >>> D-10559 Berlin >>> Phone: +49 30 23895 1814 >>> >>> E-Mail: [hidden email] >>> >>> ------------------------------------------------------------- >>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >>> >>> Geschaeftsfuehrung: >>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >>> Dr. Walter Olthoff >>> >>> Vorsitzender des Aufsichtsrats: >>> Prof. Dr. h.c. Hans A. Aukes >>> >>> Amtsgericht Kaiserslautern, HRB 2313 >>> ------------------------------------------------------------- >>> >>> >>> > |
@Hilmi:
A stack size of 1GB is too much, maybe the JVM ignores it? Can you try "-Xss2m" or so? Is this to prevent stack overflow errors in some code? Stephan On Fri, Jan 22, 2016 at 2:21 PM, Robert Metzger <[hidden email]> wrote: > Hi, > > yes, looks like the argument is passed correctly. > > By the way: Why are you sending your answers always twice to the mailing > list? > > On Fri, Jan 22, 2016 at 2:17 PM, Hilmi Yildirim < > [hidden email]> wrote: > > > Hi Robert, > > in the logs I found this > > > > 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - JVM Options: > > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - -Xms3072m > > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - -Xmx3072m > > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - -XX:MaxDirectMemorySize=3072m > > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - -Xss1024m > > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - > > > -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log > > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - -Dlogback.configurationFile=file:logback.xml > > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner > > - -Dlog4j.configuration=file:log4j.properties > > > > > > It seems that the argument is passed to the Taskmanager. Or am I wrong? > > > > Best Regards, > > Hilmi > > > > > > > > Am 22.01.2016 um 12:25 schrieb Robert Metzger: > > > >> Hi Hilmi, > >> > >> it means that each JVM the YARN session is starting (JobManager and > >> TaskManagers) is initialized with that parameter. > >> > >> Can you check the logs of the TaskManager to see if the option has been > >> passed properly? > >> > >> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim < > [hidden email]> > >> wrote: > >> > >> Hi, > >>> I added the following line to my flink-conf.yaml: > >>> > >>> env.java.opts: "-Xss1024m" > >>> > >>> But it does not work. Does this line mean that every single slot of > each > >>> task manager uses a stack size of 1 GB? > >>> > >>> Best Regards, > >>> Hilmi > >>> > >>> Am 22.01.2016 um 11:46 schrieb Robert Metzger: > >>> > >>> Hi Hilmi, > >>>> > >>>> Flink on YARN is respecting the env.java.opts configuration parameter > >>>> from > >>>> the flink-conf.yaml. > >>>> > >>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: > >>>> > >>>> Hi, > >>>> > >>>>> does anybody know how I can increase the stack size of a yarn > session? > >>>>> > >>>>> Best Regards, > >>>>> Hilmi > >>>>> > >>>>> -- > >>>>> ================================================================== > >>>>> Hilmi Yildirim, M.Sc. > >>>>> Researcher > >>>>> > >>>>> DFKI GmbH > >>>>> Intelligente Analytik für Massendaten > >>>>> DFKI Projektbüro Berlin > >>>>> Alt-Moabit 91c > >>>>> D-10559 Berlin > >>>>> Phone: +49 30 23895 1814 > >>>>> > >>>>> E-Mail: [hidden email] > >>>>> > >>>>> ------------------------------------------------------------- > >>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH > >>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern > >>>>> > >>>>> Geschaeftsfuehrung: > >>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) > >>>>> Dr. Walter Olthoff > >>>>> > >>>>> Vorsitzender des Aufsichtsrats: > >>>>> Prof. Dr. h.c. Hans A. Aukes > >>>>> > >>>>> Amtsgericht Kaiserslautern, HRB 2313 > >>>>> ------------------------------------------------------------- > >>>>> > >>>>> > >>>>> > >>>>> -- > >>> ================================================================== > >>> Hilmi Yildirim, M.Sc. > >>> Researcher > >>> > >>> DFKI GmbH > >>> Intelligente Analytik für Massendaten > >>> DFKI Projektbüro Berlin > >>> Alt-Moabit 91c > >>> D-10559 Berlin > >>> Phone: +49 30 23895 1814 > >>> > >>> E-Mail: [hidden email] > >>> > >>> ------------------------------------------------------------- > >>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH > >>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern > >>> > >>> Geschaeftsfuehrung: > >>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) > >>> Dr. Walter Olthoff > >>> > >>> Vorsitzender des Aufsichtsrats: > >>> Prof. Dr. h.c. Hans A. Aukes > >>> > >>> Amtsgericht Kaiserslautern, HRB 2313 > >>> ------------------------------------------------------------- > >>> > >>> > >>> > > > |
Hi,
I have solved the problem. I used an object which contained a huge generic MutableList. When kryo serializes the MutableList it does many recursive calls which lead to a huge stack and to the StackOverFlowError. I changed the MutableList to a Set and now kryo does not do so many recursive calls and the stack keeps small :) Best Regards, Hilmi Am 25.01.2016 um 16:34 schrieb Stephan Ewen: > @Hilmi: > > A stack size of 1GB is too much, maybe the JVM ignores it? Can you try > "-Xss2m" or so? > > Is this to prevent stack overflow errors in some code? > > Stephan > > > On Fri, Jan 22, 2016 at 2:21 PM, Robert Metzger <[hidden email]> wrote: > >> Hi, >> >> yes, looks like the argument is passed correctly. >> >> By the way: Why are you sending your answers always twice to the mailing >> list? >> >> On Fri, Jan 22, 2016 at 2:17 PM, Hilmi Yildirim < >> [hidden email]> wrote: >> >>> Hi Robert, >>> in the logs I found this >>> >>> 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - JVM Options: >>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - -Xms3072m >>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - -Xmx3072m >>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - -XX:MaxDirectMemorySize=3072m >>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - -Xss1024m >>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - >>> >> -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log >>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - -Dlogback.configurationFile=file:logback.xml >>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner >>> - -Dlog4j.configuration=file:log4j.properties >>> >>> >>> It seems that the argument is passed to the Taskmanager. Or am I wrong? >>> >>> Best Regards, >>> Hilmi >>> >>> >>> >>> Am 22.01.2016 um 12:25 schrieb Robert Metzger: >>> >>>> Hi Hilmi, >>>> >>>> it means that each JVM the YARN session is starting (JobManager and >>>> TaskManagers) is initialized with that parameter. >>>> >>>> Can you check the logs of the TaskManager to see if the option has been >>>> passed properly? >>>> >>>> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim < >> [hidden email]> >>>> wrote: >>>> >>>> Hi, >>>>> I added the following line to my flink-conf.yaml: >>>>> >>>>> env.java.opts: "-Xss1024m" >>>>> >>>>> But it does not work. Does this line mean that every single slot of >> each >>>>> task manager uses a stack size of 1 GB? >>>>> >>>>> Best Regards, >>>>> Hilmi >>>>> >>>>> Am 22.01.2016 um 11:46 schrieb Robert Metzger: >>>>> >>>>> Hi Hilmi, >>>>>> Flink on YARN is respecting the env.java.opts configuration parameter >>>>>> from >>>>>> the flink-conf.yaml. >>>>>> >>>>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <[hidden email]>: >>>>>> >>>>>> Hi, >>>>>> >>>>>>> does anybody know how I can increase the stack size of a yarn >> session? >>>>>>> Best Regards, >>>>>>> Hilmi >>>>>>> >>>>>>> -- >>>>>>> ================================================================== >>>>>>> Hilmi Yildirim, M.Sc. >>>>>>> Researcher >>>>>>> >>>>>>> DFKI GmbH >>>>>>> Intelligente Analytik für Massendaten >>>>>>> DFKI Projektbüro Berlin >>>>>>> Alt-Moabit 91c >>>>>>> D-10559 Berlin >>>>>>> Phone: +49 30 23895 1814 >>>>>>> >>>>>>> E-Mail: [hidden email] >>>>>>> >>>>>>> ------------------------------------------------------------- >>>>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >>>>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >>>>>>> >>>>>>> Geschaeftsfuehrung: >>>>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >>>>>>> Dr. Walter Olthoff >>>>>>> >>>>>>> Vorsitzender des Aufsichtsrats: >>>>>>> Prof. Dr. h.c. Hans A. Aukes >>>>>>> >>>>>>> Amtsgericht Kaiserslautern, HRB 2313 >>>>>>> ------------------------------------------------------------- >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>> ================================================================== >>>>> Hilmi Yildirim, M.Sc. >>>>> Researcher >>>>> >>>>> DFKI GmbH >>>>> Intelligente Analytik für Massendaten >>>>> DFKI Projektbüro Berlin >>>>> Alt-Moabit 91c >>>>> D-10559 Berlin >>>>> Phone: +49 30 23895 1814 >>>>> >>>>> E-Mail: [hidden email] >>>>> >>>>> ------------------------------------------------------------- >>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH >>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern >>>>> >>>>> Geschaeftsfuehrung: >>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) >>>>> Dr. Walter Olthoff >>>>> >>>>> Vorsitzender des Aufsichtsrats: >>>>> Prof. Dr. h.c. Hans A. Aukes >>>>> >>>>> Amtsgericht Kaiserslautern, HRB 2313 >>>>> ------------------------------------------------------------- >>>>> >>>>> >>>>> -- ================================================================== Hilmi Yildirim, M.Sc. Researcher DFKI GmbH Intelligente Analytik für Massendaten DFKI Projektbüro Berlin Alt-Moabit 91c D-10559 Berlin Phone: +49 30 23895 1814 E-Mail: [hidden email] ------------------------------------------------------------- Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender) Dr. Walter Olthoff Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes Amtsgericht Kaiserslautern, HRB 2313 ------------------------------------------------------------- |
Free forum by Nabble | Edit this page |