[Feature]Returning RuntimeException to REST client while job submission

[Lavkesh Lahngir]

Hello,
The change in FLINK-10312
<https://issues.apache.org/jira/browse/FLINK-10312> makes REST response of
the API
<https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run>
not
very informative. It strips the stack trace and returns a generic message.
People using flink-cluster deployment who do not have access to job manager
logs, will not be able to figure out the root cause.
In the case of when the job submission fails,
In 1.6.2, I get
{
    "errors": [
        "org.apache.flink.client.program.ProgramInvocationException: The
main method caused an error."
    ]
}

Is there a plan to improve error messages sent to the client?
Is somebody working on this already?

Thanks in advance.
~Lavkesh

[Lavkesh Lahngir]

Or maybe I am missing something? It looks like the JIRA is trying to solve
the same issues I stated 🤔
In the main method, I just threw a simple new Exception("Some message") and
I got the response I mentioned from the rest API.

Thanks.

On Wed, Jan 23, 2019 at 2:50 PM Lavkesh Lahngir <[hidden email]> wrote:

> Hello,
> The change in FLINK-10312
> <https://issues.apache.org/jira/browse/FLINK-10312> makes REST response
> of the API
> <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run> not
> very informative. It strips the stack trace and returns a generic message.
> People using flink-cluster deployment who do not have access to job manager
> logs, will not be able to figure out the root cause.
> In the case of when the job submission fails,
> In 1.6.2, I get
> {
>     "errors": [
>         "org.apache.flink.client.program.ProgramInvocationException: The
> main method caused an error."
>     ]
> }
>
> Is there a plan to improve error messages sent to the client?
> Is somebody working on this already?
>
> Thanks in advance.
> ~Lavkesh
>

[Chesnay Schepler-3]

Which version are you using?

On 23.01.2019 08:00, Lavkesh Lahngir wrote:

> Or maybe I am missing something? It looks like the JIRA is trying to solve
> the same issues I stated 🤔
> In the main method, I just threw a simple new Exception("Some message") and
> I got the response I mentioned from the rest API.
>
> Thanks.
>
> On Wed, Jan 23, 2019 at 2:50 PM Lavkesh Lahngir <[hidden email]> wrote:
>
>> Hello,
>> The change in FLINK-10312
>> <https://issues.apache.org/jira/browse/FLINK-10312> makes REST response
>> of the API
>> <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run> not
>> very informative. It strips the stack trace and returns a generic message.
>> People using flink-cluster deployment who do not have access to job manager
>> logs, will not be able to figure out the root cause.
>> In the case of when the job submission fails,
>> In 1.6.2, I get
>> {
>>      "errors": [
>>          "org.apache.flink.client.program.ProgramInvocationException: The
>> main method caused an error."
>>      ]
>> }
>>
>> Is there a plan to improve error messages sent to the client?
>> Is somebody working on this already?
>>
>> Thanks in advance.
>> ~Lavkesh
>>

[Lavkesh Lahngir]

Actually, I realized my mistake that JarRunHandler is being used in the
jar/run API call.
And the changes are done in RestClusterClient.
The problem I was facing was that It always gives me "The main method
caused an error"
without any more details.
I am thinking when we throw ProgramInvocationException in PackagedProgram.
callMainMethod()
we should add exceptionInMethod.getMessage() too.

*---
a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*

*+++
b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*

@@ -543,7 +543,7 @@ public class PackagedProgram {

                        } else if (exceptionInMethod instanceof
ProgramInvocationException) {

                                throw (ProgramInvocationException)
exceptionInMethod;

                        } else {

-                               throw new ProgramInvocationException("The
main method caused an error.", exceptionInMethod);

+                               throw new ProgramInvocationException("The
main method caused an error.: " + exceptionInMethod.getMessage(),
exceptionInMethod);

                        }

                }

                catch (Throwable t) {

What will you suggest?

On Wed, Jan 23, 2019 at 7:01 PM Chesnay Schepler <[hidden email]> wrote:

> Which version are you using?
>
> On 23.01.2019 08:00, Lavkesh Lahngir wrote:
> > Or maybe I am missing something? It looks like the JIRA is trying to
> solve
> > the same issues I stated 🤔
> > In the main method, I just threw a simple new Exception("Some message")
> and
> > I got the response I mentioned from the rest API.
> >
> > Thanks.
> >
> > On Wed, Jan 23, 2019 at 2:50 PM Lavkesh Lahngir <[hidden email]>
> wrote:
> >
> >> Hello,
> >> The change in FLINK-10312
> >> <https://issues.apache.org/jira/browse/FLINK-10312> makes REST response
> >> of the API
> >> <
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run>
> not
> >> very informative. It strips the stack trace and returns a generic
> message.
> >> People using flink-cluster deployment who do not have access to job
> manager
> >> logs, will not be able to figure out the root cause.
> >> In the case of when the job submission fails,
> >> In 1.6.2, I get
> >> {
> >>      "errors": [
> >>          "org.apache.flink.client.program.ProgramInvocationException:
> The
> >> main method caused an error."
> >>      ]
> >> }
> >>
> >> Is there a plan to improve error messages sent to the client?
> >> Is somebody working on this already?
> >>
> >> Thanks in advance.
> >> ~Lavkesh
> >>
>
>

[Chesnay Schepler-3]

I suggest that you first tell me which version you are using so that I
can a) reproduce the issue and b) check that this issue wasn't fixed in
master or a recent bugfix release.

On 23.01.2019 17:16, Lavkesh Lahngir wrote:

> Actually, I realized my mistake that JarRunHandler is being used in the
> jar/run API call.
> And the changes are done in RestClusterClient.
> The problem I was facing was that It always gives me "The main method
> caused an error"
> without any more details.
> I am thinking when we throw ProgramInvocationException in PackagedProgram.
> callMainMethod()
> we should add exceptionInMethod.getMessage() too.
>
> *---
> a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
>
> *+++
> b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
>
> @@ -543,7 +543,7 @@ public class PackagedProgram {
>
>                          } else if (exceptionInMethod instanceof
> ProgramInvocationException) {
>
>                                  throw (ProgramInvocationException)
> exceptionInMethod;
>
>                          } else {
>
> -                               throw new ProgramInvocationException("The
> main method caused an error.", exceptionInMethod);
>
> +                               throw new ProgramInvocationException("The
> main method caused an error.: " + exceptionInMethod.getMessage(),
> exceptionInMethod);
>
>                          }
>
>                  }
>
>                  catch (Throwable t) {
>
> What will you suggest?
>
> On Wed, Jan 23, 2019 at 7:01 PM Chesnay Schepler <[hidden email]> wrote:
>
>> Which version are you using?
>>
>> On 23.01.2019 08:00, Lavkesh Lahngir wrote:
>>> Or maybe I am missing something? It looks like the JIRA is trying to
>> solve
>>> the same issues I stated 🤔
>>> In the main method, I just threw a simple new Exception("Some message")
>> and
>>> I got the response I mentioned from the rest API.
>>>
>>> Thanks.
>>>
>>> On Wed, Jan 23, 2019 at 2:50 PM Lavkesh Lahngir <[hidden email]>
>> wrote:
>>>> Hello,
>>>> The change in FLINK-10312
>>>> <https://issues.apache.org/jira/browse/FLINK-10312> makes REST response
>>>> of the API
>>>> <
>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run>
>> not
>>>> very informative. It strips the stack trace and returns a generic
>> message.
>>>> People using flink-cluster deployment who do not have access to job
>> manager
>>>> logs, will not be able to figure out the root cause.
>>>> In the case of when the job submission fails,
>>>> In 1.6.2, I get
>>>> {
>>>>       "errors": [
>>>>           "org.apache.flink.client.program.ProgramInvocationException:
>> The
>>>> main method caused an error."
>>>>       ]
>>>> }
>>>>
>>>> Is there a plan to improve error messages sent to the client?
>>>> Is somebody working on this already?
>>>>
>>>> Thanks in advance.
>>>> ~Lavkesh
>>>>
>>

[Lavkesh Lahngir]

Hello,
I mentioned in the first email.

Version: 1.6.2, Commit ID: 3456ad0

On Thu, Jan 24, 2019 at 12:33 AM Chesnay Schepler <[hidden email]>
wrote:

> I suggest that you first tell me which version you are using so that I
> can a) reproduce the issue and b) check that this issue wasn't fixed in
> master or a recent bugfix release.
>
> On 23.01.2019 17:16, Lavkesh Lahngir wrote:
> > Actually, I realized my mistake that JarRunHandler is being used in the
> > jar/run API call.
> > And the changes are done in RestClusterClient.
> > The problem I was facing was that It always gives me "The main method
> > caused an error"
> > without any more details.
> > I am thinking when we throw ProgramInvocationException in
> PackagedProgram.
> > callMainMethod()
> > we should add exceptionInMethod.getMessage() too.
> >
> > *---
> >
> a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
> >
> > *+++
> >
> b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
> >
> > @@ -543,7 +543,7 @@ public class PackagedProgram {
> >
> >                          } else if (exceptionInMethod instanceof
> > ProgramInvocationException) {
> >
> >                                  throw (ProgramInvocationException)
> > exceptionInMethod;
> >
> >                          } else {
> >
> > -                               throw new ProgramInvocationException("The
> > main method caused an error.", exceptionInMethod);
> >
> > +                               throw new ProgramInvocationException("The
> > main method caused an error.: " + exceptionInMethod.getMessage(),
> > exceptionInMethod);
> >
> >                          }
> >
> >                  }
> >
> >                  catch (Throwable t) {
> >
> > What will you suggest?
> >
> > On Wed, Jan 23, 2019 at 7:01 PM Chesnay Schepler <[hidden email]>
> wrote:
> >
> >> Which version are you using?
> >>
> >> On 23.01.2019 08:00, Lavkesh Lahngir wrote:
> >>> Or maybe I am missing something? It looks like the JIRA is trying to
> >> solve
> >>> the same issues I stated 🤔
> >>> In the main method, I just threw a simple new Exception("Some message")
> >> and
> >>> I got the response I mentioned from the rest API.
> >>>
> >>> Thanks.
> >>>
> >>> On Wed, Jan 23, 2019 at 2:50 PM Lavkesh Lahngir <[hidden email]>
> >> wrote:
> >>>> Hello,
> >>>> The change in FLINK-10312
> >>>> <https://issues.apache.org/jira/browse/FLINK-10312> makes REST
> response
> >>>> of the API
> >>>> <
> >>
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run
> >
> >> not
> >>>> very informative. It strips the stack trace and returns a generic
> >> message.
> >>>> People using flink-cluster deployment who do not have access to job
> >> manager
> >>>> logs, will not be able to figure out the root cause.
> >>>> In the case of when the job submission fails,
> >>>> In 1.6.2, I get
> >>>> {
> >>>>       "errors": [
> >>>>           "org.apache.flink.client.program.ProgramInvocationException:
> >> The
> >>>> main method caused an error."
> >>>>       ]
> >>>> }
> >>>>
> >>>> Is there a plan to improve error messages sent to the client?
> >>>> Is somebody working on this already?
> >>>>
> >>>> Thanks in advance.
> >>>> ~Lavkesh
> >>>>
> >>
>
>

[Lavkesh Lahngir]

Hi,
It's not fixed in the master. I compiled and ran it yesterday.
I am not if that is an issue or design choice.

On Thu, Jan 24, 2019 at 11:38 AM Lavkesh Lahngir <[hidden email]> wrote:

> Hello,
> I mentioned in the first email.
>
> Version: 1.6.2, Commit ID: 3456ad0
>
> On Thu, Jan 24, 2019 at 12:33 AM Chesnay Schepler <[hidden email]>
> wrote:
>
>> I suggest that you first tell me which version you are using so that I
>> can a) reproduce the issue and b) check that this issue wasn't fixed in
>> master or a recent bugfix release.
>>
>> On 23.01.2019 17:16, Lavkesh Lahngir wrote:
>> > Actually, I realized my mistake that JarRunHandler is being used in the
>> > jar/run API call.
>> > And the changes are done in RestClusterClient.
>> > The problem I was facing was that It always gives me "The main method
>> > caused an error"
>> > without any more details.
>> > I am thinking when we throw ProgramInvocationException in
>> PackagedProgram.
>> > callMainMethod()
>> > we should add exceptionInMethod.getMessage() too.
>> >
>> > *---
>> >
>> a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
>> >
>> > *+++
>> >
>> b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
>> >
>> > @@ -543,7 +543,7 @@ public class PackagedProgram {
>> >
>> >                          } else if (exceptionInMethod instanceof
>> > ProgramInvocationException) {
>> >
>> >                                  throw (ProgramInvocationException)
>> > exceptionInMethod;
>> >
>> >                          } else {
>> >
>> > -                               throw new
>> ProgramInvocationException("The
>> > main method caused an error.", exceptionInMethod);
>> >
>> > +                               throw new
>> ProgramInvocationException("The
>> > main method caused an error.: " + exceptionInMethod.getMessage(),
>> > exceptionInMethod);
>> >
>> >                          }
>> >
>> >                  }
>> >
>> >                  catch (Throwable t) {
>> >
>> > What will you suggest?
>> >
>> > On Wed, Jan 23, 2019 at 7:01 PM Chesnay Schepler <[hidden email]>
>> wrote:
>> >
>> >> Which version are you using?
>> >>
>> >> On 23.01.2019 08:00, Lavkesh Lahngir wrote:
>> >>> Or maybe I am missing something? It looks like the JIRA is trying to
>> >> solve
>> >>> the same issues I stated 🤔
>> >>> In the main method, I just threw a simple new Exception("Some
>> message")
>> >> and
>> >>> I got the response I mentioned from the rest API.
>> >>>
>> >>> Thanks.
>> >>>
>> >>> On Wed, Jan 23, 2019 at 2:50 PM Lavkesh Lahngir <[hidden email]>
>> >> wrote:
>> >>>> Hello,
>> >>>> The change in FLINK-10312
>> >>>> <https://issues.apache.org/jira/browse/FLINK-10312> makes REST
>> response
>> >>>> of the API
>> >>>> <
>> >>
>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run
>> >
>> >> not
>> >>>> very informative. It strips the stack trace and returns a generic
>> >> message.
>> >>>> People using flink-cluster deployment who do not have access to job
>> >> manager
>> >>>> logs, will not be able to figure out the root cause.
>> >>>> In the case of when the job submission fails,
>> >>>> In 1.6.2, I get
>> >>>> {
>> >>>>       "errors": [
>> >>>>
>>  "org.apache.flink.client.program.ProgramInvocationException:
>> >> The
>> >>>> main method caused an error."
>> >>>>       ]
>> >>>> }
>> >>>>
>> >>>> Is there a plan to improve error messages sent to the client?
>> >>>> Is somebody working on this already?
>> >>>>
>> >>>> Thanks in advance.
>> >>>> ~Lavkesh
>> >>>>
>> >>
>>
>>

[Lavkesh Lahngir]

Hello,
I created a Jira. Can somebody review it, please? Or suggest if this is
useful?
https://issues.apache.org/jira/browse/FLINK-11423

Thank you.
~Lavkesh

On Thu, Jan 24, 2019 at 11:40 AM Lavkesh Lahngir <[hidden email]> wrote:

> Hi,
> It's not fixed in the master. I compiled and ran it yesterday.
> I am not if that is an issue or design choice.
>
> On Thu, Jan 24, 2019 at 11:38 AM Lavkesh Lahngir <[hidden email]>
> wrote:
>
>> Hello,
>> I mentioned in the first email.
>>
>> Version: 1.6.2, Commit ID: 3456ad0
>>
>> On Thu, Jan 24, 2019 at 12:33 AM Chesnay Schepler <[hidden email]>
>> wrote:
>>
>>> I suggest that you first tell me which version you are using so that I
>>> can a) reproduce the issue and b) check that this issue wasn't fixed in
>>> master or a recent bugfix release.
>>>
>>> On 23.01.2019 17:16, Lavkesh Lahngir wrote:
>>> > Actually, I realized my mistake that JarRunHandler is being used in the
>>> > jar/run API call.
>>> > And the changes are done in RestClusterClient.
>>> > The problem I was facing was that It always gives me "The main method
>>> > caused an error"
>>> > without any more details.
>>> > I am thinking when we throw ProgramInvocationException in
>>> PackagedProgram.
>>> > callMainMethod()
>>> > we should add exceptionInMethod.getMessage() too.
>>> >
>>> > *---
>>> >
>>> a/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
>>> >
>>> > *+++
>>> >
>>> b/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java*
>>> >
>>> > @@ -543,7 +543,7 @@ public class PackagedProgram {
>>> >
>>> >                          } else if (exceptionInMethod instanceof
>>> > ProgramInvocationException) {
>>> >
>>> >                                  throw (ProgramInvocationException)
>>> > exceptionInMethod;
>>> >
>>> >                          } else {
>>> >
>>> > -                               throw new
>>> ProgramInvocationException("The
>>> > main method caused an error.", exceptionInMethod);
>>> >
>>> > +                               throw new
>>> ProgramInvocationException("The
>>> > main method caused an error.: " + exceptionInMethod.getMessage(),
>>> > exceptionInMethod);
>>> >
>>> >                          }
>>> >
>>> >                  }
>>> >
>>> >                  catch (Throwable t) {
>>> >
>>> > What will you suggest?
>>> >
>>> > On Wed, Jan 23, 2019 at 7:01 PM Chesnay Schepler <[hidden email]>
>>> wrote:
>>> >
>>> >> Which version are you using?
>>> >>
>>> >> On 23.01.2019 08:00, Lavkesh Lahngir wrote:
>>> >>> Or maybe I am missing something? It looks like the JIRA is trying to
>>> >> solve
>>> >>> the same issues I stated 🤔
>>> >>> In the main method, I just threw a simple new Exception("Some
>>> message")
>>> >> and
>>> >>> I got the response I mentioned from the rest API.
>>> >>>
>>> >>> Thanks.
>>> >>>
>>> >>> On Wed, Jan 23, 2019 at 2:50 PM Lavkesh Lahngir <[hidden email]>
>>> >> wrote:
>>> >>>> Hello,
>>> >>>> The change in FLINK-10312
>>> >>>> <https://issues.apache.org/jira/browse/FLINK-10312> makes REST
>>> response
>>> >>>> of the API
>>> >>>> <
>>> >>
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jars-jarid-run
>>> >
>>> >> not
>>> >>>> very informative. It strips the stack trace and returns a generic
>>> >> message.
>>> >>>> People using flink-cluster deployment who do not have access to job
>>> >> manager
>>> >>>> logs, will not be able to figure out the root cause.
>>> >>>> In the case of when the job submission fails,
>>> >>>> In 1.6.2, I get
>>> >>>> {
>>> >>>>       "errors": [
>>> >>>>
>>>  "org.apache.flink.client.program.ProgramInvocationException:
>>> >> The
>>> >>>> main method caused an error."
>>> >>>>       ]
>>> >>>> }
>>> >>>>
>>> >>>> Is there a plan to improve error messages sent to the client?
>>> >>>> Is somebody working on this already?
>>> >>>>
>>> >>>> Thanks in advance.
>>> >>>> ~Lavkesh
>>> >>>>
>>> >>
>>>
>>>