Currently in the MemCheckpointStreamFactory.checkSize() method, it can throw an IOException with:
throw new IOException( "Size of the state is larger than the maximum permitted memory-backed state. Size=" + size + " , maxSize=" + maxSize + " . Consider using a different state backend, like the File System State backend."); But this will happen even if you’re using the File System State backend, right? This came up here: https://stackoverflow.com/questions/50149005/ioexception-size-of-the-state-is-larger-than-the-maximum-permitted-memory-backe <https://stackoverflow.com/questions/50149005/ioexception-size-of-the-state-is-larger-than-the-maximum-permitted-memory-backe> So shouldn’t it suggest using the RocksDB state backend instead? — Ken -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra |
Agree it is definitely not intuitive trying to figure out what to do based
on this message. I think the message should be changed to "please consider increasing maximum permitted memory size, increase task manager parallelism, or using a non-memory-based state backend". Could you please open a JIRA ticket in: https://issues.apache.org/jira/projects/FLINK/issues/ Thanks, Rong On Thu, May 3, 2018 at 7:56 PM, Ken Krugler <[hidden email]> wrote: > Currently in the MemCheckpointStreamFactory.checkSize() method, it can > throw an IOException with: > > throw new IOException( > "Size of the state is larger than > the maximum permitted memory-backed state. Size=" > + size + " , > maxSize=" + maxSize > + " . Consider > using a different state backend, like the File System State backend."); > > But this will happen even if you’re using the File System State backend, > right? > > This came up here: https://stackoverflow.com/questions/50149005/ > ioexception-size-of-the-state-is-larger-than-the-maximum- > permitted-memory-backe <https://stackoverflow.com/questions/50149005/ > ioexception-size-of-the-state-is-larger-than-the-maximum- > permitted-memory-backe> > > So shouldn’t it suggest using the RocksDB state backend instead? > > — Ken > > -------------------------- > Ken Krugler > +1 530-210-6378 > http://www.scaleunlimited.com > Custom big data solutions & training > Flink, Solr, Hadoop, Cascading & Cassandra > > |
> On May 3, 2018, at 9:06 PM, Rong Rong <[hidden email]> wrote: > > Agree it is definitely not intuitive trying to figure out what to do based > on this message. > I think the message should be changed to "please consider increasing > maximum permitted memory size, increase task manager parallelism, or using > a non-memory-based state backend". > > Could you please open a JIRA ticket in: > https://issues.apache.org/jira/projects/FLINK/issues/ Sure, see https://issues.apache.org/jira/browse/FLINK-9300 <https://issues.apache.org/jira/browse/FLINK-9300> — Ken > On Thu, May 3, 2018 at 7:56 PM, Ken Krugler <[hidden email]> > wrote: > >> Currently in the MemCheckpointStreamFactory.checkSize() method, it can >> throw an IOException with: >> >> throw new IOException( >> "Size of the state is larger than >> the maximum permitted memory-backed state. Size=" >> + size + " , >> maxSize=" + maxSize >> + " . Consider >> using a different state backend, like the File System State backend."); >> >> But this will happen even if you’re using the File System State backend, >> right? >> >> This came up here: https://stackoverflow.com/questions/50149005/ >> ioexception-size-of-the-state-is-larger-than-the-maximum- >> permitted-memory-backe <https://stackoverflow.com/questions/50149005/ >> ioexception-size-of-the-state-is-larger-than-the-maximum- >> permitted-memory-backe> >> >> So shouldn’t it suggest using the RocksDB state backend instead? >> >> — Ken -------------------------------------------- http://about.me/kkrugler +1 530-210-6378 |
Hi Ken and Rong,
That's a good point. The FSStateBackend holds all state in the JVM heap as well. The name is also not well chosen, IMO. Thanks for creating the JIRA ticket! Fabian 2018-05-05 0:52 GMT+02:00 Ken Krugler <[hidden email]>: > > > On May 3, 2018, at 9:06 PM, Rong Rong <[hidden email]> wrote: > > > > Agree it is definitely not intuitive trying to figure out what to do > based > > on this message. > > I think the message should be changed to "please consider increasing > > maximum permitted memory size, increase task manager parallelism, or > using > > a non-memory-based state backend". > > > > Could you please open a JIRA ticket in: > > https://issues.apache.org/jira/projects/FLINK/issues/ > > Sure, see > > https://issues.apache.org/jira/browse/FLINK-9300 < > https://issues.apache.org/jira/browse/FLINK-9300> > > — Ken > > > On Thu, May 3, 2018 at 7:56 PM, Ken Krugler <[hidden email] > > > > wrote: > > > >> Currently in the MemCheckpointStreamFactory.checkSize() method, it can > >> throw an IOException with: > >> > >> throw new IOException( > >> "Size of the state is larger than > >> the maximum permitted memory-backed state. Size=" > >> + size + " , > >> maxSize=" + maxSize > >> + " . Consider > >> using a different state backend, like the File System State backend."); > >> > >> But this will happen even if you’re using the File System State backend, > >> right? > >> > >> This came up here: https://stackoverflow.com/questions/50149005/ > >> ioexception-size-of-the-state-is-larger-than-the-maximum- > >> permitted-memory-backe <https://stackoverflow.com/questions/50149005/ > >> ioexception-size-of-the-state-is-larger-than-the-maximum- > >> permitted-memory-backe> > >> > >> So shouldn’t it suggest using the RocksDB state backend instead? > >> > >> — Ken > > -------------------------------------------- > http://about.me/kkrugler > +1 530-210-6378 > > |
Free forum by Nabble | Edit this page |