Stephan Ewen created FLINK-22752:
------------------------------------
Summary: Add robust default state configuration to StateFun
Key: FLINK-22752
URL:
https://issues.apache.org/jira/browse/FLINK-22752 Project: Flink
Issue Type: Sub-task
Reporter: Stephan Ewen
We aim to reduce the state configuration complexity by applying a default configuration with robust settings, based on lessons learned in Flink.
*(1) Always use RocksDB.*
_That is already the case._
We keep this for now, as long as the only other alternative are backends with Objects on the heap, which are tricky in terms of predictable JVM performance. RocksDB has a significant performance cost, but more robust behavior.
*(2) Activate local recovery by default.*
That makes recovery cheao for soft tasks failures and gracefully cancelled tasks.
We need to set these options:
- {{state.backend.local-recovery: true}}
- {{taskmanager.state.local.root-dirs: <dir>}} - some local directory that will not possibly be wiped by the OS periodically, so typically some local directory that is not {{/tmp}}, for example {{/local/state/recovery}}.
- {{state.backend.rocksdb.localdir: <dir>}} - a directory on the same FS / device as above, so that one can create hard links between them (required for RocksDB local checkpoints), for example {{/local/state/rocksdb}}.
Flink will most likely adopt this as a default setting as well in the future.
It still makes sense to pre-configer a different RocksDB working directory than {{/tmp}}.
*(3) Activate partitioned indexes by default.*
This may cost minimal performance in some cases, but can avoid massive performance regression in cases where the index blocks no longer fit into the memory cache (may happen more frequently when there are too many ColumnFamilies = states).
Set {{state.backend.rocksdb.memory.partitioned-index-filters: true}}.
See FLINK-20496 for details.
*(4) Increase number of transfer threads by default.*
This speeds up state recovery in many cases. The default value in Flink is a bit conservative, to avoid spamming DFS (like HDFS) by default. The more cloud-centric StateFun setups should be safe to use higher default value.
Set {{state.backend.rocksdb.checkpoint.transfer.thread.num: 8}}.
*(5) Increase RocksDB compaction threads by default.*
The number of RocksDB compaction threads is frequently a bottleneck.
Increasing it costs virtually nothing and mitigates that bottleneck in most cases.
{{state.backend.rocksdb.thread.num: 4}} (this value is chosen under the assumption that there is only one slot).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
