Nico Kruber created FLINK-15068:
-----------------------------------
Summary: Disable RocksDB's local LOG by default
Key: FLINK-15068
URL:
https://issues.apache.org/jira/browse/FLINK-15068 Project: Flink
Issue Type: Improvement
Components: Runtime / State Backends
Affects Versions: 1.9.1, 1.8.2, 1.7.2
Reporter: Nico Kruber
Assignee: Nico Kruber
Fix For: 1.10.0
With Flink's default settings for RocksDB, it will write a log file (not the WAL, but pure logging statements) into the data folder. Besides periodic statistics, it will log compaction attempts, new memtable creations, flushes, etc.
A few things to note about this practice:
# *this LOG file is growing over time with no limit (!)*
# the default logging level is INFO
# the statistics in there may help looking into performance and/or disk space problems (but maybe you should be looking and monitoring metrics instead)
# this file is not useful for debugging errors since it will be deleted along with the local dir when the TM goes down
With a custom \{{OptionsFactory}}, the user can change the behaviour like the following:
{code:java}
@Override
public DBOptions createDBOptions(DBOptions currentOptions) {
currentOptions = super.createDBOptions(currentOptions);
currentOptions.setKeepLogFileNum(10);
currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
currentOptions.setStatsDumpPeriodSec(0);
currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each
return currentOptions;
}{code}
However, the rotating logger does currently not work (it will not delete old log files - see [
https://github.com/dataArtisans/frocksdb/pull/12]). Also, the user should not have to write his own {{OptionsFactory}} to get a sensible default.
To prevent this file from filling up the disk, I propose to change Flink's default RocksDB settings so that the LOG file is effectively disabled (nothing is written to it by default).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
