[IMP] warn when using logrotate in multiprocess
The built-in log rotation is *very* unreliable in a multiprocess context, which can lead to significant data loss (up to hours of log missing) right after rotation[0][1]. Technically, according to the cookbook even filehandler does not support multiprocess logging[2]. Either way, we should at the very least warn that logrotate is a Bad Idea in multi-workers scenario. Ideally we'd just fall back on regular file/stream logging but that likely could & would break existing deployments, so maybe implement that for master instead? Alternatively: implement it properly in multi-worker scenario with only the "root" process actually logging to whatever output was set up, and all workers sending stuff to log to it via a queue or network socket or w/e. [0] https://stackoverflow.com/questions/34186774/why-doesnt-timedrotatingfilehandler-work-properly-and-how-to-solve-this-issuer [1] https://github.com/odoo/odoo/issues/16566 [2] https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
This commit is contained in:
@@ -144,6 +144,13 @@ def init_logger():
|
||||
if dirname and not os.path.isdir(dirname):
|
||||
os.makedirs(dirname)
|
||||
if tools.config['logrotate'] is not False:
|
||||
if tools.config['workers'] > 1:
|
||||
# TODO: fallback to regular file logging in master for safe(r) defaults?
|
||||
#
|
||||
# Doing so here would be a good idea but also might break
|
||||
# situations were people do log-shipping of rotated data?
|
||||
_logger.warn("WARNING: built-in log rotation is not reliable in multi-worker scenarios and may incur significant data loss. "
|
||||
"It is strongly recommended to use an external log rotation utility or use system loggers (--syslog) instead.")
|
||||
handler = logging.handlers.TimedRotatingFileHandler(filename=logf, when='D', interval=1, backupCount=30)
|
||||
elif os.name == 'posix':
|
||||
handler = logging.handlers.WatchedFileHandler(logf)
|
||||
|
||||
Reference in New Issue
Block a user