Because Python only runs signal handlers on the main thread, native
calls (IO or accept(2)) can delay these handlers running. This is
especially problematic when one such call is completely stuck and
we're trying to dump the stack to diagnose the issue.
By running the actual worker's processing in a sub-thread and leaving
the main thread sleeping, worker processes should always be able to
handle signals.
closesodoo/odoo#30688
Due to the implementation of PEP-475[1] in Python 3.5, revision
e98e8e9b1b used the recommended technique
of wakeup file descriptors in order to detect interruption of sleep() and
select() by signals, when running in multi-process mode (workers > 0).
The technique works well, however the initial patch never bothered to
read the byte that is written to the wakeup pipe when a signal is
processed. This does not matter when the signal is meant to shut down
the server, but it matters when the signal is SIGQUIT: it simply prints
thread dumps, and continues operating normally.
As a consequence, that byte remains in the wakeup pipe forever, causing
all the subsequent select/sleep calls to return immediately because
the wakeup fd *is* already ready. The symptom was that worker
processes would start cycling their main run loop very fast after
receiving a SIGQUIT signal, eating 100% CPU.
This patch ensures we always empty the wakeup pipe after an interruptible
call, to avoid this effect. It also refactors another occurrence of the
"empty pipe" pattern, in the PreforkServer, and incidentally uses readable
aliases for the file descriptors bound to the ends of the wakeup pipe
(wakeup_fd_r / wakeup_fd_w).
[1] https://www.python.org/dev/peps/pep-0475closesodoo/odoo#28356
From times to times, warning are seen on the runbot during HttpCase
tests with the chrome headless browser.
Those warning are about Odoo trying to join remaining requests threads.
In the dumpstack, the thread seems blocked in the werkzeug
handle_one_request method, when trying to read the HTTP request line.
One explanation could be that Chrome opens a pre-connect socket for
a future use. When the HttpTest cleans the browser, the page stops
loading but (probably) keeps the socket open for a while.
That could explain the problem.
With this commit, a timeout is set on the request handler,
in the hope that it closes the pre-connect socket too.
Previously broken symlinks would cause a FileNotFoundError
exception in the filesystem watcher thread spawned when using the
--dev=reload argument.
This problem appeared when using emacs to edit python files while
they are being watched by an odoo instance for changes. Emacs
creates a broken symlink as lockfile in the same directory as the
python file that it edits.
This commit makes the filesystem watcher silently ignore any
FileNotFoundError that occurs when it tries to open a file.
Fixes #21214, closes#21215
Before this commit, saving a file being watched by watchdog
(--dev=reload) with vim (and possibly other text editors) would
crash the server via a KeyboardInterrupt error.
This happens because vim, emacs cause multiple filesystem events to
happen when saving a single file, the FSWatcher registers these events
as multiple SIGHUPs which in turn calls restart() multiple times in a
very small amount of time which makes the KeyboardInterrupts completely
shut down the server
This is fixed by only calling restart() if odoo.phoenix is False, since
catching a SIGHUP sets it to true.
Fixes#21178
Special thanks to @odony for the help :)
Do not reload when a temporary Python file is
added/updated/reloaded.
This is to be able to use editors using isolation when saving file.
On save, they create a temporary file which is then moved on the target file.
The temporary file exists for milliseconds only,
and when the change was detected by watchdog and the server was reloaded,
the file already no longer existed on load.
It leaded to `No such file or directory` after the reload.
gevent 1.3.0 removed backward compatibility on gevent.wsgi.
make the new path the default
While the recommanded version on requirements.txt is 1.1, a server can not
launched with the 1.3
Closes#24780Fixes#24779
When the runbot spawns an Odoo instance, a CPU TIME limit is set.
When this limit is reached, the kernel kills the Odoo instance and no
message appears in the logs leaving the dev in a deep smoke cloud.
With this commit, the signal sent by the kernel when the soft
limit is reached (SIGXCPU), is intercepeted and Odoo shutdown gracefully with a
nice message for the dev.
Purpose: When running tests, all the tests for the installed/updated
files are done. This commit adds a 'tagged' decorator that can be used to
tag tests. Combined with a new 'test-tags' CLI option, it adds the ability
to filter which tests are executed. For example, @tagged('slow') will
add a tag 'slow' to the test. The CLI option 'test-tags="slow"' will
only run tests tagged 'slow'.
One can use prefixes to select cases with tags.
'+' or no prefix means that the tests tagged with this tag are selected
for execution. '-' prefix will exclude the tests tagged with this tag.
Exclusion takes precedence over inclusion.
Also, by default, all Odoo tests cases are tagged 'standard' and with
the technical name of the module.
This means that when selecting tests with the 'test-tags'
parameter, if '-standard' is not specified, all tests tags are
going to be executed.
When tagging tests, one can remove such automatic tag by prefixing the
tag name with '-'. E.g. @tagged('-standard') will remove the standard
tag from the test.
Another example, if one wants to test the 'sale' module alone,
even without adding any 'tagged' decorator thos tests can be selected
like that: --test-tags="sale"
Tests are selected or deselected using a TagsSelector. When instanciated,
a string is passed with comma separated tests selectors like
'+slow,-standard'. When the 'check' method is called with a test as argument,
it returns True or False if the test has to be executed or not.
Before this commit, saving a file being watched by watchdog
(--dev=reload) with vim (and possibly other text editors) would
crash the server via a KeyboardInterrupt error.
This happens because vim, emacs cause multiple filesystem events to
happen when saving a single file, the FSWatcher registers these events
as multiple SIGHUPs which in turn calls restart() multiple times in a
very small amount of time which makes the KeyboardInterrupts completely
shut down the server
This is fixed by only calling restart() if odoo.phoenix is False, since
catching a SIGHUP sets it to true.
Fixes#21178
Special thanks to @odony for the help :)
The method `_process_jobs` provides an alternative API to process the cron
jobs, that does not catch exceptions and log them. This makes the processing
of cron jobs easier for scripting.
Python 3 (before 3.6) is affected by outstanding issue 24291:
https://bugs.python.org/issue24291
In Python 3.6 the issue is gone thanks to this patch:
https://bugs.python.org/issue26721
The base StreamRequestHandler class does not properly deal with raw IO
output streams. This affects most HTTP servers that use it, including
werkzeug. It causes large HTTP responses to be truncated when the
output socket is unbuffered and in non-blocking mode.
This is the case in multi-process mode (workers > 0) due to the socket
timeout configured to avoid worker deadlocks.
Typically, large file downloads (2MB+) and large JSON responses will be
truncated and cause corrupted files or errors on the client side.
As a workaround, we turn on the buffering on the output stream by
default in multi-process mode (it is already turned on for the input
stream) when running on Python 3.5.
See also the socket.makefile() method and the docstring of
`socketserver.StreamRequestHandler` at
https://github.com/python/cpython/blob/dcb101e7f078f12fc3d2bf1730410798a880bfe3/Lib/socketserver.py#L707Fixes#20158
opw-777120
The 'xmlrpc'-based configuration parameters have been a misnomer since
the introduction of the generic HTTP service, years ago.
Hide these options from the server parameters, and replace them with
more appropriate 'http' ones:
--xmlrpc-interface -> --http-interface
--xmlrpc-port -> --http-port
--no-xmlrpc -> --no-http
The config entries for these have been adapted as well.
The old parameter names are still silently supported in both
command-line arguments and config files. However they are
stored with the new names in the `tools.config` dict,
and when saving config files (with the -s option).
Also clarified and cleaned up the descriptions of the HTTP/WEB server
parameters.
And finally, added a short version `-p`, for the `--http-port` option.
Credits to @dreispt for this (via #19518)
Closes#19518Closes#19778
This is an attempt at fixing the long timeout occurring on P3 when
trying to stop the server with Ctrl-C, both in multithreaded and
multi-process mode.
The root cause is the implementation of PEP-0475[1] as of Python 3.5,
which make system calls silently resume after an interrupt.
With this PEP, interrupting a syscall now requires the signal handler to
raise an exception, which get propagated in places where they did not
use to occur (e.g. time.sleep() used to return early when interrupted)
The solution handles 2 separate cases:
1. Multithreaded
In multithreaded mode, stopping the server requires interrupting
the main thread, while ensuring that all other threads are daemon ones.
The previous commit fixed the latter.
Interrupting the main thread is easy, because it simply sleeps until
it gets interrupted. And it already catches KeyboardInterrupt for
win32 quirks. All we need to do is thus to raise a KeyboardInterrupt
in the signal handler, as proposed in PEP 475 (Use Case 2)
2. Multi-process
In multi-process mode, stopping the server requires interrupting the
main thread of each process. The signal is already propagated to all
processes, so it should be as simple as for multithreaded mode.
Unfortunately, the main threads are far from idle in this case,
and raising a KeyboardInterrupt from the signal handler causes a
series of ugly tracebacks before stopping the server.
Effective, but frightening for the admin.
Instead of doing that, we can watch the wakeup file descriptor
(also suggested in PEP 475, Use Case 2) during the sleep() part of the
main loop of each worker. An interrupt signal won't therefore stop
the server immediately, but will wait until each worker is done with
their current request.. even better.
For select() syscalls, this is done with an extra pipe fd that is passed in
the "read fds", and used as wakeup fd.
And for sleep() syscalls, that don't support a wakeup fd, we can
replace them with an equivalent select() call on the wakeup fd, and
a timeout.
The combination of the above seems to restore a relatively fast
interrupt shutdown on both Python 2.7 and Python 3.5, in both
multithreaded and multi-process mode, with no surprising traceback.
[1] https://www.python.org/dev/peps/pep-0475/
When re-launching Odoo on an existing database in Python 3.6[0] it's
possible that the scheduler will preempt the main thread
mid-module-import and run the cron thread which causes registry (and
module) initialisation with only a subset of the Python files loaded
and thus e.g. tries loading auth_crypt with a registry in which none
of res has been loaded makes for interesting situations.
Delay spawning the cron thread until *after* the registry preload at
least, though I don't think that actually entirely fixes the issue.
[0] and probably in any Python >= 3.3, I'm guessing it's related to
the per-module import locks which replaced the old global lock:
https://docs.python.org/3.3/whatsnew/3.3.html#a-finer-grained-import-lock
though since ir.cron has little reason to import anything it could
also just be Python 3 scheduler changes which allow/cause a
context-switch during imports while the Python 2 scheduler would
not allow that. IDK.
As of P3 socket.error is replaced by IOError and the underlying error
code must be accessed with `.errno `. This also worked in P2, so simply
use that all the time.
Christophe Simonis