This patches fixes the untested and broken draft of inetd and systemd
activation support in the threaded server.
This patch also introduces a change of behaviour in the `_reexec()`
function by keeping the environment when Odoo is respawning during the
following events:
- SIGHUP signal is received
- one click install has been triggered
- code reload needed when using `--dev=reload`
Purpose: When running tests, all the tests for the installed/updated
files are done. This commit adds a 'tagged' decorator that can be used to
tag tests. Combined with a new 'test-tags' CLI option, it adds the ability
to filter which tests are executed. For example, @tagged('slow') will
add a tag 'slow' to the test. The CLI option 'test-tags="slow"' will
only run tests tagged 'slow'.
One can use prefixes to select cases with tags.
'+' or no prefix means that the tests tagged with this tag are selected
for execution. '-' prefix will exclude the tests tagged with this tag.
Exclusion takes precedence over inclusion.
Also, by default, all Odoo tests cases are tagged 'standard' and with
the technical name of the module.
This means that when selecting tests with the 'test-tags'
parameter, if '-standard' is not specified, all tests tags are
going to be executed.
When tagging tests, one can remove such automatic tag by prefixing the
tag name with '-'. E.g. @tagged('-standard') will remove the standard
tag from the test.
Another example, if one wants to test the 'sale' module alone,
even without adding any 'tagged' decorator thos tests can be selected
like that: --test-tags="sale"
Tests are selected or deselected using a TagsSelector. When instanciated,
a string is passed with comma separated tests selectors like
'+slow,-standard'. When the 'check' method is called with a test as argument,
it returns True or False if the test has to be executed or not.
Before this commit, saving a file being watched by watchdog
(--dev=reload) with vim (and possibly other text editors) would
crash the server via a KeyboardInterrupt error.
This happens because vim, emacs cause multiple filesystem events to
happen when saving a single file, the FSWatcher registers these events
as multiple SIGHUPs which in turn calls restart() multiple times in a
very small amount of time which makes the KeyboardInterrupts completely
shut down the server
This is fixed by only calling restart() if odoo.phoenix is False, since
catching a SIGHUP sets it to true.
Fixes#21178
Special thanks to @odony for the help :)
The method `_process_jobs` provides an alternative API to process the cron
jobs, that does not catch exceptions and log them. This makes the processing
of cron jobs easier for scripting.
Python 3 (before 3.6) is affected by outstanding issue 24291:
https://bugs.python.org/issue24291
In Python 3.6 the issue is gone thanks to this patch:
https://bugs.python.org/issue26721
The base StreamRequestHandler class does not properly deal with raw IO
output streams. This affects most HTTP servers that use it, including
werkzeug. It causes large HTTP responses to be truncated when the
output socket is unbuffered and in non-blocking mode.
This is the case in multi-process mode (workers > 0) due to the socket
timeout configured to avoid worker deadlocks.
Typically, large file downloads (2MB+) and large JSON responses will be
truncated and cause corrupted files or errors on the client side.
As a workaround, we turn on the buffering on the output stream by
default in multi-process mode (it is already turned on for the input
stream) when running on Python 3.5.
See also the socket.makefile() method and the docstring of
`socketserver.StreamRequestHandler` at
https://github.com/python/cpython/blob/dcb101e7f078f12fc3d2bf1730410798a880bfe3/Lib/socketserver.py#L707Fixes#20158
opw-777120
The 'xmlrpc'-based configuration parameters have been a misnomer since
the introduction of the generic HTTP service, years ago.
Hide these options from the server parameters, and replace them with
more appropriate 'http' ones:
--xmlrpc-interface -> --http-interface
--xmlrpc-port -> --http-port
--no-xmlrpc -> --no-http
The config entries for these have been adapted as well.
The old parameter names are still silently supported in both
command-line arguments and config files. However they are
stored with the new names in the `tools.config` dict,
and when saving config files (with the -s option).
Also clarified and cleaned up the descriptions of the HTTP/WEB server
parameters.
And finally, added a short version `-p`, for the `--http-port` option.
Credits to @dreispt for this (via #19518)
Closes#19518Closes#19778
This is an attempt at fixing the long timeout occurring on P3 when
trying to stop the server with Ctrl-C, both in multithreaded and
multi-process mode.
The root cause is the implementation of PEP-0475[1] as of Python 3.5,
which make system calls silently resume after an interrupt.
With this PEP, interrupting a syscall now requires the signal handler to
raise an exception, which get propagated in places where they did not
use to occur (e.g. time.sleep() used to return early when interrupted)
The solution handles 2 separate cases:
1. Multithreaded
In multithreaded mode, stopping the server requires interrupting
the main thread, while ensuring that all other threads are daemon ones.
The previous commit fixed the latter.
Interrupting the main thread is easy, because it simply sleeps until
it gets interrupted. And it already catches KeyboardInterrupt for
win32 quirks. All we need to do is thus to raise a KeyboardInterrupt
in the signal handler, as proposed in PEP 475 (Use Case 2)
2. Multi-process
In multi-process mode, stopping the server requires interrupting the
main thread of each process. The signal is already propagated to all
processes, so it should be as simple as for multithreaded mode.
Unfortunately, the main threads are far from idle in this case,
and raising a KeyboardInterrupt from the signal handler causes a
series of ugly tracebacks before stopping the server.
Effective, but frightening for the admin.
Instead of doing that, we can watch the wakeup file descriptor
(also suggested in PEP 475, Use Case 2) during the sleep() part of the
main loop of each worker. An interrupt signal won't therefore stop
the server immediately, but will wait until each worker is done with
their current request.. even better.
For select() syscalls, this is done with an extra pipe fd that is passed in
the "read fds", and used as wakeup fd.
And for sleep() syscalls, that don't support a wakeup fd, we can
replace them with an equivalent select() call on the wakeup fd, and
a timeout.
The combination of the above seems to restore a relatively fast
interrupt shutdown on both Python 2.7 and Python 3.5, in both
multithreaded and multi-process mode, with no surprising traceback.
[1] https://www.python.org/dev/peps/pep-0475/
When re-launching Odoo on an existing database in Python 3.6[0] it's
possible that the scheduler will preempt the main thread
mid-module-import and run the cron thread which causes registry (and
module) initialisation with only a subset of the Python files loaded
and thus e.g. tries loading auth_crypt with a registry in which none
of res has been loaded makes for interesting situations.
Delay spawning the cron thread until *after* the registry preload at
least, though I don't think that actually entirely fixes the issue.
[0] and probably in any Python >= 3.3, I'm guessing it's related to
the per-module import locks which replaced the old global lock:
https://docs.python.org/3.3/whatsnew/3.3.html#a-finer-grained-import-lock
though since ir.cron has little reason to import anything it could
also just be Python 3 scheduler changes which allow/cause a
context-switch during imports while the Python 2 scheduler would
not allow that. IDK.
As of P3 socket.error is replaced by IOError and the underlying error
code must be accessed with `.errno `. This also worked in P2, so simply
use that all the time.
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
Indeed, running the testsuite is not supported in multi-process (worker/pre-fork) mode, for multiple reasons: no multiple/concurrent execution of tests, necessity for in-memory cross-request statefulness (test cursor), etc.
We have no plan to work around these issues, as there seems to be no strong use case.
Our testsuite is mainly focused on the business/framework aspects, and not scalability/deployment aspects - which would require a much more complicated test infrastructure.
``import odoo.addons.foo as bar`` doesn't seem to properly trigger the
import hook in Python 3 (it blows up on decimal_precision and
base). Thus convert *all* examples of that pattern to the more
sensible ``from odoo.addons import foo as bar``.
In Python 2, to raise a new exception but reuse a traceback requires a
special form of ``raise`` (``raise etype, evalue, tb``).
In Python 3, this is now done via a ``with_traceback`` method on
exception objects.
However this requires a bit of trickery as the former is invalid
syntax in Python 3, hence pycompat bridge created via an exec for
Python 2.
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
The GeventServer process was not enforcing the
configured memory limits. In the unlikely event
of a service handling hundreds of concurrent
longpolling requests on hundreds of different
databases, this could render the Registry LRU
ineffective in keeping a low memory footprint.
Even after being evicted from the LRU, these
hundreds of registries would still be referenced
by their in-progress requests, so the memory
would not be freed. An Odoo 10 registry can
typically take 15MB of memory, so this could
become a problem.
Move the post-install tests execution outside `Registry.new`, and add a flag on
class `HttpCase` to enable/disable the registry "test mode".
This allows a test to run actual transactions that will reload the registry,
which may be used to test the creation of `ir.model` instances, etc.
* use a special arg in odoo.py
* note that the previous implementation was relying on the fact that
the openerp-gevent file was located in the same directory than the
odoo.py file. This was no longer true since rev a0eb172cab
This commit brings back this behavior.
Fabien Meghazi