A cursor created by a Connection object is not expected to be used with
environments; use registry.cursor() instead.
closesodoo/odoo#78143
X-original-commit: 6aeb73f10953f3547b7cf830718c02a3933df9e0
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The XML-RPC interface has a compatibility shim for binaries as
historically Odoo has returned "binary" data as base64 strings. To
avoid breakages during the Python 3 transition, the shim was
introduced to decode the output binary data (under the assumption that
it'd be ASCII-compatible).
In the case where the data is *not* ascii-compatible, however, it can
generate invalid XML documents: "C0" control codes (with the exception
of tab, LF, and CR) are not valid in XML 1.0 (which XML-RPC is an
application of), however they're perfectly valid string characters and
the standard library's marshaller does not check for them, embedding
them directly in the output document and breaking the client's
decoding.
Work around the issue by replacing such binary data with an empty
string.
While at it, move the bytes shim to the customized marshaller, this
way everything's at the same place and it's not necessary to waste
time trying to understand why the marshaller is just not calling what
it's supposed to call.
Fixes#61919closesodoo/odoo#75973
Forward-port-of: #75952
Forward-port-of: #74699
X-original-commit: 1a0b3f7
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Refactor the Environments object into a Transaction object, which is
bound to one cursor, and is no longer shared among several cursors.
The following methods/properties have been changed:
- Environment.envs no longer works (because of the design change);
- Environment.manage() is deprecated (no longer useful);
- Environment.reset() is now an instance method;
- env.clear_upon_failure() is deprecated in favor of cr.savepoint().
closesodoo/odoo#75598
Related: odoo/enterprise#20451
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
Before this commit the PyInotify filesystem watcher used by the code
autoreload feature (`--dev=reload`) would not get a chance to free
it's inotify watches before the reexec, hence at each reexec triggered
by a code reload the inotify watches where accumulated until potentially
reaching the kernel limit `fs.inotify.max_user_watches`.
This patch ensures that inotify properly closes it's file descriptor
before we reexec:
https://github.com/dsoprea/PyInotify/blob/f77596a/inotify/adapters.py#L79closesodoo/odoo#71302
X-original-commit: 8703ff1e3d9be6f2f5fce2e8c4e62589b05133fb
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Creating a database from the Odoo CLI miserably fails with the error
"psycopg2.ProgrammingError: set_session cannot be used inside a
transaction".
Setting con.autocommit = True fails if some transaction is already
started, which is the case when creating a database. The fix consists
in rolling back the existing transaction (with only a SELECT) before
switching to autocommit.
closesodoo/odoo#68549
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Following 7a235c19ff, use an alternative
API to the method autocommit(). Several functions managing databases
use a connection in autocommit mode to execute some commands outside of
a transaction.
closesodoo/odoo#68491
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
We have some tests in odoo/upgrade that are sensitive to the order on
which they are executed. Specifically: IntegrityCase tests need to be
run after all UpgradeCase tests across all Odoo modules.
To support this we implemented a sorting mechanism for tests based on
the test_sequence class attribute. This is intended to be used by meta
cases, not by individual tests.
closesodoo/odoo#66521
Related: odoo/upgrade#2184
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Start odoo in threading mode with bus installed. Login in the browser
using any internal user. Make sure the browser call the
/longpolling/poll uri. While the browser is waiting for a response, stop
the server. The server takes up to 50 seconds to stop.
When started in threading mode, a request to /longpolling/poll is served
by a casual http thread. It searches for messages enqueued in the bus
and returns them. If there are no message for the user in the queue yet,
it creates a `threading.Event`, attach it to the user in a shared
dictionnary and `wait()` on it with a timeout of 50 seconds (hardcoded
value). When the bus thread (the one responsible to listen on the
database) receives new messages, it `set()` the events which resume any
http thread that was waiting.
Because when we stop the server, there is no way to server new requests,
there are no way new messages arrive in the bus. All the threads that
were waiting for a new message will just wait until the event timeouts
which slow down the shutdown of the server.
Now we actively `set()` all events in order to resume all those workers
when we stop the server.
The `ImDispatch.poll` signature has been changed too so it is possible
to change (via code) the hardcoded default. The function was using the
object referenced by `TIMEOUT` at the time the function was defined,
using `timeout None` then `if None: timeout=TIMEOUT` ensures we lookup
the variable.
closesodoo/odoo#64530
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
Introduce a way to schedule the execution of cron jobs *soon*. Triggered
jobs are included in the next execution batch.
Heavy refactor of the `ir.cron` model so the various parallel queries
use the (not so new) `SKIP LOCKED` postgresql select option which skip
rows that are locked instead of throwing an exception like `NOWAIT`
would do. Various methods has been renamed and the overall selection,
execution and update of job records have been re-architectured.
The cron workers can now to wake up early via a notification on the
`cron_trigger` channel of the meta `postgres` database.
closesodoo/odoo#62124
Task: 2368911
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Old naming has been deprecated for a long time, it's been removed
entirely in 3.9 (bpo-37804).
X-original-commit: c4a94bc99285aba085badeb1a41e8fe04562ef50
Before this fix, trying to authenticate via
xml-rpc call from PHP following the documentation at
odoo.com/documentation/14.0/webservices/odoo.html#logging-in
raised an error:
> $uid = $common->authenticate($db, $username, $password, array());
TypeError: 'list' object is not a mapping
Because PHP doesn't have separate array and mapping types, the
XML-RPC encoder disambiguates based on the existence of
key => value pairs, such disambiguation yields an empty xmlrpc
array for an empty PHP array, which is unexpected on the Python
side.
Relax the check on the Python side:
* fixing this on the client side requires adding arbitrary and
meaningless key => value to the empty array to force the
correct disambiguation which is ugly and weird
* the example code has been there for a long time, so there's
probably lots of such PHP code in the wild
opw-2388141
closesodoo/odoo#62101
X-original-commit: 9f97b7435977c81a079004b7c2186ce75526490c
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
`registry._init_modules` is a set so its iteration order is
non-deterministic (it's randomised on interpreter initialisation
unless PYTHONHASHSEED is provide through the environment). This can
lead to annoying non-deterministic behavior: while the non-determinism
is only at the module level, it's easy enough for modules to have
python-level side-effects (e.g. patch methods, update globals, ...),
which may only be surfaced by an other module executing after them,
but not if said module executes before.
By sorting the modules we should make this much more reliable one way
or another.
closesodoo/odoo#60028
X-original-commit: f9169a468a2328a691ec4f32233ba3bad3622282
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
While individual addons paths are normalised, module and resource
paths are not.
As a result, when symlinking modules into directories on the
addons path, the path of test modules is only half-normalized: it's
normalised up to the addon path (which likely did not need it in that
setup) but not above that.
This is an issue when using `--test-file`, because that path is fully
normalised, and so the path of the provided test file and that of the
corresponding test module will not match, leading to the tests
unexpectedly not getting run.
Normalize the test module's path before the comparison, using the same
routing used for --test-file.
closesodoo/odoo#59822
X-original-commit: 536809662e542994451be793cd09e87adbf31776
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
odoo/odoo#53938 improved the SQL linter and used psycopg2.sql to
silence the linter where that still made sense. However I forgot to
mark table names (pretty much exclusively) as `sql.Identifier` in a
few somewhat rare callsites, which consequently break when invoked as
a simple string is not a Composable and psycopg2 therefore rejects it
when composing the query.
odoo/odoo#54556 fixed a few mis-updated ones, but apparently I still
managed to miss one here.
closesodoo/odoo#56191
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Leftover from testing a post-load report of all test failures.
Intent was to provide test failure details either during or after
loading, but feature was not actually developed and I forgot to remove
this bit before merging.
closesodoo/odoo#56186
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
* remove useless OdooTestRunner
* don't log results & time per-file, log a module-level tally instead
* add number of tests to post-test results
* generate a single test suite per module (see note)
* use the previous item to split out the at_install test-running in
two steps: generating the suite for the module then running that
suite, this way for modules which have no test, or for
which all tests have been deselected by test tags, we can avoid some
of the setup necessary to prepare for running tests but possibly
quite expensive (e.g. `setup_models`)
Note: single test suite per module
I wanted to stop creating a test result for (essentially) every file
in the module, however because of the class-level ``addCleanup``, a
TestResult can't be reused by independent suites:
In order to run class-level cleanup, the test suite checks between
tests if the test it's *preparing* to run is in the same class as the
last test it ran, and if not applies the class-level cleanup.
The problem is that the "previous test class" is stored on the result
object, which is never cleaned up, and the "between tests" check is
really performed *before each test*.
This means when reusing results across suites it will run the
class-level cleanup at the end of one suite and immediately at the
start of the next, which will cause issues if class-level cleanups are
not idempotent (thankfully ``TestTestCursor`` has a non-idempotent
``tearDownClass` which let me discover the error).
Possible fixes are:
* don't reuse results
* clear the relevant states / attributes between suites
* put individual suites in a Big Suite for running
The latter seems simpler: just create a single suite for the entire
odoo-level module instead of creating one suite per test module.
Note to the note: the case of nested suite is taken in account, the
"end of suite" cleanup only runs at the end of the top-level suite, so
technically we don't have to unwrap suites for *that* purpose, we're
doing so in order to filter the test cases inside the suites. But
maybe we could integrate this feature to the suites themselves...
closesodoo/odoo#55185
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
That's a not-very-useful subset of OdooTestResult, so:
* make results merge-able (aka add ability to update a result with the
contents of another)
* remove support for test data files, and transmission of the
assertion report thing through the data-files loading
* replace "legitimate" uses of assertion report by test result
* have run_unit_tests manipulate and return a result instead of weird
flags & ternaries
`struct_rusage.ru_utime` and `struct_rusage.ru_stime` are float
seconds.
`setrlimit()` takes a tuple of *integers*, and recent versions of
Python have started triggering warnings:
DeprecationWarning: an integer is required (got type
float). Implicit conversion to integers using __int__ is
deprecated, and may be removed in a future version of Python.
Convert the soft cpu time limit to an integer explicitly to suppress
the warning.
closesodoo/odoo#49710
Signed-off-by: Damien Bouvy (dbo) <dbo@odoo.com>
Before this, invalidations to the UID cache is not synchronised
between workers because it's an ad-hoc solution (so a user changing
their password or an admin disabling a user would only lock out an
attacker currently using the API of one of possibly several
workers). Shift the entire thing to ormcache which already has proper
support for synchronising cache invalidation between workers.
Also simplify the cache invalidation mess in Users.write because the
caches have been unified into a single registry-level LRU, so the
half-dozen cache clears on specific ormcached methods & models is
pretty much the same as repeatedly calling clear_caches on the current
model.
**However** registry.cache is trivially accessible from server actions
and safe_eval as long as they provide access to a model (through
`model.pool.cache`). Which is common, and an issue given we're very
much putting sensible data in there.
Fix this by renaming `Registry.cache` to `Registry.__cache`, this
requires few editions and mangled names are not accessible from
safe_eval contexts.
The alternative would have been to add more bespoke handling of the
uid cache to hook it into the cache invalidation propagation
machinery.
After discussion with (@)odony, fixing LRU access and using that seems
cleaner and less error-prone.
Note on lazy_property
=====================
Make Registry.cache / Registry.__cache into a regular attribute: the
overhead of the LRU is not that high (compared to that of the registry
itself), it's rare that we *don't* need it, and it's assumed to be a
persisted attribute (it's not just a cache) so making it a normal
attribute seems fine; and lazy_property doesn't work for mangled
names: the name of the property is mangled using the name of the
definition class, but the name of the symbol (fget) is not mangled so
lazy_property would set the __cache attribute but then Python would
lookup _Registry__cache, creating a new cache every access.
And we can't (always) mangle things correctly on `__get__(obj,
owner)`: `owner` is just `type(obj)`, meaning in the case of
inheritance the type we get is the type through which the property is
accessed rather than the one it's defined on. So it would work in the
cases where no inheritance is involved (such as Registry.__cache) but
not in general (lest we want to play around walking the MRO ourselves
to find the definition source, which doesn't seem worth it).
lazy_property *could* be made to work properly on Python 3.6+: the
descriptor protocol gains `__set_name__(name, owner)`, which is called
with the properly mangled name — and with the definition class to boot
(though there might still be issues when overriding lazy properties as
the override will be mangled & named differently... or maybe that's a
feature?). However we're still supporting 3.5 at this point, AFAIK, so
that's not an option. Plus it feels unnecessary / not very useful.
However add an assertion to `lazy_property` so it signals when we try
to use it on a mangled method (as otherwise it kinda sorta work in the
sense that the property / object is accessible but is in effect a
slower way to write a regular property).
Allows accessing various keys, especially whether this is an
interactive login or not.
Also have the xml-rpc `login` delegate to `authenticate` instead of
having its own half-assed implementation.
And remove some dead code: as far as I can tell, Session.authenticate
is never called with a uid.
Currently there are a few issues with testing reporting (at the
command-line):
1. there is a global report after at_install tests, but it gets
"scrolled off" by long post_install tests, and is thus easy to
miss
2. with test tags, it's easy to fat finger a typo and run 0 tests,
which look like everything's running fine (no failure)
To improve this, print a global report at shutdown (in
`--stop-after-init` mode if tests are enabled) which recapitulates the
test results *and prints a warning if no tests were run at all*.
Also update the reporting collection to make this more reliable:
* have `run_unit_tests` return `None` if it has run no tests, the
assertion reporting machinery counts this as neither success nor
failure which is exactly what we want
* have load_test only report a success *if files were actually
loaded* (by having `load_data` return that information)
Task 2301268
closesodoo/odoo#54812
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
odoo/odoo#53938 improved the SQL linter and used psycopg2.sql to
silence the linter where that still made sense. However I forgot to
mark table names (pretty much exclusively) as `sql.Identifier` in a
few somewhat rare callsites, which consequently break when invoked as
a simple string is not a Composable and psycopg2 therefore rejects it
when composing the query.
closesodoo/odoo#54556
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
The linter would miss / fail to warn on injection of *local variables*
in some cases.
Try to improve it to be stricter and more reliable, after discussion
with odo, sql which is "correctly" dynamic should use psycopg2's sql
package in order to bypass the linter (bonus: it should also properly
escape & quote identifiers).
closesodoo/odoo#53938
Related: odoo/enterprise#11718
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Using a few regex like
\((_\(.*%s.*)(\) % )([\w\[\]][\w .\[\]\(\)'"]*)\)
($1, $3))
Old syntax is still compatible but starts the migration to the new
syntax that catches error.
As indicated in the comment, it's much preferred to perform response
buffering at the reverse proxy level than to increase the socket
timeout. It will free up HTTP workers for other requests faster, while
the proxy does the work of buffering the stream on disk as needed.
/!\ The timeout is also used to protect from accidental DoS effects
in situations of low worker availability, due to idle connections
caused e.g. by wkhtmltopdf's connection pooling.
Setting a high timeout will make the protection less effective, so
ensuring you have enough free HTTP workers at all times becomes critical.
In our tests with nginx's defaut buffering on a typical hardware with
SSD storage, buffering up to 1GB responses did not require any change
of the socket timeout on the Odoo side, though your mileage may vary.
See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config
directives.
OPW-2247730
See also: #20158closesodoo/odoo#51982
X-original-commit: d78ea126b8d2a72ae626880b0ec64bb7a39a07ac
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Those were deprecations implemented in 0.15
* all middlewares have been moved from `werkzeug.wsgi` to
`werkzeug.middleware`, including the `SharedDataMiddleware` we use
* ProxyFix was moved to werkzeug.middleware.proxy_fix, this had
already been fixed but I forgot the import
* sessions support was moved to a separate package
(`pallets/secure-cookies`), however while distros are starting to
update werkzeug to 1.0 (e.g. done on Arch, and in Debian
Experimental) they're not bundling secure-cookies so using a
vendored version seems like the least bad thing we can do, even more
so as conditional dependencies are not really a thing (e.g. even
with just pip we can't depend on secure-cookie iff werkzeug >= 1.0)
Also non-browser jsonrpc (as it goes through a similar process): for
internal performance reasons, name_search and read_group have been
converted to a *lazy* name_get, so the "display name" is not
unnecessarily computed.
However this is an issue for the RPC endpoints (/xmlrpc and /jsonrpc)
as they have no support for `lazy` and thus tend to blow up and / or
do the wrong thing when trying to output a lazy:
* xmlrpc has no way to handle lazy at all and straight blows up
* jsonrpc falls back to `json_default` so they try to stringify the
lazy, which might have worked except
*Problematically* both endpoints delegate the actual work to
`dispatch_rpc` which handles dispatching between various services and
ultimately creates a *new* cursor before calling model
methods (`object` service and `execute`/`execute_kw`).
This means by the time the result is serialized to be output, the
lazy's cursor has long been closed, and thus any access to an
unevaluated `lazy` errors out when trying to fetch the underlying
item.
This also means we can't just add a hook to serialize the lazy
in the xmlrpc marshaller, though we do have to do that. We *also* (for
both xmlrpc and jsonrpc) have to force evluation of lazy values before
our cursor is closed, meaning it has to be done right after the method
is invoked, iterating the entire response.
Related to task 2170343
closesodoo/odoo#49286
X-original-commit: e2b5a359c1d5eccbe725c1c3169b4130d7bca49b
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
TL;DR: remember `osv` and `except_orm` ? You can forget about them.
* Deprecated `except_orm` dropped.
* `UserError` elevated as super type of all user-related
errors.
* Unused `DeferredException` dropped.
* Unused `QWebException` dropped (real one is in `qweb.py`).
* `MailDeliveryException` made a python exception.
* `name` legacy exception attribute made an alias of the python standard
`args[0]` attribute and deprecated.
* `value` legacy exception attribute dropped.
* `exception_type` RPC error response key dropped.
* Deprecated `osv` module dropped.
* `--osv-memory-age-limit` cli option made an alias of
`--transient-age-limit` and deprecated.
The `odoo.exceptions.Warning` have long been a deprecated alias to
`UserError`. It is going to be removed in a future version but first we
explicitly deprecate it with a warning.
The `odoo.exceptions.DeferredException` was a very old internal
exception, it has been removed without deprecation notice as it is never
raised.
The `odoo.exceptions.except_orm` has been a deprecated exception type
with deprecation warning for 5 years, it has been removed in favor of
UserError which becomes the super class of all user-related errors.
The `odoo.base.models.ir_mail_server.MailDeliveryException` was
inheriting `except_orm`. As it is not related to a user error but is
more of a problem an admin much take care of, the exception has been
made a Python error.
The `exception_type` JSON key in RPC error responses was holding an
hardcoded value derived from the exception type. Its usage has been
dropped in favor of the `name` JSON key that holds the precise exception
name. Again as it was hardly used in the source code (beside the crash
manager) it has been dropped without deprecation warning.
Since we are here trying to clean odoo custom exceptions, we are also
deprecating the `name` exception attribute in favor of the more standard
`args[0]` attribute.
The `name` (along with `value`) were two attributes used to raise
`except_orm` exceptions before the introduction of `UserError`,
`AccessError` and related exceptions. The `name` attribute, at the time,
was holding the exception type/title. Nowadays it contains the error
message. The `value` attribute, at the time, was holding the error
message. Nowadays it is no more used.
The `osv` module contains very old deprecated aliases. There is no
simple way to log a deprecation warning for osv, osv_memory and
osv_abstract but as they have not been in use for ages, they have been
removed too. To be consistent, the `--osv-memory-age-limit` cli option
has been made a deprecated alias to the `--transient-age-limit`.
closesodoo/odoo#45723
Task: 2187728
Related: odoo/enterprise#9162
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Iteration methods on LRU were removed because they were not
thread-safe and it's not clear that making them thread-safe is the
correct thing to do, so not providing them seems saner.
I thought I'd looked for usages of the LRU but apparently didn't look
hard enough as I missed that it's used by the cron workers (apparently
using the threaded server we only run crons for dbs currently living
in the registry cache, the more you know).
Convert these to iterating on the LRU's internal mapping, and also
don't iterate on the LRU to clear its entries one by one when we can
just clear the entire thing safely, although Registry.delete_all
really seems completely unused.
closesodoo/odoo#49023
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Avoid logging an error in the logs for an operation that is not an
closesodoo/odoo#48957
Error: if the extension already exists, this is a success.
X-original-commit: 2f884942c94a640ae5107ff2acb27ba4ecc60bc0
Signed-off-by: Damien Bouvy (dbo) <dbo@odoo.com>
Before this commit, a lot of leftover import shims existed in the
codebase for py2-py3 compatibility, these are no longer needed since
Odoo 13.0+ doesn't support Python 2 anymore and is (finally) in EOL.
With this commit, these shims are dropped, making the code cleaner,
easier to read and with one less dependency.
Queue -> queue -> py2-py3 compatibility
xmlrpclib -> xmlrpc.client -> py2-py3 compatibility
ConfigParser -> configparser -> py2-py3 compatibility
itertools.izip_longest -> itertools.zip_longest -> py2-py3 compatibility
urllib -> urllib.request -> py2-py3 compatibility
__builtins__ -> builtins -> py2-py3 compatibility
_winreg -> winreg -> py2-py3 compatibility
mock -> unittest.mock -> merged into CPython
The debian/fedora packages and requirements.txt have been updated accordingly
closesodoo/odoo#44601
Related: odoo/enterprise#8141
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
The unaccent extension was installed only at restore, not at creation
of a database.
Make both consistent.
closesodoo/odoo#46534
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Sort of but not really, this commit fixes a special case in which
launching a --test-file of a file with at least two SavepointCases would
create a postgresql deadlock and it would be impossible to terminate the
Odoo process without sending a SIGKILL or waiting for the lock to
timeout.
This was introduced at #39368 and happens because of the way that
unittests unwraps suites, to keep it short, when it unwraps the custom
OdooSuite class internally, it ends up with a vanilla TestSuite with
which to run the different test cases, and since #39368 depends on the
overrides added to OdooSuite to function, the class cleanups are not
triggered at the end of a test class (rollback, cache cleanups, env
reset, registry reset, etc.).
The fix is to manually unwrap the suite of tests to keep OdooSuite as
the suite with which to call the tests, which was already done for
--test-enable (although for different reasons, --test-tags?) which is
why --test-enable didn't have any problems.
This commit also fixes a typo I found on the backport, which meant
classCleanups were not being executed if the setUpClass failed, but it
had no effect on classCleanups during tearDownClass.
Task-ID 2160398
Depends on #43135closesodoo/odoo#43296
X-original-commit: 7a5ded7d40afc29043d356b5dece0dbe1fbd5ab3
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
An http-port provided on the command line (may also have been an issue
for config files, didn't check) would not be taken in account anymore,
because `odoo.tests.common` would be imported during the import of
`odoo` itself (when loading odoo.service.server), itself importing
`odoo.tools.config` leading to a default configuration being set up.
* remove `odoo.tests.common.PORT`, `config['http_port']` should be
used always
* defer the import of odoo.tests.common by moving it inside
load_test_file
* stop generating default configs
closesodoo/odoo#43283
X-original-commit: 45871f498ea4cf3ada719692e69cd413883ab442
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Before this commit nothing prevented high concurrency on a threaded http
server to consume too much resources, ending up failing requests either
because the OS is unable to spawn that many threads
(`RuntimeError: can't start new thread`), either because the Odoo db
connection pool is full (`PoolError: The Connection Pool Is Full`).
This commit adds the ODOO_MAX_HTTP_THREADS environment variable which
allows to limit the amount of concurrent socket connections accepted by
a threaded server, implicitly limiting the amount of concurrent threads
running for http requests handling.
Note that if a value has been provided to ODOO_MAX_HTTP_THREADS that cannot
be parsed as an integer, a value will be automatically set to half the
db connection pool size (which defaults to 64). This dynamic value is
chosen because while most requests will borrow only one cursor
concurrently, there are some exceptions where some controllers might
allocate two or more cursors.
closesodoo/odoo#42327
X-original-commit: d42a951369ee87b50e834f73cd2af5cf031d43f7
Signed-off-by: Christophe Simonis <chs@odoo.com>
glibc's malloc() uses arenas [1] in order to efficiently handle memory
allocation of multi-threaded applications. This allows better memory
allocation handling in case of multiple threads that would be using
malloc() concurrently [2].
Due to the python's GIL, this optimization have no effect on
multithreaded python programs. Unfortunately, a downside of creating one
arena per cpu core is the increase of virtual memory which Odoo is based
upon in order to limit the memory usage for threaded workers.
On 32bit systems the default size of an arena is 512K while on 64bit
systems it's 64M [3], hence a threaded worker will quickly reach it's
default memory soft limit upon concurrent requests. We therefore set the
maximum arenas allowed to 2 unless the MALLOC_ARENA_MAX env variable is
set.
This commit also brings the following changes:
- allow to disable the memory hard limit for all servers if the provided
value is 0 (instead of crashing)
- increase the log level for threaded server in case of limits reached
Note: Setting MALLOC_ARENA_MAX=0 allow to explicitely set the default
glibs's malloc() behaviour.
[1] https://sourceware.org/glibc/wiki/MallocInternals#Arenas_and_Heaps
[2] https://www.gnu.org/software/libc/manual/html_node/The-GNU-Allocator.html
[3] https://sourceware.org/git/?p=glibc.git;a=blob;f=malloc/malloc.c;h=00ce48c;hb=0a8262a#l862closesodoo/odoo#42323
X-original-commit: 85fe2c6e60f7f1f6ea72cb55e85f85d420ce6616
Signed-off-by: Christophe Simonis <chs@odoo.com>
A deadlock can occur between threads when concurrent requests
acquire the registry lock and conflicting database-level locks
in different orders. The database won't be able to detect and
break the deadlock because it involves an external, Python-level
lock. This situation is more likely to occur during module
installations [1].
If the server is started with the `limit_time_real` option,
it should be able to abort the deadlocked requests after the
timeout, and restart. However that could not work because
the recovery initiated by `reload()` is blocked at the end
of the `stop()` method, as it cannot acquire the registry
lock either, necessary for `Registry.delete_all()`.
Since that deletion step is in fact not necessary, it can
be skipped, avoiding the deadlock entirely.
Indeed there's no real reason anymore to delete the DB's
registry upon shutdown. This was introduced for 7.0 by
b5daffc115, in order to perform
other cleanups (including cron agent threads). These other
cleanups are not necessary anymore, and when the stop()
method of the ThreadedServer completes, the next step is
either a restart of the whole process (via execve() through
_reexec()), or a full process exit. Keeping the registry in
memory for a few cycles until this happens makes no difference.
When such a deadlock occurs, it's always possible to manually
kill the server with 2 `kill` commands, or 1 `kill -9`.
~~~~~~~~~~~~~~~~~~~~~~
[1] Reproduction info:
The following deadlock was observed in Odoo threaded server mode:
1. incoming request spawns a new thread A
A starts a transaction and does a "SELECT ... FROM res_users ..."
getting an ACCESS SHARE lock on the table
2. incoming request spawns a new thread B
B is a request that calls `button_immediate_install`, that will
install new modules and alter the res_users table.
3. B takes and holds the registry lock and executes "ALTER TABLE
res_users ...", that waits to get the ACCESS EXCLUSIVE lock on the
table until A's transaction releases the ACCESS SHARE lock.
4. A continues code execution and reaches a .sudo() call, it tries to
create a new environment. The creation of the new environment
requires to wait for the registry's lock to be release but it's held
by B.
-> A waits for B's registry lock to be released
-> B waits for A's ACCESS SHARE lock to be released
-> Deadlock that can't be broken except by force-killing the server
closesodoo/odoo#40664
X-original-commit: 9e67525418b3b0a48a796044ef24b227946ceb8f
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
This commit partially backports bpo-24412, which allows the definition of
class cleanups (addClassCleanup) and module cleanups (omitted),
similar to instance cleanups (addCleanup).
This is useful for tests that override unittest's setUpClass and
could crash during its execution: If this happens, it is possible that a
bunch of crap is left in the database or even worse, the cursor becomes
completely fucked; Thanks to the addClassCleanup, we can undo the damage
done by the setUpClass.
Another benefit is that it is called unconditionally after tearDownClass
is called, so it can also be called as a replacement and/or safer
tearDownClass.
odoo/odoo#30688 (6ce2d6efb5) added an
indirection in prefork workers: Python-level signal handlers are
delayed until native calls have ended (e.g. accept() or
execute()). Running the actual work in a sub-thread allowed the main
thread to handle signals in all cases.
However there is apparently an issue with SIGXCPU on linux (possibly
other cases as well): SIGXCPU is delivered to the child thread (if
possible?) and Thread.join apparently stops it from redelivered to
the main thread (Thread.join is signal-interruptible since 3.2 but
possibly not Python-interruptible).
Blocking SIGXCPU on the child thread causes the OS to deliver on the
main thread and fixes the issue.
Also split set_limits so it sets the signal handler in the parent
thread but properly updates the soft limit in the child after each
request, as the goal is to put a hard limit on the CPU time per
request, not on the worker. 6ce2d6ef would set the limit once then
never update it, likely cycling workers more than desired.
While at it:
* block other signals with a handler set, they seem to work
regardless on linux but other OS may have a different way of
dispatching process-directed signals
* unset signals which are set by the prefork server but whose
set behavior makes no sense in workers:
- TERM and CHLD were already unset
- HUP is used to restart the server, workers can just be killed
- TTIN and TTOU configure the number of workers
closesodoo/odoo#39731
X-original-commit: 549bd199bad269e4e28efac933efac3f41495877
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
* walksymlinks is useless, os.walk got followlinks in 2.6
* tempdir is redundant with tempfile.TemporaryDirectory added in 3.2
* listdir(recursive=False) is just os.listdir (I guess recursive=True
is somewhat useful)
* zip_dir is *not* redundant with shutil.make_archive:
- zip_dir handles the archived root differently
- zip_dir sorts files, make_archive sorts directories
- make_archive explicitly adds entries for directories (including
empty), sorts directories, doesn't sort files; zip_dir sorts
files (including with a custom key function) but ignores directories
- zip_dir filters out a bunch of trash files
closesodoo/odoo#39573
Signed-off-by: Christophe Simonis <chs@odoo.com>
Werkzeug 0.15 modified ProxyFix such that by default it only forwards
the REMOTE_ADDR when enabled, whereas before 0.15 it would also
forward scheme and host. This breaks proxied odoo as the base url
becomes incorrect (cf #34412).
Use properly configured ProxyFix when running with werkzeug 0.15, old
configuration otherwise.
Backport of 4057227def which was merged
in master, because many people apparently run 0.15 now.
Closes#35085closesodoo/odoo#36212closesodoo/odoo#37708
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
So the logs contain some indication as to what was exceeding the limits.
closesodoo/odoo#37303
X-original-commit: 0a266f444a0abda026d09ad88024dca1c50c92b9
Signed-off-by: Denis Vermylen <Icallhimtest@users.noreply.github.com>
omission in d26e253edd
kill -3 (SIGQUIT) is not processed like the other signals so it doesn't
go through this bit of code, but leaving the AttributeError lurking
there is not a good idea.
closesodoo/odoo#37221
X-original-commit: ac65ef0208a5ec814d263aeb44a98b47a5ee3947
Signed-off-by: Denis Vermylen <Icallhimtest@users.noreply.github.com>
Initial bug report, full of red herrings:
receive two support requests by email while on Odoo.sh, the first creates a
ticket, while the second replies to the server with the error:
550 Requested action not taken: mailbox unavailable
When a request is done, the service check function tries each request multiple
times when getting Operational errors.
When two email are received within a 'short' amount of time (e.g .1 second)
the two requests are initiated with a new cursor.
When both mails generate the creation of a new record of the same model,
the second cursor is going to fail because of an:
ERROR: could not serialize access due to concurrent update
However, the record we used in the computations needed in the creation of the
second record have been added to the environment.
When that second creation is retried, it will operate on the same records,
and find them in the todo, with the environment that was obtained for the first
trial. Since the cursor is closed, this crashes.
To explain the bug report, Odoo.sh transfers the mail through an xmlrpc
to process the mail.
Because of this crash, the server interprets the lack of well-formed response
by a generic 'mailbox unavailable'.
opw 2060476
closesodoo/odoo#37165
X-original-commit: 845eecdfa7b27d84862e94f4b63d40a851e0c410
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
[PEP-594] is deprecating the `imp` module, that module is used in
`module.py` in order to dynamically import addons using any of the
`odoo.addons` or `openerp.addons` import anchor.
We are deprecating `openerp` module/addons imports in v13 in order to
remove the support in v14 and greatly simplify how modules/addons are
loaded. If you are still using the old `import openerp` or `import
openerp.addons`, `import odoo` and `import odoo.addons` are drop-in
replacements.
The `odoo.modules.module.ad_paths` addon paths list has been deprecated
too. The list is now accessible on `odoo.addons.__path__` where they
are now directly loaded [2].
See also:
[PEP-594]: https://python.org/dev/peps/pep-0594/
[2]: https://packaging.python.org/guides/packaging-namespace-packages/closesodoo/odoo#36597
Task: 2003936
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Albin