During tests on runbot, the main reason tours are slow to start is
because the first loading of "/web" need to generate assets bundles.
Generation can take up to 5 seconds time the number of tour.
Before this, the first requests to /web takes around 7 seconds on
runbot and the next ones less than 1 second.
After this commit, the first request to /web takes around 1.5 seconds
Note that the main difficulty is to choose when to generate the assets.
(With optimisations from following commits) the generation time is
arround 30 seconds (21 css + 12 js) the first time, for all modules.
If the attachements exists the generation time is arround 3 seconds
(2.5 js + 0.5 css) mainly because of globs to find usefull files.
In practice for runbot the ideal would be to generate them at
the end of the install so that it is shared for all post_install builds.
Doint it at the end of an install is not wanted in all cases, saas
pregenerated templates and upgrade may avoid doing that.
Doing it in a special runbot step (subcommand) is not possible for niglty
execution (no split, in one go). This needs to be in the code.
It is also not practical for devs wanting to have the same behaviour on
a local machine.
A solution to make the test conditionnal at install is to base the
condion on an existing test_module. test_assetsbundle is a good
candidate here. On runbot, the at_install step (befor split) will
automatically generate assets with this solution.
Assets are also generated before post_install tests. This should be fast
if they already exists and ensure that assets are up to date after
modifying sources locally or after downloading a database on runbot.
It should be fast enough for small database and may be even faster
in the future for small diffs.
Part-of: odoo/odoo#99176
In order for websocket upgrade to work with firefox, the http version
must be set to `HTTP/1.1`.
Until now, this was done in the `send_response` method. The issue is
that this method is also used when sending an error. This is problematic
because we use environ to know whether or not the version should be changed
which means any error during `BaseHTTPRequestHandler.parse_request` (such as
wrong http version) would have led to an AttributeError being raised.
In order to solve this issue, this modification is done when making environ.
Moreover, we previously used the request uri to know whether or not the version
should be changed, this was not really reliable, we now use the upgrade header
for this purpose.
closesodoo/odoo#99535
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
We added trigram index for char fields since https://github.com/odoo/odoo/pull/83015.
But if `unaccent` is installed in the database (and isn't force to
`False` on field), these new trigram indexes are pointless and cost a
lot for nothing (almost nothing, it still can be used for equality
operator but in this case a btree will be far more efficient).
The simple way to fix it is to add `unaccent(<column>)` in the index
trigram definition, but unfortunately `unaccent` is not immutable and
may therefore not be indexed. In order to make `unaccent` indexable, we
must declare it as immutable (see
https://stackoverflow.com/questions/11005036/does-postgresql-support-accent-insensitive-collations/11007216#11007216
for more information and how to do that).
With this patch, trigram indexes are created with `unaccent(<column>)`
if the function `unaccent` is available in the database, and for the
fields that are not declared with `unaccent=False`. Moreover, we issue
a warning when `unaccent` is available but is not immutable, in which
case most trigram indexes will be useless.
odoo/upgrade#3736
task-2551518
closesodoo/odoo#95943
Signed-off-by: Rémy Voet <ryv@odoo.com>
Currently dbs can only be managed via the UI in order to take
filestores in account: while it's possible to load/copy/rename/drop
databases via `psql`, that will not manage the related filestores so
the result of the operation is incomplete DBs and leftover filestores
littering the disk.
Seems like a good idea to add a CLI to perform the same
tasks. Currently the CLI calls into the corresponding service, rather
than both calling into (possibly better designed) unified APIs, but
that seems fine for an initial version.
The top-level `db` command acts as a db manager, with git-style
sub-sub-commands for the various operations:
- `load` to load a dump file into a database (with a specified name or
not)
- `dump` to dump a local db to a zip dump (pg_dump can be created via
the corresponding command so not a concern)
- `duplicate` and `rename`
- `drop` in order to drop both the database itself and the
corresponding filestore
Notably, `create` is currently left out because a database can
trivially be created by invoking odoo using a dbname which doesn't
exist, so doesn't seem useful.
`list` is also left out, because `psql -l` generally does the job.
closesodoo/odoo#97365
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit is part of the websocket integration in Odoo.
The longpolling port does not make sense anymore: longpolling has been dropped.
This commit deprecate the `--longpolling-port` option and replace it by the
`--gevent-port` option.
closesodoo/odoo#75510
Related: odoo/enterprise#23184
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
This commit is the first commit of the websocket integration in Odoo.
It focuses on the implementation of the websocket protocol as per RFC6455.
The implementation is tested thanks to the autobahn test suite.
A config parameter is available to customize the websocket connection:
- websocket_keep_alive_timeout (default 600): Integer specifying how
many seconds a websocket connection should be kept alive
Part-of: odoo/odoo#75510
It's really not a general purpose utility, it's just a way of forcing
a lazy collection's evaluation.
Alternatively, add a helper for this to / alongside `lazy`? Note that
the lazy object(s) may not be the top-level.
closesodoo/odoo#98023
Related: odoo/upgrade#3776
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
They're not used much and they don't seem much of an advantage
compared to just calling `subprocess` with the discovery utility
functions.
While at it, fix `dump_db` to not unnecessarily have an open stdin to
`pg_dump`.
Part-of: odoo/odoo#98023
Uses a dedicated logger (for easier filtering / silencing) for
results output, and provides rough (module-level) stats in INFO but
detailed (test-level) in DEBUG.
Also updates the global query counter (`odoo.sql_db.query_counter`) to
update after each query rather than on close: with test cursors the
actual underlying counter is only rarely flushed.
closesodoo/odoo#95420
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When attempting to run the Odoo server on macOS Monterey (currently
12.4), it crashes with the following traceback:
```
Traceback (most recent call last):
File "/Users/app/Code/odoo/odoo-bin", line 8, in <module>
odoo.cli.main()
File "/Users/app/Code/odoo/odoo/cli/command.py", line 61, in main
o.run(args)
File "/Users/app/Code/odoo/odoo/cli/server.py", line 179, in run
main(args)
File "/Users/app/Code/odoo/odoo/cli/server.py", line 173, in main
rc = odoo.service.server.start(preload=preload, stop=stop)
File "/Users/app/Code/odoo/odoo/service/server.py", line 1342, in start
rc = server.run(preload, stop)
File "/Users/app/Code/odoo/odoo/service/server.py", line 553, in run
self.start(stop=stop)
File "/Users/app/Code/odoo/odoo/service/server.py", line 491, in start
set_limit_memory_hard()
File "/Users/app/Code/odoo/odoo/service/server.py", line 83, in set_limit_memory_hard
resource.setrlimit(rlimit, (config['limit_memory_hard'], hard))
ValueError: current limit exceeds maximum limit
```
Actually, this issue is not specific to Odoo but affects Python on macOS
as a whole (see [1] and [2]).
As our memory management is based on Linux - which is our primary
deployment target - and non-POSIX systems were already excluded, this
commit escapes the rlimit modification on non-Linux systems to prevent
this kind of issue.
References:
[1] https://bugs.python.org/issue34602
[2] https://github.com/python/cpython/pull/14546
Related issue:
https://github.com/odoo/odoo/issues/79112closesodoo/odoo#93381
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Stopping a threaded odoo server would spam the postgresql logs with
multiple:
<...> LOG: could not receive data from client: Connection reset by peer
Let's avoid being rude and not hang up on postgresql connections
unexpectedly.
closesodoo/odoo#89955
X-original-commit: d0da17b167edee82381b6add573742e1cf663a9c
Signed-off-by: Raphael Collet <rco@odoo.com>
Every request comes with a session, a dictionary that is persisted on
the filesystem and that saves various information such as the user
cart on the ecommerce.
When a user simply visits the website, a default session is created and
saved on disk, this bloats the filestore with many sessions. Creating
the session on-the-fly is cheaper than loading it from the filesystem.
With this work the default session is not saved on disk anymore unless
explicitly asked via `session.touch()`.
An exception to the statement "creating the session on-the-fly is
cheaper" is geoip, the ip geolocalization is not cheap. In this work,
geoip have been moved from http_routing/request.session.geoip to a
lazy property core/request.geoip. When requested the info is persisted
on the session. Like other keys from the default session, geoip will not
be persisted unless there is non-default stuff in the session.
Because the CSRF-TOKEN is based on the session-id, it is important the
session-id stays the same across multiples requests even when the
session is not persisted on disk. Even when a session is not persisted
on disk, the session-id cookie is still set so that the next session
created on-the-fly uses the same session-id.
Technical note regarding the session, it has been decided to drop the
session-snapshot protocol and to reintroduce a "modified" flag. It has
been decided not to use werkzeug's session (which natively comes with a
"modified" flag) and to keep our own session object. We decided to
extend MutableMapping instead of dict; using MutableMapping we only
have to override __setitem__ and __detitem__; using dict we would had to
override update()/pop()/... too.
Task: 2789035
Part-of: odoo/odoo#86015
When setting up the PG replication, the slave is considered in recovery
mode. However, `LISTEN / NOTIFY` is not supported in this mode, leading
to an endless loop of crashes.
In recovery mode, we simply deactivate the feature.
closesodoo/odoo#87619
X-original-commit: 6389a64953081130d62f2a1703a27ddfa751098b
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
This commit is the 14th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
* `request.uid = x` => `request.update_env(user=x)`.
* `request.context = x` => `request.update_env(context=x)`.
* `request.context = dict(request.context, x=y)`
=> `request.update_context(x=y)`.
* `request.cr = None` => `request.cr.close()`.
* `http.mono_db()` => `request.db`.
* `http.dispatch_rpc()` => `service.dispatch_rpc()`.
* `@service.model.check` => `service.model.retrying()`.
* `request.endpoint`
=> `env['ir.http']._match(request.httprequest.path)[0].endpoint`.
* `request.routing_iteration `=> `removed`.
* `request.jsonrequest` => `request.dispatcher.jsonrequest`.
Note that `request.params` is now set much later in the process. If you
are in a situation where you values from the query string or the
http body you can use `request.get_http_params()`.
Note that using the new `request.future_response`, it is possible to
add headers and cookies on the response object before the response
object is initialized. Please note that headers/cookies saved on
the future response will NOT be injected in case of error.
PR: odoo#78857
Task: 2571224
This commit is the 9th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals. See also [REF] core: HTTPocalypse (11) ir.http base model.
This is a two-part commit with "(11) ir.http base model". In this commit
we focus on the initialization of the various ORM objects, namely: the
registry, the cursor, the environment, the user and the context. In the
other n°11 commit we focus on the ir.http model and its relation to the
current http.py module.
One of the objectives of this comprehensive refactor was to ease the
cognitive complexity of the http framework, in other words to make it
simplier. One of the problem identified quite early during the
preparation of this work is the way the various ORM objects are
initialized, modified and cleaned during the request lifetime.
Before this work, all the ORM internals were lazily initialized via
properties. It is the first time one uses `request.cr` that a cursor is
opened to `request.db` and stored on `request._cr`. It is the first time
one uses `request.env` that an environment is create with the current
`request.user` and `request.context`. Upon user or context modification,
the current environment is discarded, the next usage of `request.env`
will create yet another environment on the fly using the modified user
and/or context.
Using this model, no ressource is initialized if not necessary. It is
possible for nodb-compatible endpoint to be served via the db-compatible
router and ir.http without ever opening a cursor to the database.
But this model is harder to reason about and ultimately to maintain.
In this work we propose to drop the lazy approach for a greedy one. In
this work the first steps of `_serve_db`, the db-compatible counter-part
of `_serve_nodb`, are dedicated to setup a registry, open a cursor to
the database and create an environment using the session's user and
context. In this work, when one wants to change the environ's user or
context, he must call `request.update_env` or `request.update_context`,
both method will recreate the environment *now* with the given values.
The downside of this approach is that resources are always allocated
even when it is not necessary. We argue that, in general, the
controllers that do not use the ORM are rare thus it is rare we allocate
unecessary resources. We also argue that the cognitive benefits are more
than welcome and that the new APIs will help at writing more robust
applications.
PR: odoo#78857
Task: 2571224
This commit is the 1st commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
Our application framework is quite complicated. There are a few explicit
middlewares: `ProxyFix`, `DisableCacheMiddleware`, `SharedDataMiddleware`.
There are many (13, in total) implicit ones in the form of
`ir.http._dispatch` overrides. There are a few frames that don't have
much values by themselves and are very much implementation details
there are: `application_unproxied`, `_call_function`, `checked_call` and
`EndPoint.__call__`.
The ORM initialisation is quite magical, the cursor, registry and
environment are all setup lazily thanks to class properties. The
context and uid of the environment are aliased on the request. The
abuse of properties makes it impossible to reason about where and when
the environment is actually instantiated and modified.
The dispatch of a request follow the next scheme:
`odoo.http/Root.dispatch` -> `odoo.addons.base.models/ir.http._dispatch`
-> `odoo.http/(Http|Json)Request.dispatch` -> `odoo.http/@route` ->
controller. Because the request becomes http or json specialized quite
late, the many ir.http override all need to parsimony try/except their
code in order to manually call `_handle_exception()` uppon error, they
cannot let the error bubble-up. This back-and-forth between odoo.http
and ir.http is source of some headaches.
Speaking about error, the `odoo.http/WebRequest._handle_exception`
implementation is quite complicated, it basically craft a new exception
out of the passed exception object in order to correct its traceback.
If the error had been bubbled-up instead of handled by
`_handle_exception()` such python hack would not have been necessary.
The objectives of this refactor are about refounding the technical dept:
* We want shorter error reporting;
* We want simpler request and ir.http APIs;
* We want better integration of all the ir.http extensions.
PR: odoo#78857
Task: 2571224
When setting the password in the database manager the config is saved
automatically in a odoorc file.
This can be problematic, especially when testing locally, with
specific options.
Going in the database manager and creating a new database or changing
admin pasword will save all those options.
This commit proposes to only save the admin_passwd when modified.
Part-of: odoo/odoo#82874
A cursor created by a Connection object is not expected to be used with
environments; use registry.cursor() instead.
closesodoo/odoo#78143
X-original-commit: 6aeb73f10953f3547b7cf830718c02a3933df9e0
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The XML-RPC interface has a compatibility shim for binaries as
historically Odoo has returned "binary" data as base64 strings. To
avoid breakages during the Python 3 transition, the shim was
introduced to decode the output binary data (under the assumption that
it'd be ASCII-compatible).
In the case where the data is *not* ascii-compatible, however, it can
generate invalid XML documents: "C0" control codes (with the exception
of tab, LF, and CR) are not valid in XML 1.0 (which XML-RPC is an
application of), however they're perfectly valid string characters and
the standard library's marshaller does not check for them, embedding
them directly in the output document and breaking the client's
decoding.
Work around the issue by replacing such binary data with an empty
string.
While at it, move the bytes shim to the customized marshaller, this
way everything's at the same place and it's not necessary to waste
time trying to understand why the marshaller is just not calling what
it's supposed to call.
Fixes#61919closesodoo/odoo#75973
Forward-port-of: #75952
Forward-port-of: #74699
X-original-commit: 1a0b3f7
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Refactor the Environments object into a Transaction object, which is
bound to one cursor, and is no longer shared among several cursors.
The following methods/properties have been changed:
- Environment.envs no longer works (because of the design change);
- Environment.manage() is deprecated (no longer useful);
- Environment.reset() is now an instance method;
- env.clear_upon_failure() is deprecated in favor of cr.savepoint().
closesodoo/odoo#75598
Related: odoo/enterprise#20451
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
Before this commit the PyInotify filesystem watcher used by the code
autoreload feature (`--dev=reload`) would not get a chance to free
it's inotify watches before the reexec, hence at each reexec triggered
by a code reload the inotify watches where accumulated until potentially
reaching the kernel limit `fs.inotify.max_user_watches`.
This patch ensures that inotify properly closes it's file descriptor
before we reexec:
https://github.com/dsoprea/PyInotify/blob/f77596a/inotify/adapters.py#L79closesodoo/odoo#71302
X-original-commit: 8703ff1e3d9be6f2f5fce2e8c4e62589b05133fb
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Creating a database from the Odoo CLI miserably fails with the error
"psycopg2.ProgrammingError: set_session cannot be used inside a
transaction".
Setting con.autocommit = True fails if some transaction is already
started, which is the case when creating a database. The fix consists
in rolling back the existing transaction (with only a SELECT) before
switching to autocommit.
closesodoo/odoo#68549
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Following 7a235c19ff, use an alternative
API to the method autocommit(). Several functions managing databases
use a connection in autocommit mode to execute some commands outside of
a transaction.
closesodoo/odoo#68491
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
We have some tests in odoo/upgrade that are sensitive to the order on
which they are executed. Specifically: IntegrityCase tests need to be
run after all UpgradeCase tests across all Odoo modules.
To support this we implemented a sorting mechanism for tests based on
the test_sequence class attribute. This is intended to be used by meta
cases, not by individual tests.
closesodoo/odoo#66521
Related: odoo/upgrade#2184
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Start odoo in threading mode with bus installed. Login in the browser
using any internal user. Make sure the browser call the
/longpolling/poll uri. While the browser is waiting for a response, stop
the server. The server takes up to 50 seconds to stop.
When started in threading mode, a request to /longpolling/poll is served
by a casual http thread. It searches for messages enqueued in the bus
and returns them. If there are no message for the user in the queue yet,
it creates a `threading.Event`, attach it to the user in a shared
dictionnary and `wait()` on it with a timeout of 50 seconds (hardcoded
value). When the bus thread (the one responsible to listen on the
database) receives new messages, it `set()` the events which resume any
http thread that was waiting.
Because when we stop the server, there is no way to server new requests,
there are no way new messages arrive in the bus. All the threads that
were waiting for a new message will just wait until the event timeouts
which slow down the shutdown of the server.
Now we actively `set()` all events in order to resume all those workers
when we stop the server.
The `ImDispatch.poll` signature has been changed too so it is possible
to change (via code) the hardcoded default. The function was using the
object referenced by `TIMEOUT` at the time the function was defined,
using `timeout None` then `if None: timeout=TIMEOUT` ensures we lookup
the variable.
closesodoo/odoo#64530
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
Introduce a way to schedule the execution of cron jobs *soon*. Triggered
jobs are included in the next execution batch.
Heavy refactor of the `ir.cron` model so the various parallel queries
use the (not so new) `SKIP LOCKED` postgresql select option which skip
rows that are locked instead of throwing an exception like `NOWAIT`
would do. Various methods has been renamed and the overall selection,
execution and update of job records have been re-architectured.
The cron workers can now to wake up early via a notification on the
`cron_trigger` channel of the meta `postgres` database.
closesodoo/odoo#62124
Task: 2368911
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Old naming has been deprecated for a long time, it's been removed
entirely in 3.9 (bpo-37804).
X-original-commit: c4a94bc99285aba085badeb1a41e8fe04562ef50
Before this fix, trying to authenticate via
xml-rpc call from PHP following the documentation at
odoo.com/documentation/14.0/webservices/odoo.html#logging-in
raised an error:
> $uid = $common->authenticate($db, $username, $password, array());
TypeError: 'list' object is not a mapping
Because PHP doesn't have separate array and mapping types, the
XML-RPC encoder disambiguates based on the existence of
key => value pairs, such disambiguation yields an empty xmlrpc
array for an empty PHP array, which is unexpected on the Python
side.
Relax the check on the Python side:
* fixing this on the client side requires adding arbitrary and
meaningless key => value to the empty array to force the
correct disambiguation which is ugly and weird
* the example code has been there for a long time, so there's
probably lots of such PHP code in the wild
opw-2388141
closesodoo/odoo#62101
X-original-commit: 9f97b7435977c81a079004b7c2186ce75526490c
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
`registry._init_modules` is a set so its iteration order is
non-deterministic (it's randomised on interpreter initialisation
unless PYTHONHASHSEED is provide through the environment). This can
lead to annoying non-deterministic behavior: while the non-determinism
is only at the module level, it's easy enough for modules to have
python-level side-effects (e.g. patch methods, update globals, ...),
which may only be surfaced by an other module executing after them,
but not if said module executes before.
By sorting the modules we should make this much more reliable one way
or another.
closesodoo/odoo#60028
X-original-commit: f9169a468a2328a691ec4f32233ba3bad3622282
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
While individual addons paths are normalised, module and resource
paths are not.
As a result, when symlinking modules into directories on the
addons path, the path of test modules is only half-normalized: it's
normalised up to the addon path (which likely did not need it in that
setup) but not above that.
This is an issue when using `--test-file`, because that path is fully
normalised, and so the path of the provided test file and that of the
corresponding test module will not match, leading to the tests
unexpectedly not getting run.
Normalize the test module's path before the comparison, using the same
routing used for --test-file.
closesodoo/odoo#59822
X-original-commit: 536809662e542994451be793cd09e87adbf31776
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
odoo/odoo#53938 improved the SQL linter and used psycopg2.sql to
silence the linter where that still made sense. However I forgot to
mark table names (pretty much exclusively) as `sql.Identifier` in a
few somewhat rare callsites, which consequently break when invoked as
a simple string is not a Composable and psycopg2 therefore rejects it
when composing the query.
odoo/odoo#54556 fixed a few mis-updated ones, but apparently I still
managed to miss one here.
closesodoo/odoo#56191
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Leftover from testing a post-load report of all test failures.
Intent was to provide test failure details either during or after
loading, but feature was not actually developed and I forgot to remove
this bit before merging.
closesodoo/odoo#56186
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
* remove useless OdooTestRunner
* don't log results & time per-file, log a module-level tally instead
* add number of tests to post-test results
* generate a single test suite per module (see note)
* use the previous item to split out the at_install test-running in
two steps: generating the suite for the module then running that
suite, this way for modules which have no test, or for
which all tests have been deselected by test tags, we can avoid some
of the setup necessary to prepare for running tests but possibly
quite expensive (e.g. `setup_models`)
Note: single test suite per module
I wanted to stop creating a test result for (essentially) every file
in the module, however because of the class-level ``addCleanup``, a
TestResult can't be reused by independent suites:
In order to run class-level cleanup, the test suite checks between
tests if the test it's *preparing* to run is in the same class as the
last test it ran, and if not applies the class-level cleanup.
The problem is that the "previous test class" is stored on the result
object, which is never cleaned up, and the "between tests" check is
really performed *before each test*.
This means when reusing results across suites it will run the
class-level cleanup at the end of one suite and immediately at the
start of the next, which will cause issues if class-level cleanups are
not idempotent (thankfully ``TestTestCursor`` has a non-idempotent
``tearDownClass` which let me discover the error).
Possible fixes are:
* don't reuse results
* clear the relevant states / attributes between suites
* put individual suites in a Big Suite for running
The latter seems simpler: just create a single suite for the entire
odoo-level module instead of creating one suite per test module.
Note to the note: the case of nested suite is taken in account, the
"end of suite" cleanup only runs at the end of the top-level suite, so
technically we don't have to unwrap suites for *that* purpose, we're
doing so in order to filter the test cases inside the suites. But
maybe we could integrate this feature to the suites themselves...
closesodoo/odoo#55185
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
That's a not-very-useful subset of OdooTestResult, so:
* make results merge-able (aka add ability to update a result with the
contents of another)
* remove support for test data files, and transmission of the
assertion report thing through the data-files loading
* replace "legitimate" uses of assertion report by test result
* have run_unit_tests manipulate and return a result instead of weird
flags & ternaries
`struct_rusage.ru_utime` and `struct_rusage.ru_stime` are float
seconds.
`setrlimit()` takes a tuple of *integers*, and recent versions of
Python have started triggering warnings:
DeprecationWarning: an integer is required (got type
float). Implicit conversion to integers using __int__ is
deprecated, and may be removed in a future version of Python.
Convert the soft cpu time limit to an integer explicitly to suppress
the warning.
closesodoo/odoo#49710
Signed-off-by: Damien Bouvy (dbo) <dbo@odoo.com>
Before this, invalidations to the UID cache is not synchronised
between workers because it's an ad-hoc solution (so a user changing
their password or an admin disabling a user would only lock out an
attacker currently using the API of one of possibly several
workers). Shift the entire thing to ormcache which already has proper
support for synchronising cache invalidation between workers.
Also simplify the cache invalidation mess in Users.write because the
caches have been unified into a single registry-level LRU, so the
half-dozen cache clears on specific ormcached methods & models is
pretty much the same as repeatedly calling clear_caches on the current
model.
**However** registry.cache is trivially accessible from server actions
and safe_eval as long as they provide access to a model (through
`model.pool.cache`). Which is common, and an issue given we're very
much putting sensible data in there.
Fix this by renaming `Registry.cache` to `Registry.__cache`, this
requires few editions and mangled names are not accessible from
safe_eval contexts.
The alternative would have been to add more bespoke handling of the
uid cache to hook it into the cache invalidation propagation
machinery.
After discussion with (@)odony, fixing LRU access and using that seems
cleaner and less error-prone.
Note on lazy_property
=====================
Make Registry.cache / Registry.__cache into a regular attribute: the
overhead of the LRU is not that high (compared to that of the registry
itself), it's rare that we *don't* need it, and it's assumed to be a
persisted attribute (it's not just a cache) so making it a normal
attribute seems fine; and lazy_property doesn't work for mangled
names: the name of the property is mangled using the name of the
definition class, but the name of the symbol (fget) is not mangled so
lazy_property would set the __cache attribute but then Python would
lookup _Registry__cache, creating a new cache every access.
And we can't (always) mangle things correctly on `__get__(obj,
owner)`: `owner` is just `type(obj)`, meaning in the case of
inheritance the type we get is the type through which the property is
accessed rather than the one it's defined on. So it would work in the
cases where no inheritance is involved (such as Registry.__cache) but
not in general (lest we want to play around walking the MRO ourselves
to find the definition source, which doesn't seem worth it).
lazy_property *could* be made to work properly on Python 3.6+: the
descriptor protocol gains `__set_name__(name, owner)`, which is called
with the properly mangled name — and with the definition class to boot
(though there might still be issues when overriding lazy properties as
the override will be mangled & named differently... or maybe that's a
feature?). However we're still supporting 3.5 at this point, AFAIK, so
that's not an option. Plus it feels unnecessary / not very useful.
However add an assertion to `lazy_property` so it signals when we try
to use it on a mangled method (as otherwise it kinda sorta work in the
sense that the property / object is accessible but is in effect a
slower way to write a regular property).
Allows accessing various keys, especially whether this is an
interactive login or not.
Also have the xml-rpc `login` delegate to `authenticate` instead of
having its own half-assed implementation.
And remove some dead code: as far as I can tell, Session.authenticate
is never called with a uid.
Currently there are a few issues with testing reporting (at the
command-line):
1. there is a global report after at_install tests, but it gets
"scrolled off" by long post_install tests, and is thus easy to
miss
2. with test tags, it's easy to fat finger a typo and run 0 tests,
which look like everything's running fine (no failure)
To improve this, print a global report at shutdown (in
`--stop-after-init` mode if tests are enabled) which recapitulates the
test results *and prints a warning if no tests were run at all*.
Also update the reporting collection to make this more reliable:
* have `run_unit_tests` return `None` if it has run no tests, the
assertion reporting machinery counts this as neither success nor
failure which is exactly what we want
* have load_test only report a success *if files were actually
loaded* (by having `load_data` return that information)
Task 2301268
closesodoo/odoo#54812
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
odoo/odoo#53938 improved the SQL linter and used psycopg2.sql to
silence the linter where that still made sense. However I forgot to
mark table names (pretty much exclusively) as `sql.Identifier` in a
few somewhat rare callsites, which consequently break when invoked as
a simple string is not a Composable and psycopg2 therefore rejects it
when composing the query.
closesodoo/odoo#54556
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
The linter would miss / fail to warn on injection of *local variables*
in some cases.
Try to improve it to be stricter and more reliable, after discussion
with odo, sql which is "correctly" dynamic should use psycopg2's sql
package in order to bypass the linter (bonus: it should also properly
escape & quote identifiers).
closesodoo/odoo#53938
Related: odoo/enterprise#11718
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Using a few regex like
\((_\(.*%s.*)(\) % )([\w\[\]][\w .\[\]\(\)'"]*)\)
($1, $3))
Old syntax is still compatible but starts the migration to the new
syntax that catches error.
As indicated in the comment, it's much preferred to perform response
buffering at the reverse proxy level than to increase the socket
timeout. It will free up HTTP workers for other requests faster, while
the proxy does the work of buffering the stream on disk as needed.
/!\ The timeout is also used to protect from accidental DoS effects
in situations of low worker availability, due to idle connections
caused e.g. by wkhtmltopdf's connection pooling.
Setting a high timeout will make the protection less effective, so
ensuring you have enough free HTTP workers at all times becomes critical.
In our tests with nginx's defaut buffering on a typical hardware with
SSD storage, buffering up to 1GB responses did not require any change
of the socket timeout on the Odoo side, though your mileage may vary.
See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config
directives.
OPW-2247730
See also: #20158closesodoo/odoo#51982
X-original-commit: d78ea126b8d2a72ae626880b0ec64bb7a39a07ac
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Those were deprecations implemented in 0.15
* all middlewares have been moved from `werkzeug.wsgi` to
`werkzeug.middleware`, including the `SharedDataMiddleware` we use
* ProxyFix was moved to werkzeug.middleware.proxy_fix, this had
already been fixed but I forgot the import
* sessions support was moved to a separate package
(`pallets/secure-cookies`), however while distros are starting to
update werkzeug to 1.0 (e.g. done on Arch, and in Debian
Experimental) they're not bundling secure-cookies so using a
vendored version seems like the least bad thing we can do, even more
so as conditional dependencies are not really a thing (e.g. even
with just pip we can't depend on secure-cookie iff werkzeug >= 1.0)
Also non-browser jsonrpc (as it goes through a similar process): for
internal performance reasons, name_search and read_group have been
converted to a *lazy* name_get, so the "display name" is not
unnecessarily computed.
However this is an issue for the RPC endpoints (/xmlrpc and /jsonrpc)
as they have no support for `lazy` and thus tend to blow up and / or
do the wrong thing when trying to output a lazy:
* xmlrpc has no way to handle lazy at all and straight blows up
* jsonrpc falls back to `json_default` so they try to stringify the
lazy, which might have worked except
*Problematically* both endpoints delegate the actual work to
`dispatch_rpc` which handles dispatching between various services and
ultimately creates a *new* cursor before calling model
methods (`object` service and `execute`/`execute_kw`).
This means by the time the result is serialized to be output, the
lazy's cursor has long been closed, and thus any access to an
unevaluated `lazy` errors out when trying to fetch the underlying
item.
This also means we can't just add a hook to serialize the lazy
in the xmlrpc marshaller, though we do have to do that. We *also* (for
both xmlrpc and jsonrpc) have to force evluation of lazy values before
our cursor is closed, meaning it has to be done right after the method
is invoked, iterating the entire response.
Related to task 2170343
closesodoo/odoo#49286
X-original-commit: e2b5a359c1d5eccbe725c1c3169b4130d7bca49b
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Xavier-Do