Refactor the `load_views` API so it no longer sends multiple times the same
fields description.
e.g.
When `load_views` is called to get the kanban, tree and form views,
the list of fields of the model was sent 4 times:
- Once for each view, with only the fields used in the view,
in `['fields_views']['kanban']['fields']` for instance
- Once globally, with all the fields of the model, in `['fields']`
The goal of this revision is to change that so it sends the list of all fields
only once.
In addition, if a view contains x2many fields,
the fields description of the comodel is also sent.
It was sent in the `views` key of the view fields dict.
e.g.
When calling `load_views` of `res.partner` to get the kanban,
tree and form views,
the `res.partner` fields description was actually sent 6 times:
- Once for each view
- Once globally
- Once for each view of the many2many field `child_ids` of the form view, in
- `['fields_views']['form']['fields']['child_ids']['views']['kanban']['fields']`
- `['fields_views']['form']['fields']['child_ids']['views']['form']['fields']`
The change suggested in this revision is to:
- Remove the fields description for each view in `['fields_views']`.
As it no longer contains the fields,
the key becomes `['views']` instead of `['fields_views']`.
- Replace the dict key `['fields']` by `['models']`,
which is a dict with as key the model name and as values
the model fields description. It contains the fields description
for all models implied in the view:
the model of the main view and the model of all one2many and many2many fields.
With this change, the fields description will only be sent once by model
implied in the view.
In addition, the web client was getting the information about the fields
sometimes in the global fields description list (e.g. `['fields']`),
sometimes in the fields description list of the view type
(e.g. `['fields_views']['form']['fields']`),
making it a pain to try to make changes / performance gain
in these field description dictionaries, because you never knew in which dict
the web client was getting its info.
Now, as there is only one place to get the fields description from,
it's clearer and cleaner.
- one2many and many2many fields views are passed directly in the main view
architecture rather than being put in the `views` key
of the field description.
This is actually easier to treat by the web client,
and this will allow in a future work to cache an entire view in one block
of text rather than having to combine multiple cached blocks of text
to return one view.
- one2many and many2many fields which do not have directly embedded views
have their views directly injected in the architecture,
so the web client doesn't have to do RPC calls to `load_views`
for each one2many and many2many fields not having embedded views.
For instance, this allow to reduce the number of RPC calls to `load_views`
from 8 to 1 when loading the form of `product.product`.
Currently, this behavior is limited to 1 level deep but we consider making it
go all the way down in future works. We did not do it for the moment because
in certain cases it rises the processing time and the size (bytes) too much.
e.g. the sale.order view can be 5 levels deep,
meaning you can reach 4 dialogs on top the main view.
```
sale.order form > order_line > sale.order.line form > invoice_lines >
account.move.line form > asset_ids > account.asset form >
depreciation_move_ids > account.move form.
```
This will also benefit in future works to cache an entire view in one block
of text rather to having to combine multiple cached block of text
to get one view.
- `fields_view_get` becomes `get_view`.
As it no longer returns the fields description,
keeping the `fields` in the name `fields_view_get` no longer makes sense.
Hence removing `fields` from the method name, it becomes `view_get`.
As it gets renamed anyway, we take the opportunity to rename it `get_view`,
which is more in line with the general getter/setter guidelines
in the model object world.
- `_fields_view_get` becomes `_get_view`. For the same reasons than above.
- `load_views` becomes `get_views`.
This is not mandatory, there is no technical reason to rename `load_views` as
it practically sends the same info as before,
the view architectures and their fields description. Just in another way.
We just take the opportunity of this pull request to suggest a cleaner API:
`_get_view`, `get_view` and `get_views`.
- Arguments `toolbar=False, submenu=False` fo the methods
`_fields_view_get` and `fields_view_get` are converted to a kwargs `**options`
in `_get_view` and `get_view`.
The rationale is that submenu was already no longer used (deprecated)
and the mobile options is introduced.
The mobile options is necessary to tell the server to send the mobile views
for x2many fields (kanban instead of tree).
Instead of adding a new argument each time we add a new option to
`fields_view_get`, it seems wiser to have a kwargs `**options` to avoid
to re-write all overrides each time a new option is introduced.
- `_fields_view_get` returned a dict containing the arch in text and some of the
view information. Now, `get_view` returns a tuple with the view architecture
as an `etree` node, and the view as a browse record. The rationale is that all
overrides of `_fields_view_get` were about modifying the arch only
(e.g. changing the address format/re-organizing the address related field
nodes of the partner according to the company country).
To do so, all these overrides were doing `etree.fromstring` to parse the arch
which was sent in text to convert it to an `etree`,
then operations were done on the `etree`,
and then `etree.tostring` was called to convert back the arch to string.
With this change of signature to send the arch as an `etree`,
all these back and forth `etree.fromstring` -> `etree.tostring` are avoided,
allowing some performance gain and less code in the end.
- A cleanup of the keys returned in the dict of `fields_view_get`
has been performed in `get_view`:
- `fields` is removed, as explained above,
- `view_id` is renamed `id`,
- `name` is removed, it was unused by the web client,
- `type` is removed, it was unused by the web client,
- `field_parent` is removed, it was unused by the web client,
- `base_model` is removed, it was unused by the web client.
- `filters` is moved from the global dict returned by `load_views`
(now `get_views`) to the dict returned by `fields_view_get` (now `get_view`)
as it applies only to the `search` view type.
- Retro-compatible methods for the 3 methods
`fields_view_get`, `_fields_view_get` and `load_views` are provided,
with deprecation warnings in them.
- The web client could cache the model fields description
(as it already caches the views),
so it doesn't need to fetch them again if it asks for another view of a model
for which he already has the fields description.
If we do so, `get_views` could return only the list of models used by
the views, without the fields description as of now,
and the web client would then call `fields_get` independently only for
the models for which it doesn't have yet the fields description.
This would avoid the server to return the fields description
and to call `fields_get`, which is costly, for each `get_views`,
therefore gaining performances.
- Inject the views of the one2many and many2many fields all the way down,
unlimited depth level, as explained above.
- Cache with `ormcache` the architecture of back-end views.
This is already done for qweb views, it's not done for back-end views.
Therefore the postprocessing of the views is performed for each `get_views`,
which is costly, while the view architecture doesn't change for users
belonging to the same groups, according to the groups implied by the view.
This pull request is co-authored by
Aaron Bohy (aab) for the web client part and
Denis Ledoux (dle) for the server part.
Part-of: odoo/odoo#87522
Before this commit, since https://github.com/odoo/odoo/pull/83574,
attempting to join a call through the channel guest page while being
logged as a portal user would give an acess error.
This commit fixes this issue
closesodoo/odoo#87995
X-original-commit: e6a637dc1d4e024c67b946fbe26a1b1e0dd144f3
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Because the `savepoint` and `clear` calls are nested inside the
`assertRaises` context, if one of them happens to throw *the exception
we're looking for* the interpreter will jump back to the `with`, the
`assertRaises` will swallow the exception (and count it as a success)
and the function will end having not gone through a `yield`.
This is rather frustrating to debug as it's easy to forget that a
`with` is a control flow structure, leading to a seemingly impossible
error.
We can fix this by initializing and `__enter__`-ing the savepoint
first, but doing this by hand is a bit iffy and not really
future-proof as the addition of more fallible steps during the
initialization phase of `_assertRaises` could lead to the savepoint
not being properly disposed of. Furthermore once in the scope of the
"actual" assertRaises we want the savepoint to unwind first (otherwise
the savepoint won't be rolled back when the exception *we are
expecting* gets raised).
As it turns out `ExitStack` offers the solution to our woes though
it's a bit tricky at first glance: while modifying the cleanup queue
in-place is haram, `pop_all` allows moving cleanup callbacks from one
queue to the next.
This means we can first add the savepoint to one stack and get its
errors (if any) correctly reported, cover the rest of the
initialization, then move the savepoint from one stack to an other, in
order to correctly order the coverage of the `yield` (and the userland
code).
closesodoo/odoo#87733
X-original-commit: b1cd4e4e3c918b4b08d27b30b1017fc898d4b08a
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
For payment_buckaroo and payment_sips (at least) running with a
non-standard port is an issue to the `test_redirect_form_value` tests:
while the form and test will use respectively the base_url and the
configuration port, both check a response signature which is
predicated upon a base url of `http://127.0.0.1:8069`.
This means the test does not pass when run with a different port, and
may not pass if the database was installed with a different port
either (because this may have caused the `web.base_url` to be set to
the installation port).
The other payment modules don't seem to have such signature
verification and thus apparently don't mind running with non-default
port.
Update in 15.2: `test_webhook_notification_confirms_transaction` also
broke but differently, because the payment utils would fetch (and use)
the `web.base.url` they get confused if a db is installed using one
port then the tests are run using an other (or something along those
lines), despite `HttpCase` trying to set the `web.base.url` (could be
an ordering thing).
Anyway a working solution seems to be to *remove* the bespoke code
from `PaymentTestUtils` and fix `HttpCase.base_url()` so it uses the
right port (apparently that'd never been fixed). This does require
adapting the patch being forward-ported as `base_url` is now a
callable, not an attribute.
Also re-remove the attractive nuisance of the odoo.tests.common.PORT
constant which does not work: it is evaluated before the configuration
has been loaded and is thus always set to the default (8069).
closesodoo/odoo#86068
X-original-commit: c28c99f7399da91e1a6176d6f97d4fc947013cae
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit is the 14th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
* `request.uid = x` => `request.update_env(user=x)`.
* `request.context = x` => `request.update_env(context=x)`.
* `request.context = dict(request.context, x=y)`
=> `request.update_context(x=y)`.
* `request.cr = None` => `request.cr.close()`.
* `http.mono_db()` => `request.db`.
* `http.dispatch_rpc()` => `service.dispatch_rpc()`.
* `@service.model.check` => `service.model.retrying()`.
* `request.endpoint`
=> `env['ir.http']._match(request.httprequest.path)[0].endpoint`.
* `request.routing_iteration `=> `removed`.
* `request.jsonrequest` => `request.dispatcher.jsonrequest`.
Note that `request.params` is now set much later in the process. If you
are in a situation where you values from the query string or the
http body you can use `request.get_http_params()`.
Note that using the new `request.future_response`, it is possible to
add headers and cookies on the response object before the response
object is initialized. Please note that headers/cookies saved on
the future response will NOT be injected in case of error.
PR: odoo#78857
Task: 2571224
When a test fails and asks for a screenshot, it seems that in some
conditions, the browser has already navigated to the blank page when
capturing the screenshot, resulting in a useless blank screenshot.
With this commit, all the futures are awaited to avoid this race
condition.
closesodoo/odoo#84566
X-original-commit: 37a7b81c28a7ad7c4f225e1463df96bbaf2d5c8f
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Before this, running Chrome was only supported on POSIX-type OS.
This commit adds support for running Chrome on Windows, by swapping
fork()/exec() with `subprocess.Popen`, which is
cross-platform. Linux-specific fixups are implemented via a
`preexec_fn`.
Also fixes the JSON path generation: the url path separator is `/`,
not `os.path.sep`.
closesodoo/odoo#83209
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Backport from the master-httpocalypse-juc test suite in order to observe
the perf of the master branch and to compare them with the httpocalypse.
closesodoo/odoo#84156
Task: 2571224
Signed-off-by: Raphael Collet <rco@odoo.com>
This commit brings the ability to use some feature detection
mechanism used in Odoo. The jquery.touchSwipe library
implementation requires the browser to get ontouch* events from
the window to be available to assert that the browser has touch.
The switch will make the mechanism to work as expected, and
allow tests to use touch events while being correctly executed
because the library expects such events.
closesodoo/odoo#83588
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
HttpCase.base_url() was introduced with ded278b9c2 but many tests were
using the name attribute already. The tests have been adapted to use the
new method.
closesodoo/odoo#82910
Signed-off-by: Julien Castiaux <juc@odoo.com>
Since auto-retry, real errors will lead to doubled errors on runbot.
This can be noisy and hard to understand.
This commit will help with this by using a lower log level on the first
execution. Log 25 are kepts.
closesodoo/odoo#80378
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When running test tours logging a message to console.error causes the
test tour to fail. Only one such message can cause the tour to fail, if
other message are written on the console they are simply logged in the
odoo logs. The offending message, however, is only shown at the end of
run, as part of the failing test logs. Arguably, it is better to
include it in the browser logs as well.
closesodoo/odoo#75197
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
The http.addons_manifest is a map {module: manifest_dict} that is
populated upon the first http request. This map is basically a module
manifest cache with an extra `addons_path` key, the path of the module
on the file-system. This cache is eagerly populated upon the first http
request, the map is empty in non-http contextes (e.g. cron) which have
been a source of bugs (e.g. 50c8eb1).
A manifest cache is necessary because reading and parsing python files
from the file-system is not that cheap but there is no reason that cache
is located in `odoo.http`. A thin cache layer now wraps
`load_information_from_description_file()`/`load_manifest()` and is
lazily populated.
The `http.addons_manifest` have been removed. The extra `addons_path`
key is now present in the "normal" manifest. The `read_manifest()` was
hardly used so it has been deprecated. The only way to retrieve a
manifest is now `load_information_from_description_file()` which was
renamed `load_manifest()` (no cache) and `get_manifest()` (cache).
Side note about performances, the cache is necessary. Addons manifest
are read-only and reading + parsing python files from the file system is
not a cheap operation. Running the e-commerce tour
`@website_sale.test_04_admin_website_sale_tour` without cache on
`load_manifest()` requires 68,29 secs to complete on my laptop,
exceeding the default 1-minute time frame allowed in tests. Using a
cache the time is down to 36,53 secs. The performance impact is huge.
Part-of: odoo/odoo#79977
* add configuration for `flake8[flake8-rst-docstring]`
* enable docstring-related checks
* fix invalid docstrings in odoo's core & `base`
* fix a few more bits (mostly missing or incorrect `:param:` info
fields) are out of scope for the lint but my editor catches
closesodoo/odoo#74604
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When Chrome is spawned, the `DevToolsActivePort`file is awaited to read
the port. Sometimes the file is read but is still empty, resulting in a
ValueError when trying to cast into integer. This happens when Chrome
did not have time yet to write into file.
With this commit, we expect the file to contain at least 5 bytes which
is enough to contain the max port number.
closesodoo/odoo#80791
X-original-commit: aab53fbb4e02461e0f23cf202d393e30a73fbc92
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Since #77735 the chrome runner uses a receiver thread to better handle the
full-duplex nature of Chrome's websocket communication.
However that thread was missing the `dbname` threadlocal, leading to some of
the logging and reporting to be mis-configured, and thus the filtering on CI
(runbot) to exlude any logging message emitted from the receiver thread, such
as the screenshot notifications.
Fix by passing the current `dbname` to the receiver thread when spawning it,
and having said receiver start by setting up the threadlocal.
closesodoo/odoo#79808
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
The current system uses an imperative request/response model to a
communication which is full-duplex and thus (hopefully) more suited to
a reactive approach.
This updates the Chrome runner to:
* use a separate thread for reading from Chrome, which limits stalling
/ buffering
* uses futures for request/response cycles
* uses a callbacks system for events (unrequested messages from
chrome)
The test end (success / failure) is also signaled via a future.
This requires a bit more cleanup as the browser is reused for all
tests of a *class*, so e.g. the results future must be reset between
tests.
Also for convenience add debug logging of WS requests / responses,
it's convenient for looking at a test / tour's traffic.
Part-of: odoo/odoo#77735
Explicit flushing of the cursor had been implemented in the "legacy"
`url_open` helper, but it was missing when going through `requests`
directly (via the `opener`), or when performing XML-RPC calls.
Fix that:
* extend `requests.Session` and `xmlrpc.client.Transport` so they take
a cursor
* move the setup of the XML-RPC clients to `setUp` so they can *get*
the cursor
* move the rest of `HttpCase.__init__` to `setUpClass` and drop the
override entirely
* remove the now-redundant flush in `url_open` (as it's done by the
`opener`)
closesodoo/odoo#77359
X-original-commit: c7bdae34690f31960d1fa95b5c3b4052540f5d07
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Jinja as a templating engine was problematic in differents respect:
- introduce external dependency to Odoo (less controll)
- add another templating mechanism in the stack
- specific feature in qweb cannot be reused
- difficulty in rendering easily editable templates
- more knowledge required with no betterment
By replacing jinja with qweb we can now build tools to edit a qweb
that will work with the previously jinja encoded document
(essentially `mail.template` records).
There is a catch however. Some email fields (eg. email_to) used jinja
syntax for rendering dynamic variables (ie. ${object.something} and
${object.something_that_should_not_be_escaped | safe}).
We still want user to use dynamic variables for some char fields (eg.
subject, from, to, ...). We made a new rendering engine called
"inline_template" that will render an expression enclosed by `{{` and
`}}`.
To be able to edit the templates from the backend interface, a
plugin to the Odoo editor has been made for seamlessly edit the
document.
This qweb plugin includes:
- make dynamic variables (eg. `<t t-out="variable"/>`) not editable
(for preventing the user to shoot himself in the foot)
- group and hide related logical branching (ie. t-if, t-elif, and t-else)
in order to see only one at once
- a floating select input to switch visibility of a particular logical
branching
Task-27033
X-original-commit: odoo/odoo@68182baff4
Part-of: odoo/odoo#77377
This pr proposes an auto retry mechanism for tests. This shouldn't
impact normal testing: tests are not supposed to fail, but the growing
number of tests and pull requests can lead to some bottleneck when a
staging fails because of a random error. This mechanism should help to
reduce splits/the need to retry a failed pr.
This branch have been tested with the nightly multi build, creating 40
identical build without test-tags to disable know random errors.
This multi build is used to detect test failing randomly, this is an
excellent candidate to detect the effect of the retry.
On average, with the current base of this pull request, there is between
10 en 15 failures over 40 build.
With the auto retry mechanism, only 1 build failed over 40 builds since
the same error was triggered twice.
This is simply because with the retry mechanism, an error that has a
probability of p to fail randomly will still have a probability of p² to
fail with the retry mechanism. A error that occurs 10% of the time
should only appear 1% of the time with one retry. In most of the case,
the retry is sucessfull: https://runbot.odoo.com/runbot/build/10053257
The current solution to allow to enable this mechanism only in some
cases (staging) is to check an environment variable
"ODOO_TEST_FAILURE_RETRIES" that defines a number of retry.
This will allow to retry more than once if an error still occurs to ofen
with the autoretry.
The mechanism will run multiple time the same test on the same
test_case, meaning that some modification on self may impact the second
execution. The following code is an example of how this could be
problematic, but also a good example to test the auto-retry mechanism.
```python
class TestRetry(HttpCase):
def test_fail(self):
self.t = getattr(self, 't', 0) + 1
if True or self.t == 1:
import logging
_logger = logging.getLogger('test_a')
with self.assertLogs(level="ERROR"):
_logger.error("This shouldn't be log at all")
with mute_logger('test_a'):
_logger.error("This shouldn't be logged (mute)")
_logger.error("This should be log")
```
As we can see here the error logs are also managed, and emit at a lower level the first time, butany log higher than 25 will make the test "failed" and the autoretry mechanism will be triggered. The second time, everything is logged normally. We also need to replace Traceback by _Traceback to avoid being catched by runbot Traceback detection regexes.
The inspiration here commes from the assertLogs, that replace all handlers. The mute_logger had to be adapted to use the same strategy, so that quite_logger won't detect logs catched by mute_logger or assertLogs.
closesodoo/odoo#76336
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
When a subtest fails, the failure message maybe a bit cryptic like
`Fail: Subtest (login=admin)`
With this commit, the parent test case and test method are displayed
too.
closesodoo/odoo#76046
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Refactor the Environments object into a Transaction object, which is
bound to one cursor, and is no longer shared among several cursors.
The following methods/properties have been changed:
- Environment.envs no longer works (because of the design change);
- Environment.manage() is deprecated (no longer useful);
- Environment.reset() is now an instance method;
- env.clear_upon_failure() is deprecated in favor of cr.savepoint().
closesodoo/odoo#75598
Related: odoo/enterprise#20451
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
Explain the situation when a manual change has been done to the sequence
of `account.move`. This was needed because some user didn't realize that
they changed the sequence, and when they realized it, it had polluted
multiple numbers after that.
closesodoo/odoo#74326
X-original-commit: 56f7afb253da6560cddae121f5b6cdab3e3b1c6e
Signed-off-by: Laurent Smet <smetl@users.noreply.github.com>
Signed-off-by: William André (wan) <wan@odoo.com>
Emit JS warnings as Python warnings, otherwise it's a pain in the ass
to identify the t-raw messages (they get lost in the INFO spam).
Also just log exceptions when looking for specific messages instead of
re-raising them: if the pipe from chrome is full of exception reports,
the runner may blow its stack as it looks for the screenshot message:
on the first it takes a screenshot, then looks for the screenshot
message, finds an exception, takes a screenshot, looks for a
screenshot message, finds an exception, takes a screenshot, looks for
a s...
While HttpCase did set `web.base.url` before starting a browser, in
the non-browser test cases (or cases which would mix browser and
non-browser) it would not do so.
This is an issue when installing the database with one http-port and
running tests with an other e.g. after duplicating the database (or
even not duplicating it) in order to run multiple test instances
concurrently, which requires using different http ports.
Tests would then see the base url generated during installation,
embedding the port used at installation, and would break weirdly (at
best exploding due to not finding any server to bind to, and at worst
making request on the wrong instance entirely). Simply updating the
base url during setup seems to fix most of the tests.
Notes:
* Some tests (e.g. survey) don't flush() their create/update before
calling `start_tour` or `browser_js`, the implicit flush because of
the ICP handled the issue. Perform an explicit flush of base
(similar to `url_open`) to ensure they keep working correctly.
* `url_join` should handle absolute URIs correctly, it does imply
slightly different semantics in case the `base_url` has a non-empty
path, but that seems like a very limited risk (and possibly
convenient to boot).
* `payment` needed a fix because the vagaries of the MRO led to the
extra parameter internally used by the thing to be passed to
`HttpCase`'s `setUpClass`, which would not expect it.
closesodoo/odoo#72645
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit add tests to ensure every dispatch flows redirect to a 301 if the
requested URL has a trailing slash.
It also ensures the URL params are not lost in the process.
- Basic controller: `/my/?a=b`
- Website Pages: `/my-page/?a=b`
- Basic controller with language: `/fr_BE/my/?a=b`
- Website Pages with language: `/fr_BE/my-page/?a=b`
- Homepage with language (special case/controller): `/fr_BE/?a=b`
opw-2505818
opw-2513575
closesodoo/odoo#71065
Community: https://github.com/odoo/odoo/pull/71065
Enterprise: https://github.com/odoo/enterprise/pull/18615
Related: odoo/enterprise#18615
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
This commit adds tooling to profile performance and save execution by
saving stack traces and queries to a file/database in specific format.
----------
Collectors
----------
For now, three different profiling modes (aka Collectors) are available
even if a last once should be introduced by @Gorash to profile qweb
execution.
- SQLCollector (or 'sql'): Saves the current stack trace and the query
every time Cursor.execute() is called. Any query executed on the thread
will be collected, no matter the cursor.
- PeriodicCollector (or 'traces_async'): Saves the stack trace every
'interval' seconds using a parallel thread to profile the caller thread.
The python implementation was optimized to minimize impact on
performance while remaining portable and easy to enable/disable
inside a odoo execution. Higher the frequency (lower the interval),
more impactful the profiling will become on the execution and increase
memory usage. From last experiments, 1ms looks to be a good minimum for
short executions.
- SyncCollector (or 'traces_sync'): Saves the stack trace every function
call/return. This collector is obviously quite impactful on performance
and can quickly overload the memory for long executions, but this is
quite useful to understand the precise path followed by some short
executions. Any time related information will be almost irrelevant with this
collector.
A base Collector defining minimal collectors features can easily be
extended to create custom collectors if needed.
----------------
Profiler & Usage
----------------
Collectors are not supposed to be used by themselves, but should be
given to a Profiler. The Profiler will synchronize collectors starts and
stop, and manage saving them to a file of in a ir_profile in the
database.
Exemple of usage:
```
with Profiler():
do_stuff()
```
This simple example will use the default collectors (sql and
traces_async) and save them to the database. The database is defined
automatically from current_thread 'dbname' if available.
Example of usage:
```
with Profiler(collectors=['sql'], db=False, path=/home/user/logs/do_stuff_profile/{time}):
do_stuff()
```
This more complex example disable the default behavior consisting
to save to the database, gives a path where the profile will be saved
and specify to only use the 'sql' collector. Note that
collectors=[SQLCollector()] would have the same behavior since
Collectors can be either a Collector instance or a string describing the
desired collector. This allows to define custom params for the
collectors and use custom collectors if needed.
Note that it is always possible to get results after execution without
saving it since they are available on the profiler.
```
with Profiler(collectors=['sql'], db=False) as p:
do_stuff()
print(len([None for entry in p.collectors[0].entries if ...]))
```
Profiler will also save the stack below the profiler start point, and
collectors will only collect the part of the stack over this stack.
This is a good way to reduce collectors CPU and memory usage.
Collected entries will be saved as follows:
```
[{
'start': 2.0,
'context': {},
'stack': [
['path_to_file', lno, 'func_name', 'line_content'],
...
],
},
...
]
```
SQLCollector will add three additional keys on each entry:
- query (query without parameters)
- full_query (mogrified query with parameters)
- time (the 'exact' execution time of the query)
----------------
ExecutionContext
----------------
A last tool, ExecutionContext, allows to define some context on some block of code:
Example of usage:
```
def process_modules(modules)
for module in modules:
with ExecutionContext(module=module): # note the 'not linter frienldy but still convenient' 2 spaces indentation
do_stuff(module):
```
This context will automatically be added in the stack as a virtual frame between
process_modules and do_stuff in order to split do_stuff from one single frame to
one frame per module.
----------
Speedscope
----------
The saved data are in a simple json format easy to analyze, but can't be visualized in
speedscope as they are. A utility class `Speedscope` can be used to generate a format
readable by speedscope. The used format is actually the format defined by speedscope,
meaning that all features should be available using it.
The output format is evented, meaning that we need to transform a list of samples
(a list of stack) to a list of event (going in/out a frame).
This is the main task of the Speedscope, as well as combining samples from different
sources, to display SQLCollector and PeriodicCollector results mixed together.
When stored on an ir_profile, the default speedscope generation can easily be generated
with the speedscope computed field.
This class can be used as it is but will mainly be useful for the next commit.
Special thanks to @rco-odoo for the in depth review and @Gorash for support.
This replaces the setup of the attribute 'currency_field', which depends
on the presence of other fields on the model, and may therefore vary
from one registry to another. This is necessary to make monetary fields
shareable across registries.
Doing domains like [('something_ids', 'in', 4)] crashed the emulator because it was trying to check something was 'in 4', instead of reversing the check (as real form views do).
closesodoo/odoo#68988
X-original-commit: eb14583442f2729182ffb8ec7b069e2505df3577
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit changes the way assets are declared in Odoo modules.
Before: assets were declared in template files. Template bundles were
generated from primary templates, so technically any qweb template could
have been called as an asset bundle, with the 't-call-assets' directive.
Being standard qweb templates, they had access to standard HTML tags
(script, link, with or without raw scripts or style definition), qweb
directives (t-call, t-raw, etc.) and could be inherited by other
templates.
Now: assets are defined in the module's manifest and generated by the
't-call-assets' directive.
More information on the new system can be found on the updated user
documentation (see the "JavaScript Reference" section).
Task: 2352566
Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
When testing cron triggers, it is common to get the newly created
triggers in order to validate they exist and are scheduled are the right
moment.
This new helper is a context manager that capture all triggers or
triggers created for a specific cron. The created triggers are
accessible via the context's object `records` attribute.
The various tests have been updated so they use that new helper.
closesodoo/odoo#68163
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
Co-authored-by: Raphaël Collet <rco@odoo.com>
We have some tests in odoo/upgrade that are sensitive to the order on
which they are executed. Specifically: IntegrityCase tests need to be
run after all UpgradeCase tests across all Odoo modules.
To support this we implemented a sorting mechanism for tests based on
the test_sequence class attribute. This is intended to be used by meta
cases, not by individual tests.
closesodoo/odoo#66521
Related: odoo/upgrade#2184
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
In some situations, like during tests on runbot, the data-dir location
may vary.
With this commit, the `data-dir` CLI argument is added to the
test_module_operations script.
closesodoo/odoo#66492
X-original-commit: 781c91784a0a997967a0152dca2c59a0cb2bab74
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
This should be a smarter and properly reliable version of #42071: in
that, the runner requests a port, closes it, and gives the port to
Chrome. However this apparently turns out to be less reliable than
hoped for and the port we just released can immediately be picked up
by somebody else (the original PR assumed the allocation of ephemeral
ports would be random or FIFO but that may not be the case, especially
inside containers).
This uses the same technique of requesting port 0 so the OS allocates
one, but it's Chrome requesting & immediately connecting so there
should be no race condition possible, and we keep the property that as
long as ephemeral ports are available Chrome will be able to open one
without conflicts or overlaps.
This leaves the issue of *retrieving* the port chrome got. Thankfully
it turns out we use a custom user-data-dir in which case Chrome writes
the port it got to `$DATA_DIR/DevToolsActivePort`[0]. Despite the
file's name it *also* contains the path for the devtools endpoint so
we need to only read the first line (rather than be able to read and
intify the entire thing).
Wait up to 10s before giving up entirely, and wait 100ms between each
check for the file's existence: on my machine without significant load
the file appears after 80 to 150ms, waiting up to 90ms seems ok (it's
not like we're in a super hurry as tours tend to be pretty long).
Other paths explored before moc used his eyes and brain and found out
about DevToolsActivePort:
* Chrome prints ws URL on the stderr, however because we don't know
how much garbage Chrome might send there we need to send it to a
continuous sink otherwise Chrome *might* end up blocking on its
stderr because we're not reading from it. This turns out to be a bit
of a mess of processes or additional threads.
* xdo suggested we check what ports Chrome listens on using something
like netstat/ss (turns out `psutil` has support for that OOTB),
which worked great except on WSL (where it didn't work at all), and
the future-proofness was a bit questionable as Chrome might add
other servers in the future.
* fme suggested using socket activation support[1] and passing in the
port we'd opened without closing it, which would really have been
ideal, however it turns out it was removed a few months later when
chrome added pipes support[2], which was a pain to realize as chrome
doesn't exactly do any useful error reporting (so unknown options
just disappear into a void to be never seen or heard of ever).
* And while the pipes system[3] has *serious* positive attributes
(even lower initialization overhead, we could remove the websocket
dependency, also avoids wasting sockets though that's not too much
of an issue here) it would require rewriting a lot more than just
the initialization as it uses its own logical protocol
(NUL-terminated JSON). TBF most of the messaging stuff is properly
contained into just a few `_websocket` methods but still...
[0] https://bugs.chromium.org/p/chromium/issues/detail?id=624837#c4
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=624837
[2] https://chromium-review.googlesource.com/c/chromium/src/+/954405/3#message-ab7415a7db7b94787300d987216e9ce60db47bc2
[3] https://chromium-review.googlesource.com/c/chromium/src/+/954405/3
opw-2378464
closesodoo/odoo#65195
X-original-commit: b679d97a83f1ac898ee0916e92a5b440702bdcdf
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
Co-authored-by: Christophe Monniez <moc@odoo.com>
Allow a False email as valid valid. Generate an email only if not given but
keep void values.
Ensure company_id / company_ids match, notably to ease creation of users
in a multi company test environment.
LINKS
Task ID-2421795
COM PR odoo/odoo#63677
X-original-commit: 21999058800957a08c5e129c527afe6b84812a8a
Bug
===
Sometimes, the registration testing tour failed.
The bug can be semi-deterministic if we add a "sleep(1)" in the endpoint
"/event/<event>/track".
Reason
======
The reason for that is the service worker. It will pre-fetch all the
links in the page (see "prefetch-pages"), so for the "Online Reveal"
we will pre-fetch ~100 pages... If the server is slow, it can cause
issues.
If one endpoint takes some time, all other HTTP requests done by the
service worker will be waiting for it.
So, at the end of the testing tour, the service worker will continue to
make HTTP requests (because it makes the request sequentially) and so
some threads will still be created after the tour.
Even if "_wait_remaining_requests" is called to wait those threads, as
the service worker is still running, it will still continue to make HTTP
requests, creating new threads...
Fix
===
The solution to this issue is to kill the service workers of the browser
when we stop the tour before waiting for the end of the "HTTP request
threads".
Task 2381066
closesodoo/odoo#62827
X-original-commit: 6d08408a33880221b28f0f8a81d699825d927a48
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
The now unique class behaves as the former class `SavepointCase`. It is
now up to the developer to use `setUp` or `setUpClass` for preparing the
tests.
This commit fix the "Unsupported M2M command 1" raised `Form` helper.
The `onchange()` method will in fact emit UPDATE command for many2many
fields when the value submitted an the one in database has changed
(this is the case for example for nested m2m in form views)
OPW-2044631
closesodoo/odoo#59943
X-original-commit: 76bd8208a416cefab6becff9f2d7204c7d196201
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Signed-off-by: Xavier ALT <xavieralt@users.noreply.github.com>
Denis Ledoux