1. move cache to _get_asset_paths
The `_get_asset_content` cache has many cache key that are related to a
posprocessing of the `_get_asset_paths` result, the heavy part of this
method. Moving the cache to _get_asset_content will have the benefit
to create less duplicates entries in the ormcache as well as less cache
miss.
To simplify even further, the css and js parameters are removed since
they only filter the output of get_paths, the heavy part of globing the
file will be done before that. Anyway, they are both true when called
from _get_asset_content, and the only other call, in
`_get_related_bundle` don't really need to filter them since it is not
a critical part regarding performance, and the funtional result will
stay the same.
The initial orm cache key was using `_get_template_cache_keys`, a little
overkill and possibly creating duplicates entries again. The only
context key needed is website_id for `_get_related_assets`.
Note that it is not really enough, the orm cache key should actually
contain `request.session.get('force_website_id')` as well has
`request.httprequest.host`. This will be addressed latter since a nicer
solution would be to have website_id as a unique parameter computed
earlier.
2. better _get_asset_paths cache key
The orm cache key was simplified in previous point but there is still
one concern, the website_id depends on more parameters than that:
- request.session.get('force_website_id')
- request.httprequest.host
- existing websites
The idea here is to call `get_current_website` instead of using all
parameters that could define the webiste.
In the same spirit of `_get_template_cache_keys` `_assets_path_params`
can be overriden to give extra params that are usefull to list assets
path. Those params are computed before entering the method
`_get_asset_paths`. This may latter put at a higher level latter, in
get_asset_node, to simplify the _generate_asset_nodes_cache key.
3. better assets_node caches key
The main purpose of this part is to improve ormcache containing assets
nodes. The ormcache key contains
- to much context key
- missing session/host/env info
- unwanted boolean options.
- keys leading to the same cache value
The main goal being to reduce the size of the cache keys, decrease the
number of cache entries and improve the cache hit.
This will also make the behaviour more coherent and hopefully less bug
prone because of mismatch in parameters.
The main reason of the orm cache is the slowness of the validation of
the assets. This includes:
- listing files (dedicated orm cache)
- computing version
The cache key was depending on
- `debug`
The only relevant value for debug is "contains assets"
We dont need to differ between debug='', debug='1', debug='test',
and 'debug=assets', 'debug=tests,assets', ...
- `defer_load`, `lazy_load`, `media`
Those values are only useful to generate html node, a leightweight
operations that does not really needs to be in cache. `media` was also
used in the generation but it looks useless if we have the media on the
node. THIS NEEDS TO BE VALIDATED but in any case, since media is not
used to generate the url, it doesn't make sence to use it in the
generation.
The main idea to remove them from the ormcache key is simply to generate
the nodes outide the ormcached values.
-`async_load`
This one is similar to `defer_load` and `lazy_load` but it looks like
it wasn't used anymore. This was simply removed
- context.get('lang')
The only information needed is the direction, rtl or ltr. This means
en and fr languages, despite sharing the same css assets, will duplicate
the ormcache entries.
-`_get_template_cache_keys`
Only the lang and webiste where really relevant in this flow. Other
keys are actually useless in this flow.
Some information used in the generation where not in the orm cache key
- `self.env.user.lang` if there is no lang in the context
- `request.session.get('force_website_id')`
- `request.httprequest.host`
- ...
The proposed solution is to:
- extract any informùation needed from thecontext, request, environment
before entering the ormcache, reduce it to the minimal possible set of
values needed
```
rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl'
assets_params = self.env['ir.asset']._get_assets_params() # website_id
debug_assets = debug and 'assets' in debug
```
and remove a leightweight part of the logic
```
def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None):
links = self._get_asset_links(bundle, css=css, js=js, debug=debug)
return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media)
```
Where _get_asset_links is the cached part, and _links_to_nodes is the
lightweight part generating the nodes based on the `defer_load`, ....
Additionnal notes:
- data-asset-version and data-asset-bundle are removed from the node
since they don't seem to be used anymore since 65d70acdbf
- async_load is removed since there is no occurence of this in the code.
- a small hack is still needed to pass javascript content instead of
links, this is only to manage css compile error and will hopefully be
removed in the future.
- a context key is still in use to generate the bundle, the
`commit_assetsbundle` but it has no impact on content and will hopefully
be removed in the future.
4. Add test for ormcache hit/miss
In this context, hit/miss is about having the same cache key for the
same result. This test demonstrates the current state, were entries are
create in the ormcache only if the key is really different and will lead
to a different result.
5. remove cache invalidation
This cache invalidation is quite agressive since everytime an
assetbundle is updated, all workers will clear their cache.
The concerned cache by this clear_cache is `_generate_asset_nodes_cache`
throug `_get_asset_nodes`.
The cache is ignored, both in dev=xml and debug=assets.
This clear cache was made conditionnal in 553ea82f81 but this does
not solve an issue we can have in production.
Lets imagine a clean solution
- all sources are updated
- all workers are restarted.
The orm caches are all empty, but since the sources
changed, all bundles will be recomputed. This means that every bundle
updated in database with save_attachement will invalidate the cache of
all workers. Rendering a pdf report of any kind using a specific bundle
will invalidate all cache. Starting a debug=assets for the first time
will invalidate all cache, even if the cache is not used in this case.
But for a regenerated bundle we would expect the ormcache to be:
- empty (did not generate the same bundle yet)
- have the same value (concurrent generation of the same bundle)
Having a different value would mean that the bundle was generated with
another version of the sources. In this case it is maybe even better not
to invalidate the cache since it could lead to an invalidation war
between two workers.
The only case where invalidating this cache is useful is when a bundle
changes, Usually if an ir_asset is created, modified, ...
There is still another rare but possible possibility to have a 404 if
the transaction is rollbacked after populating the assets node cache.
In this case, we only need to clear the cache locally in case of
rollback.
Part-of: odoo/odoo#121376
Since #99912 logging an error message doesn't always end qunit tests.
This was mainly to allow to failfast logging qunit errors earlier
without stopping the tests in order to test all qunit anyway.
The logic was to have an end message that stops the test.
Unfortunately some errors will prevent the qunit suite to start
and the test will wait a 1800 long timer. An example was because of
a Missing dependencies. https://runbot.odoo.com/runbot/build/19306352
This new approach will avoid to stop only if the message looks like a
qunit failure and the final message is not there (to be sure).
closesodoo/odoo#100238
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Right now the qunit will log all results at the end.
This means that the runbot may wait for all qunit before detecting the
failure.
This also mean that all failure are in one ir.logging on runbot, making
the automated parsing difficult if multiple modules fails during the
same build.
We could also log all failure immediately, but grouping them my qunit
module will avoid duplicating logs for linked causes (one failure
leading to a `Expected %s assertions, but %s were run` message)
closesodoo/odoo#99912
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Auto retry can be usefull to avoid breaking a build because of a
small tour or query count, but for long tests like qunit, this can be
painfull when a real error is triggered.
This commit proposes to disable autoretry on demand for some tests
to solve this issue.
This may be applied on all tests longer than a few seconds.
closesodoo/odoo#95440
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Before this commit Chrome's "touch mode" was enabled in both desktop and
mobile-like tests suite (when run headless).
To better match real usecases, this commit adds an option to
enable "touch mode" only in mobile tests suites; keeping it disabled in
desktop ones.
Part-of: odoo/odoo#95924
* remove `component_extension_tests`, that's a test from web and
already tested by web, no idea why it's rerunning in POS
* remove all failfast specs, after discussion with ged, seb, and
xdo it was considered not useful: it's mostly for the runbot but
turns out to be pretty minor as far as positives go, and it makes
fixing errors much harder
Part-of: odoo/odoo#77735
Regarding previous commit, the option parameter can be removed
from _get_asset_content api, followed by a nice snowball effect.
Part-of: odoo/odoo#75248
This commit is the first phase of the conversion of the web/ JS
codebase to the owl framework. The impact of this commit is two-fold.
First, it rewrites the framework part of web with a new system of
services and registries. Services allow to execute code (e.g. do rpcs,
setup things) before launching the application. They can also expose
an API to be used by other parts of the application (e.g. a notification
service would expose a function to display notifications). Services are
often a good extension point for external modules that want to execute
code at webclient startup. Registries offer another way to extend the
application. They provide well designed extension points to add
elements/behaviors from the outside (for instance, to add a systray item,
an error handler...).
Second, this commit initiates the conversion of the webclient to owl
with a top-down approach, around those notions of services and registries.
The root of the web application is now an owl application. Among others,
the WebClient, ActionManager, Navbar, UserMenu, DebugManager, Dialogs,
services (e.g. notification, ajax...) have been converted to the new
framework/architecture.
Legacy views and client actions are still supported (and used). They
will be converted in the next months, and at some point, the support
will be dropped.
Co-authored-by: Aaron Bohy <aab@odoo.com>
Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Géry Debongnie <ged@odoo.com>
Co-authored-by: Samuel Degueldre <sad@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Simon Genin (ges) <ges@odoo.com>
Co-authored-by: Francois (fge) <fge@odoo.com>
Co-authored-by: Michael Mattiello (mcm) <mcm@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais (lpe) <lpe@odoo.com>
Co-authored-by: Jorge Pinna Puissant <jpp@odoo.com>
* QWeb bodies should be markup-safe so `0` should always be
markup-safe.
* `head` is qweb-rendered so the same.
* The `json` pseudo-module in qweb templates is `json.scriptsafe`,
which should be markup-safe.
This commit changes the way assets are declared in Odoo modules.
Before: assets were declared in template files. Template bundles were
generated from primary templates, so technically any qweb template could
have been called as an asset bundle, with the 't-call-assets' directive.
Being standard qweb templates, they had access to standard HTML tags
(script, link, with or without raw scripts or style definition), qweb
directives (t-call, t-raw, etc.) and could be inherited by other
templates.
Now: assets are defined in the module's manifest and generated by the
't-call-assets' directive.
More information on the new system can be found on the updated user
documentation (see the "JavaScript Reference" section).
Task: 2352566
Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
Disabled in odoo/odoo@b992619840
by mistake.
Also, restore the "failfast" for the QUnit tests (disabled in the same
commit...).
X-original-commit: 7f8e0fe4d2a47968bdcb136fd7e6d96a184f9eae
Was not super visible, especially without HR as the dialog immediately
closes (limitation of the web client), or when an administrator user
massively disables totp.
Show a toast notification indicating the success of the action.
Also fix the logging of `totp_disable` so it correctly handles being
called on more than one record, and change the logging data to
<browse_record> (<logins>) across the board.
Also add a way to provide an action to execute after a notification:
by default nothing happens, which leads to dialogs not closing and
forms not reloading, and there is no good way to show a notification
and/then do some other thing, which is inconvenient.
Assets bundle names have been changed in #48676 but QUnit.only uses check was
still using old names. This commit makes that check use the new names.
task-2274185
closesodoo/odoo#52687
Signed-off-by: Julien Mougenot (jum) <jum@odoo.com>
When loading a page on an existing starting database, registry is not
fully loaded causing potential error when trying to access model
existing in database (views, menitem, ...) since model added in last
loaded module does not exist in registry.
Thus, executing browser js test may lead to errors when executed during
an update on a database with other modules installed.
HTTPCase should be executed post_install to ensure that registry is
fully loaded to avoid this problem.
Since HTTPCase are slower than other test, it is also a good idea to
execute them at the end, in order to prioritize fast fail.
With this commit, a warning is isued if such a test class is tagged to
run at install time.
While at it, remove deprecated at_install and post_install helpers and
remove the deprecated phantom_js alias.
closesodoo/odoo#39462
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Before this commit, some views appeared broken on mobile.
We don't consider this as a bug because Odoo is not supposed
to be responsive in Community edition. This feature is only
available on Enterprise.
This commit aims at clarifying this situation by clearly moving
the base requirements to support browsers' responsive to the
Enterprise edition and stating that the Community edition only
targets desktop.
It's important to note that, for technical reasons, some
responsive-related JS/CSS code may still be present in the
Community codebase but they are not meant to provide any
explicit or implicit support for responsive/mobile devices.
Technically speaking, this commit removes the viewport meta tag.
This change will then force the "desktop-like" rendering even on
small screens for the Community edition and will keep the current
behavior (scaling/zooming on small screens) for the Enterprise edition.
By doing this we do not let the users think that something is broken
and must be fixed.
So,
in Community edition, desktop is always displayed;
In Enterprise edition, Responsive Web Design is available.
We also had to adapt a test to be able to apply XPath adding
viewport meta tag in enterprise.
Task ID: 2090202
Before this rev., the mobile js tests suite was run on a desktop
like chrome-headless (1366x768). This can be a problem for this
suite which tests mobile behaviors and thus clicks on parts of the
interface that are only visible on small devices (thanks to scss
media queries).
This wasn't really a problem until now, because the test framework
allowed to click on invisible elements. But it's no longer the
case since the introduction of click helpers.
When in Python 3, then function `open` defaults to open the file in text mode
with and the encoding (if not passed) is taken from
`locale.getpreferredencoding`.
Using `getpreferredencoding` is not necessarily a good practice. Because
files might no be encoded that way.
When testing Odoo not in runbot (which has LANG set to C.UTF-8) you may get:
Traceback (most recent call last):
File "/home/gitlab-runner/src/odoo11/addons/web/tests/test_js.py", line 24, in test_check_suite
self._check_only_call('web.qunit_suite')
File "/home/gitlab-runner/src/odoo11/addons/web/tests/test_js.py", line 37, in _check_only_call
if RE_ONLY.search(fp.read()):
File "/home/gitlab-runner/virtualenvs/odoo11-MlnlRiTaP2/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4757: ordinal not in range(128)
The proposed solution is to read the file in binary mode, and decode to
unicode/str ourselves using utf-8. We don't use the encoding argument of
`open`, so that the code works in both Python 3 and Python 2.
PR #24434
With this commit, we introduce a new 'fail fast' feature to our qunit
test suite: when it is activated, the qunit test suite will immediately
stop after the first failed test.
It is accessible as a flag in the url (failfast), or by clicking the
checkbox in the UI. It is currently not activated by default.
Note that this commit also change the url for the runbot phantomjs test
in order to activate this feature. This allows us to increase the global
timeout for the js test suite without fear!
Purpose: When running tests, all the tests for the installed/updated
files are done. This commit adds a 'tagged' decorator that can be used to
tag tests. Combined with a new 'test-tags' CLI option, it adds the ability
to filter which tests are executed. For example, @tagged('slow') will
add a tag 'slow' to the test. The CLI option 'test-tags="slow"' will
only run tests tagged 'slow'.
One can use prefixes to select cases with tags.
'+' or no prefix means that the tests tagged with this tag are selected
for execution. '-' prefix will exclude the tests tagged with this tag.
Exclusion takes precedence over inclusion.
Also, by default, all Odoo tests cases are tagged 'standard' and with
the technical name of the module.
This means that when selecting tests with the 'test-tags'
parameter, if '-standard' is not specified, all tests tags are
going to be executed.
When tagging tests, one can remove such automatic tag by prefixing the
tag name with '-'. E.g. @tagged('-standard') will remove the standard
tag from the test.
Another example, if one wants to test the 'sale' module alone,
even without adding any 'tagged' decorator thos tests can be selected
like that: --test-tags="sale"
Tests are selected or deselected using a TagsSelector. When instanciated,
a string is passed with comma separated tests selectors like
'+slow,-standard'. When the 'check' method is called with a test as argument,
it returns True or False if the test has to be executed or not.
A timeout of 300 seconds is not enough anymore to run all tests on some
runbots (e.g. runbot12) or Odoo.sh installations. The duration on these
servers is often slightly over 300, leading to a failing test.
This increase is logical since unit tests are added on a regular basis.
opw-805418
opw-804621
This rev. introduces a new test suite meant to test the webclient
components on mobile devices. The key 'config.device.isMobile' is
forced to true in this test suite, so that mobile specific JS files
are properly executed, which isn't the case in the classic JS test
suite (setting isMobile to true in the test definition is too late,
as the JS files are already processed).
For now, this new test suite contains a single test, which was
skipped until this rev. as it couldn't be executed in the classical
JS test suite.
Both suites are executed at each build of the runbot, and they
can be manually executed from the webclient as well (via the debug
manager).
We have a new runbot server which is quite slow. Because of this, it
takes too much time to run this unit test suite (for comparison, one of
the last build by the main runbot takes 126s).
This is necessary for now, until we use chrome headless or something...
A few builds fail when run in 'primetime', because the js test suite
takes more than 120 seconds (on the slow runbots). With this commit, we
increase our timer to 180s to make sure that this does not happen again.
This test executes the QUnit js test suite. Since the new views,
this suite keeps growing, which is good. The default timeout
for phantomjs tests is 60 seconds, which starts to be tight for
test_01_js when the runbot is overloaded.
So this commit increases the timeout of this test to 120 seconds.
This commit introduce a full redesign of all JS views. We started
basically from scratch. The goal was to unify all the various views
under a common framework, to make them testable, to make then usable in
different conditions (in studio, or in the frontend), and to make our
lives easier.
Some important points are:
- we introduced new coding guidelines (camelCase, 80 chars width, ...)
- we have a brand new testing framework (still QUnit based)
- kanban view moved to the web addon
- calendar view (formerly web_calender) moved to web as well
- the tree view was removed
- all new code should be documented
We hope that this code is the start of a new era for the Odoo web
client, we want to have a high quality codebase, well documented, well
tested, well designed.
Work done by the framework team: mostly aab, ged, chm, dmo, qsm
A squashed merge is required as the conversion of the apiculture branch from
bzr to git was not correctly done. The git history contains irrelevant blobs
and commits. This branch brings a lot of changes and fixes, too many to list
exhaustively.
- New orm api, objects are now used instead of ids
- Environements to encapsulates cr uid context while maintaining backward compatibility
- Field compute attribute is a new object oriented way to define function fields
- Shared browse record cache
- New onchange protocol
- Optional copy flag on fields
- Documentation update
- Dead code cleanup
- Lots of fixes
split automated suite so web only runs its own JS tests, and web_tests_demo runs its own.
Also reversed changes to index.
todo: use hack cursor thing?
bzr revid: xmo@openerp.com-20140221124557-s875nj0xrf2t85i9
To tun the tests:
createdb t
/server/openerp-server --addons-path=addons,web/addons server -d t -i web_tests_demo
Once the server is ready, run, in an other shell:
/server/openerp-server --addons-path=addons,web/addons testjs
bzr revid: al@openerp.com-20121209024810-cdi2s9ftr97x8l5p
* Global timeout significantly increased to correctly handle e.g. db creation and teardown
* Per-test timeout brought back down to 2s
bzr revid: xmo@openerp.com-20121115091501-ys8opo0ccr3dq29h
Xavier-Do