Commit Graph
47 Commits
Author SHA1 Message Date
Xavier-Do 595aa24843 [IMP] registry: multiple ormcache
One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.

Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.

The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.

Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.

The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.

We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.

Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.

Note that this poc is inspired from the long term cache but not all
use case where applie yet.

Part-of: odoo/odoo#119813
2023-07-18 11:42:26 +02:00
Xavier-Do ca8dc2d9b4 [IMP] base, website: small refactoring
Mainly to simplify website overrides and general api

closes odoo/odoo#121376

Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2023-06-10 11:14:12 +02:00
Xavier-Do 6d5d234f15 [IMP] base: better ormcache management
1. move cache to _get_asset_paths

The `_get_asset_content` cache has many cache key that are related to a
posprocessing of the `_get_asset_paths` result, the heavy part of this
method. Moving the cache to _get_asset_content will have the benefit
to create less duplicates entries in the ormcache as well as less cache
miss.

To simplify even further, the css and js parameters are removed since
they only filter the output of get_paths, the heavy part of globing the
file will be done before that. Anyway, they are both true when called
from _get_asset_content, and the only other call, in
`_get_related_bundle` don't really need to filter them since it is not
a critical part regarding performance, and the funtional result will
stay the same.

The initial orm cache key was using `_get_template_cache_keys`, a little
overkill and possibly creating duplicates entries again. The only
context key needed is website_id for `_get_related_assets`.

Note that it is not really enough, the orm cache key should actually
contain `request.session.get('force_website_id')` as well has
`request.httprequest.host`. This will be addressed latter since a nicer
solution would be to have website_id as a unique parameter computed
earlier.

2. better _get_asset_paths cache key

The orm cache key was simplified in previous point but there is still
one concern, the website_id depends on more parameters than that:
- request.session.get('force_website_id')
- request.httprequest.host
- existing websites

The idea here is to call `get_current_website` instead of using all
parameters that could define the webiste.

In the same spirit of `_get_template_cache_keys` `_assets_path_params`
can be overriden to give extra params that are usefull to list assets
path. Those params are computed before entering the method
`_get_asset_paths`. This may latter put at a higher level latter, in
get_asset_node, to simplify the _generate_asset_nodes_cache key.

3. better assets_node caches key

The main purpose of this part is to improve ormcache containing assets
nodes. The ormcache key contains
- to much context key
- missing session/host/env info
- unwanted boolean options.
- keys leading to the same cache value

The main goal being to reduce the size of the cache keys, decrease the
number of cache entries and improve the cache hit.
This will also make the behaviour more coherent and hopefully less bug
prone because of mismatch in parameters.

The main reason of the orm cache is the slowness of the validation of
the assets. This includes:
- listing files (dedicated orm cache)
- computing version

The cache key was depending on
- `debug`
The only relevant value for debug is "contains assets"
We dont need to differ between debug='', debug='1', debug='test',
and 'debug=assets', 'debug=tests,assets', ...
- `defer_load`, `lazy_load`, `media`
Those values are only useful to generate html node, a leightweight
operations that does not really needs to be in cache. `media` was also
used in the generation but it looks useless if we have the media on the
node. THIS NEEDS TO BE VALIDATED but in any case, since media is not
used to generate the url, it doesn't make sence to use it in the
generation.
The main idea to remove them from the ormcache key is simply to generate
the nodes outide the ormcached values.
-`async_load`
This one is similar to `defer_load` and `lazy_load` but it looks like
it wasn't used anymore. This was simply removed
- context.get('lang')
The only information needed is the direction, rtl or ltr. This means
en and fr languages, despite sharing the same css assets, will duplicate
the ormcache entries.
-`_get_template_cache_keys`
Only the lang and webiste where really relevant in this flow. Other
keys are actually useless in this flow.

Some information used in the generation where not in the orm cache key
- `self.env.user.lang` if there is no lang in the context
- `request.session.get('force_website_id')`
- `request.httprequest.host`
- ...

The proposed solution is to:
- extract any informùation needed from thecontext, request, environment
before entering the ormcache, reduce it to the minimal possible set of
values needed
```
    rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl'
    assets_params = self.env['ir.asset']._get_assets_params()  # website_id
    debug_assets = debug and 'assets' in debug
```

and remove a leightweight part of the logic

```
    def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None):
        links = self._get_asset_links(bundle, css=css, js=js, debug=debug)
        return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media)
```

Where _get_asset_links is the cached part, and _links_to_nodes is the
lightweight part generating the nodes based on the `defer_load`, ....

Additionnal notes:
- data-asset-version and data-asset-bundle are removed from the node
since they don't seem to be used anymore since 65d70acdbf
- async_load is removed since there is no occurence of this in the code.
- a small hack is still needed to pass javascript content instead of
links, this is only to manage css compile error and will hopefully be
removed in the future.
- a context key is still in use to generate the bundle, the
`commit_assetsbundle` but it has no impact on content and will hopefully
be removed in the future.

4. Add test for ormcache hit/miss

In this context, hit/miss is about having the same cache key for the
same result. This test demonstrates the current state, were entries are
create in the ormcache only if the key is really different and will lead
to a different result.

5. remove cache invalidation

This cache invalidation is quite agressive since everytime an
assetbundle is updated, all workers will clear their cache.

The concerned cache by this clear_cache is `_generate_asset_nodes_cache`
throug `_get_asset_nodes`.

The cache is ignored, both in dev=xml and debug=assets.

This clear cache was made conditionnal in 553ea82f81 but this does
not solve an issue we can have in production.

Lets imagine a clean solution
- all sources are updated
- all workers are restarted.

The orm caches are all empty, but since the sources
changed, all bundles will be recomputed. This means that every bundle
updated in database with save_attachement will invalidate the cache of
all workers. Rendering a pdf report of any kind using a specific bundle
will invalidate all cache. Starting a debug=assets for the first time
will invalidate all cache, even if the cache is not used in this case.

But for a regenerated bundle we would expect the ormcache to be:
- empty (did not generate the same bundle yet)
- have the same value (concurrent generation of the same bundle)

Having a different value would mean that the bundle was generated with
another version of the sources. In this case it is maybe even better not
to invalidate the cache since it could lead to an invalidation war
between two workers.

The only case where invalidating this cache is useful is when a bundle
changes, Usually if an ir_asset is created, modified, ...

There is still another rare but possible possibility to have a 404 if
the transaction is rollbacked after populating the assets node cache.
In this case, we only need to clear the cache locally in case of
rollback.

Part-of: odoo/odoo#121376
2023-06-10 11:14:11 +02:00
Xavier-Do 5594d8f191 [IMP] base, *: speedup assets unique computation
One of the most costly part of a page loading when the ormcache is cold
is computing the assets node, the unique identifier of an attachment to
validate whether the existing attachment is still valid with the current
version of the static files.

This operation needs to glob assets path in the filesystem,
get the modification date, check attachments, ...

Right now this task is not really optimized and can take some time
because of an excessive number of glob on the filesystem, unnecessary
exists to define absolute path, double computation of file list and
modified times when getting js and css bundle separately, ...

A list of modifications mainly discussed in the pr message are made
with this commit to speedup things.

- split css and js unique
- prepare api for an in memory glob
- change api to propagate absolute path and meta information through
`ir.asset._get_paths`-> _get_asset_paths -> `_get_asset_content` ->
`AssetsBundle`

closes odoo/odoo#121159

Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2023-05-17 14:47:12 +02:00
Géry Debongnie 44052d58d7 [FIX] web,base: do not display js module errors in some cases
Since saas 16.1, we added a more visible error information box when the
module system cannot find some dependencies, or if there is an error in
the JS code. This error box is very useful to understand a lot of
typical devlopment issues.  However, it was occasionally displayed in
production code.

However, we sometimes observe that the error information box was
displayed, because the request to load an assets failed with a 404.

It is unclear in which circumstance this can happen, but it seems that
the following elements are involved:

- the user has an open tab with odoo, kept open for a while
- some js code is changed on the server, which causes the previous
assets to be deleted
- the user reload its open tabs, so the browser will load the /web page
from disk, which points to the old assets files
- then it will try to load all assets, with a 404 on one of the script
- the error information box is displayed

This commit intercepts the loading error and stop displaying the error
information box, which is basically the same behaviour as 16.0 and
before. We could force a reload in that case, but it seems dangerous,
since in case of errors, it could easily lead to an infinite loop.

closes odoo/odoo#117625

X-original-commit: 0a44b5b1e008b591a716efd26217c7db7360c37b
Signed-off-by: Géry Debongnie <ged@odoo.com>
2023-04-04 16:02:12 +02:00
Xavier-Do cd309a200f [IMP] base: don't add bundle name in bundles
The bundle name is irrelevant in the bundle content and will prevent
attachment to store the same file if the bundles are exactly the same.

closes odoo/odoo#102502

X-original-commit: b1d57adf6ff358aa79f41817f4f7bfe60f9e5ac0
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2022-10-07 11:12:59 +02:00
Xavier-Do f68f7fc2e6 [FIX] base: use all date for assetsbundle version
The current asset_bundle version uses the last modified date.
This can be problematic in some case.

Even if it is a long time issue, the problem was rediscovered on runbot
with a commit being in the future. Runbot will export all file and set
the write date of the file to the commit date. The main purpose is to
have deterministic bundle version between different builds. This also
allows to generate assets bundle once at install for all post install
subbuild.

The issue here is that the commit date was greater than now(), meaning
that some tests setting custom css in attachment won't trigger the
regeneration of assets bundle. This wasn't really noticeable before
assets pregeneration.

This is not the first time strange issues occurs because of the
last modified logic.

This commit combined all last_modified to generate the bundle
version.

The current adaptation is quick and dirty and this will be reworked in
another post 16.0 freeze assets refactoring and cleanup.

closes odoo/odoo#100160

Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2022-09-16 11:12:34 +02:00
Gorash 5410b7c238 [IMP] base/web: XML templates are added into the asset bundles.
XML files are now declared in python module manifests. During the qweb
't-call-asset' directive, assetbundle will fetch the declared xml files,
apply the inheritance (t-inherit) and create a javascript service (for
eg: 'web.assets_backend.bundle.xml') which is added at the end of the
*.js mimifier file.

When the debug mode is activated, comments are added in the template
indicating which file the template comes from as well as the
inheritances applied to it.

****

JavaScript:

assets.js (module @web/core/assets) takes care of loading libraries,
javascripts and styles.
`loadJS(url)` (loads the javascript and returns a resolved promise when
the templates are also loaded via the '*.bundle.xml' service)
`loadCSS(url)` (loads the style a resolved promise when the file is
loaded)
`loadXML(xml, app=assets.defaultApp)` (load template into
application/owl, used by the `*.bundle.xml` services)
`getBundle(bundleName)` (get the bundle descriptor)
`loadBundle(desc)` (load the files and bundle from a descriptor)

templates (XML element content all owl templates)

A new `ready(serviceName)` method on boot.js lets you know when a
service is loaded are the require.

The xmlDependencies attribute no longer exists.

Python:

The xmls taken into account by assetbundle.py, applying `t-inherit`
inheritances and adding an `name_of_the_bundle.bundle.xml` service in
the generated JavaScript file.

****

Every manifest changes is into the next commit, except 'web_tour' in
this current commit as example.

Part-of: odoo/odoo#95500
2022-09-14 20:25:01 +02:00
Raphael Collet 6cf8db906f [REF] *: adapt code to new flush API
closes odoo/odoo#87527

Related: odoo/upgrade#3497
Related: odoo/enterprise#26939
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-05-25 18:00:47 +02:00
Gorash 880954ebfc [IMP] *: remove _render from ir.ui.view and simplify report
There were inconsistencies in the calls to `_render`.
* the view context could contain information that misled developers.
Indeed, the context and value of the view are not supposed to be found
in the rendering. Thus by calling `ir.qweb` with the name of the
template, we ensure that there is no unwanted information and in
addition the cache key is that of the name of the template which saves
a query.
* the context used for rendering was modified by a method on
`ir.ui.view`, except this is not information used by this model. There
is now a `_prepare_environment` method residing on `ir.qweb`. This
method allows to modify the value dictionary as well as the context in
which the rendering will be done. This preparation of the data as well
as my security check is done only once per rendering. This also saves
some queries
* Freeze options for rendering were inconsistent. It could be that
options on which rendering depends were not part of the cache key. Thus,
depending on the user who generated the generation of the rendering
function, there was or was not information in the template. For example
for automatic branding. This is no longer possible, because it is the
context that is used. The options serving as a cache key are only
recorded for information (for the profiling system for example). A
simplification of the `ir.qweb.field` models could be made.

The report rendering and call `ir.qweb` instead of `ir.ui.view`.

Part-of: odoo/odoo#85110
2022-03-29 10:56:15 +02:00
Gorash e830953570 [IMP] IrQweb: refactoring and add technical documentation
Major changes:
- Remove some of the recursively when compiling
- Compile attributes became a directive
- Compile options became a directive
- `t-field` compilation now uses the same logic as `t-out`
- Simplification of ``t-if`` directive compilation
- Improved handling of errors wrapped by QWebException
- Constants defined outside the class

    Odoo
     ┗━► _render (returns MarkupSafe)
        ┗━► _compile (returns function)                                        ◄━━━━━━━━━━┓
           ┗━► _compile_node (returns code string array)                       ◄━━━━━━━━┓ ┃
              ┃  (skip the current node if found t-qweb-skip)                           ┃ ┃
              ┃  (add technical directives: t-tag-open, t-tag-close, t-inner-content)   ┃ ┃
              ┃                                                                         ┃ ┃
              ┣━► _directives_eval_order (defined directive order)                      ┃ ┃
              ┣━► _compile_directives (loop)    Consume all remaining directives ◄━━━┓  ┃ ┃
              ┃  ┃                              (e.g.: to change the indentation)    ┃  ┃ ┃
              ┃  ┣━► _compile_directive                                              ┃  ┃ ┃
              ┃  ┃    ┗━► t-if            ━━► _compile_directive_if                 ━┫  ┃ ┃
              ┃  ┃    ┗━► t-foreach       ━━► _compile_directive_foreach            ━┛  ┃ ┃
              ┃  ┃    ┗━► t-inner-content ━━► _compile_directive_inner_content ◄━━━━━┓ ━┛ ┃
              ┃  ┃    ┗━► t-options       ━━► _compile_directive_options             ┃    ┃
              ┃  ┃    ┗━► t-call          ━━► _compile_directive_call               ━┫ ━━━┛
              ┃  ┃    ┗━► t-att           ━━► _compile_directive_att                 ┃
              ┃  ┃    ┗━► t-tag-open      ━━► _compile_directive_open          ◄━━┓  ┃
              ┃  ┃    ┗━► t-tag-close     ━━► _compile_directive_close         ◄━━┫  ┃
              ┃  ┃    ┗━► t-out           ━━► _compile_directive_out             ━┛ ━┫ ◄━━┓
              ┃  ┃    ┗━► t-field         ━━► _compile_directive_field               ┃   ━┫
              ┃  ┃    ┗━► t-esc           ━━► _compile_directive_esc                 ┃   ━┛
              ┃  ┃    ┗━► t-*             ━━► ...                                    ┃
              ┃  ┃                                                                   ┃
              ┗━━┻━► _compile_static_node                                           ━┛

closes odoo/odoo#81024

Related: odoo/enterprise#22942
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
2022-02-03 08:09:31 +00:00
Gorash 9ce5bc8881 [IMP] IrQweb: merge Qweb engine file qweb.py and ir_qweb.py
QWeb is the primary templating engine used by Odoo. It is an XML
templating engine and used mostly to generate XML, HTML fragments and
pages.

To create new XML template, please see :doc:`QWeb Templates documentation
<https://www.odoo.com/documentation/15.0/developer/reference/frontend/qweb.html>`

In **input** you have an XML template giving the corresponding input
etree. Each etree input nodes are used to generate a python function.
This fonction is called and will give the XML **output**.
The ``_compile`` method is responsible to generate the function from the
etree, that function is a python generator that yield one output line at a
time. This generator is consumed by ``_render``. The generated function is
orm cached.

In the graphic below you can see theresume of the call of the methods
performed in the IrQweb class.

    Odoo
     ┗━► _render (returns MarkupSafe)
        ┗━► _compile (returns function)                                        ◄━━━━━━━━━┓
           ┗━► _compile_node (returns code string array)                       ◄━━━━━━━┓ ┃
              ┃  (add technical directives: t-inner-content, t-tag)                    ┃ ┃
              ┣━► _directives_eval_order (defined directive order)                     ┃ ┃
              ┃                                                                        ┃ ┃
              ┣━► _compile_directives                              (recursive) ◄━━━━┓  ┃ ┃
              ┃  ┣━► _compile_directive                                             ┃  ┃ ┃
              ┃  ┃    ┗━► t-if            ━━► _compile_directive_if                ━┫  ┃ ┃
              ┃  ┃    ┗━► t-foreach       ━━► _compile_directive_foreach           ━┫  ┃ ┃
              ┃  ┃    ┗━► t-*             ━━► ...                                  ━┛  ┃ ┃
              ┃  ┃    ┗━► t-inner-content ━━► _compile_directive_inner_content ◄━━━━┓ ━┛ ┃
              ┃  ┃    ┗━► t-tag           ━━► _compile_directive_tag               ━┫    ┃
              ┃  ┃    ┗━► t-call          ━━► _compile_directive_call              ━┫ ━━━┛
              ┃  ┃    ┗━► t-out           ━━► _compile_directive_out           ◄━┓ ━┫
              ┃  ┃    ┗━► t-field         ━━► _compile_directive_field          ━┛  ┃
              ┃  ┃                                                                  ┃
              ┗━━┻━► _compile_static_node                                          ━┛

Part-of: odoo/odoo#81024
2022-02-03 08:09:31 +00:00
Paul Morelle 0b22f4deee [FIX] base: keep cache when regenerating assets
When the assets are regenerated, the previous attachments are deleted.
This may happen during the execution of t-call-assets directives in a
qweb view.

However, some properties that have been accessed with a sudo() were
accessible in the cache. Clearing the cache during the view rendering
could lead to access errors, which wouldn't be present without this
directive.

This commit works around this problem by removing these attachments with
a SQL query, without relying on the classic unlink, so that the cache is
preserved in this case.

By the way, sanitize the filename when marking it for deletion.

closes odoo/odoo#83570

X-original-commit: 627d508edfb62449067a05b0e3c1a0004f0af32c
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Paul Morelle <pmo@odoo.com>
2022-01-28 14:10:15 +00:00
Yannick TivisseandVictor Feyens 18952cdc76 [IMP] *: Convert single create method into multi
Taskid: 2703085
Part-of: odoo/odoo#80824
Co-authored-by: Victor Feyens <vfe@odoo.com>
2021-12-14 19:13:18 +00:00
Julien Castiaux 2e29a93503 [REF] core: remove http.addons_manifest
The http.addons_manifest is a map {module: manifest_dict} that is
populated upon the first http request. This map is basically a module
manifest cache with an extra `addons_path` key, the path of the module
on the file-system. This cache is eagerly populated upon the first http
request, the map is empty in non-http contextes (e.g. cron) which have
been a source of bugs (e.g. 50c8eb1).

A manifest cache is necessary because reading and parsing python files
from the file-system is not that cheap but there is no reason that cache
is located in `odoo.http`. A thin cache layer now wraps
`load_information_from_description_file()`/`load_manifest()` and is
lazily populated.

The `http.addons_manifest` have been removed. The extra `addons_path`
key is now present in the "normal" manifest. The `read_manifest()` was
hardly used so it has been deprecated. The only way to retrieve a
manifest is now `load_information_from_description_file()` which was
renamed `load_manifest()` (no cache) and `get_manifest()` (cache).

Side note about performances, the cache is necessary. Addons manifest
are read-only and reading + parsing python files from the file system is
not a cheap operation. Running the e-commerce tour
`@website_sale.test_04_admin_website_sale_tour` without cache on
`load_manifest()` requires 68,29 secs to complete on my laptop,
exceeding the default 1-minute time frame allowed in tests. Using a
cache the time is down to 36,53 secs. The performance impact is huge.

Part-of: odoo/odoo#79977
2021-12-14 12:39:54 +00:00
Samuel Degueldre e060539e77 [FIX] base: avoid crashing when removing empty list from bundle
Previously, having a remove directive in a bundle that didn't match any
file of that bundle's type would raise an error, saying that the path
we're attempting to remove could not be located in the current bundle.

This basically breaks the remove directive inside of xml/js-css hybrid
bundles, because getting the assets path in xml-mode will cause all
remove directives that only target css/js files to fail and vice versa.

This commit fixes that by only raising an error when there were files
that match the extensions currently enabled, and those files were not
found in the bundle.

closes odoo/odoo#75790

Signed-off-by: Simon Genin (ges@odoo) <ges@odoo.com>
2021-09-01 08:26:06 +00:00
Xavier-Do fdaee845d5 [FIX] base: remove options parameters
Regarding previous commit, the option parameter can be removed
from _get_asset_content api, followed by a nice snowball effect.

Part-of: odoo/odoo#75248
2021-08-26 10:21:10 +00:00
Gorash 7df343dd1b [IMP] base: QWeb _render return Markup unicode instead of utf8 bytes
In order to limit encoding decoding, the _render method returns a
unicode string in the markup safe object instead of a MarkupSafeBytes

closes odoo/odoo#68299

Related: odoo/upgrade#2454
Related: odoo/enterprise#17270
Signed-off-by: Antony Lesuisse (al) <al@openerp.com>
2021-08-03 16:20:22 +00:00
Julien Mougenot 3e3dce0eb8 [REF] *: rename assets 'glob' to 'path'
Rationale:
The majority of cases where an ir.asset is manually declared
outside of manifest files is to specifically add a single asset file.
This means developers are specifying a single asset *path*, and not a
glob expression. In this context, it seems better to name the filepath
field `path`, and document that it can be specified with a glob
expression when (seldom) needed, rather than making the exception appear
to be the norm - possibly puzzling many developers (What's a glob and
why do I need one?)

The doc is updated as well, and some spell-checking and wording
improvements were done too.

This required some adaptations to the existing `ir.asset` declarations:
- odoo/enterprise#17465
- odoo/design-themes#459

closes odoo/odoo#68695

Related: odoo/upgrade#2348
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
2021-04-07 20:39:10 +00:00
8cc066173d [IMP] *: Improve assets management
This commit changes the way assets are declared in Odoo modules.

Before: assets were declared in template files. Template bundles were
generated from primary templates, so technically any qweb template could
have been called as an asset bundle, with the 't-call-assets' directive.

Being standard qweb templates, they had access to standard HTML tags
(script, link, with or without raw scripts or style definition), qweb
directives (t-call, t-raw, etc.) and could be inherited by other
templates.

Now: assets are defined in the module's manifest and generated by the
't-call-assets' directive.

More information on the new system can be found on the updated user
documentation (see the "JavaScript Reference" section).

Task: 2352566

Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
2021-03-31 13:57:17 +02:00
Raphael ColletandVictor Feyens 840609975a [IMP] core: better way to set/update magic fields
The goal of this change is to simplify the code managing `create_date`
and `write_date` in methods `create()` and `write()`, and also to remove
weird behaviors caused by the way those fields were updated.

Assume we update a simple field on a record.  This adds pending updates
for the field and `write_date`.  However, the value of `write_date` is
not known yet: it will be updated as `NOW() AT TIME ZONE 'UTC'` in SQL.
So `write_date` is actually given a dummy value in pending updates, and
it is invalidated from cache, until its value is flushed to the database
and fetched again.

Now assume we access another field on the record, and that field is not
in cache.  The prefetching mechanism will read all column fields,
including `write_date`, and flush them first.

    # this adds pending updates foo: 42, write_uid: 1, write_date: False
    record.foo = 42

    # assume 'bar' is not in cache; this prefetches all column fields,
    # which flushes the pending updates above before reading them back
    result = record.bar

We can avoid flushing pending updates if the values read from database
do not overwrite existing values in cache.  If you assume that the value
of a pending update is in cache (in the example, `foo: 42`), you don't
need to flush the corresponding field.  Indeed, the value of `foo` will
remain 42 in cache, whatever its value in the database.  This assumption
(pending updates are in cache) is true for all fields *except* for
`write_date`: it is invalidated from cache, and given a dummy value in
pending updates.  This branch actually makes this assumption true for
all fields.  The avoidance of flushing pending updates will be done in
another commit.

In order to directly assign `write_date` its value, we use a cache for
the value `NOW() AT TIME ZONE 'UTC'` from the database.  This costs at
most one query per transaction, and potentially saves a few queries.

Co-authored-by: Victor Feyens <vfe@odoo.com>
2021-02-22 16:20:55 +00:00
Francois (fge) 4627c224ef [IMP] base: add sourcemap support for CSS files.
Improve the development experience in debug=assets mode by reducing the
number of requests to the server. We are adapting the solution used for
the JS files to the CSS files. This solution consists of no longer
sending all the files separately, but sending only the bundles
associated with their sourcemap. This allows us to keep the same
debugging experience while drastically reducing the number of requests
to the server.

Benchmark:
saas 14.2                   917 requests    domcontentloaded after 3.76s
master (bundling du js)     299 requests    domcontentloaded after 2.03s
branch (bundling js+css)    36  requests    domcontentloaded after 1.01s

Task id : 2463840

closes odoo/odoo#66169

Related: odoo/design-themes#453
Signed-off-by: Géry Debongnie (ged) <ged@openerp.com>
2021-02-18 08:51:02 +00:00
Simon Genin (ges)andFrancois 929fec3a54 [IMP] base: add support for native JS modules
Because of the way Odoo works at its core, we do not know before hand
which files will be loaded as an asset in the browser, because it
depends on the installed Odoo addons.  This is why it is historically
difficult to integrate Odoo with standard JS tooling, and this is why
Odoo needs to use a custom javascript module system.

However, there is a way to use native JS modules (and gain all the
benefits from it: IDE autocompletion, ease of refactoring, intellisense,
...): we can write JS as native JS modules, but convert them at runtime
into Odoo custom modules. This is exactly the strategy applied by this
PR.

This has a lot of benefits, but there is a downside: we can no longer
serve statically JS files in debug=assets.  This would be a dealbreaker,
if we did not have sourcemaps (implemented in the next commit).

This commit introduces the python code that will transpile native JS
modules into odoo JS modules.

Task ID: 2414902
PR: 63177

Co-authored-by: Francois (fge) <fge@odoo.com>
2021-02-15 10:18:11 +01:00
Lucas Perais (lpe) 9708c6e992 [IMP] bus: notify user when assets have changed
Use case:
When the server is restarted, the python is updated,
but some users may have an ongoing session in a browser tab
This may lead to code being unsynchronized and ultimately to some
odd bugs.

Purpose:
When we are in such a case, that is, the assets were recomputed
after a update of the code and a restart of the server by the request of another user,
notify connected users that assets have changed.
Then propose them to reload the page.

Known caveats:
- This is not a developer's feature.
Since assets computing is ORM cached, they have limited
opportunities to rebuild. Namely, the feature won't trigger
each time the JS has changed, rather, it will
when JS has changed AND the cache has been reset somehow (e.g. when the server is restarted).

- This not a portal/website feature either, but only in backend.
Business clients won't be notified that the JS has changed.

- While requests debug=assets do trigger a recomputing
of the *components* of bundles, they do not save a bundle
This means that the requests that sends the notification
cannot be debug=assets.

Task 2034462

closes odoo/odoo#39875

Signed-off-by: Mathieu Duckerts-Antoine <Polymorphe57@users.noreply.github.com>
2020-08-21 12:16:25 +00:00
Martin Trigaux d9287caf94 [IMP] *: convert to private methods
render, render_template, load, activity_schedule_with_view,
get_website_pages should all be private:
It should not be possible to render an aribtrary template only with
its name or id

Still need to render some qweb views from js so the method
render_template is kept public.
This explains why the website editor still need read access on
ir.ui.view as we want to allow any snippet to be rendered.
2020-05-14 13:59:10 +02:00
Martin Trigaux 49fbab5329 [IMP] base: split and deprecate load_lang
load_lang was a kind of hybrid method trying to active or creating a
language if not found. This was error prone.
Instead rely on two methods with clear purpose:
ResLang._create_lang(lang, lang_name=None)
  - create a new res.lang entry using the locale of the server
    return the res.lang record to match the API of _activate_lang

ResLang._active_lang(code)
  - activate the given code lang

Most of the time, _active_lang is what is expected

tools.trans_load_data and IrTranslation._load_module_terms no longer
activate the language if not active.
Loading the translations should be explicit on an activated language,
it is too error prone to silently activate/create a language if not
found.
Remove lang_name from trans_load_data as no longer needed.
2019-11-19 10:37:01 +01:00
Christophe Monniez 8d5da6e4be [IMP] tests: log a warning when HttpCase test in at_install
When loading a page on an existing starting database, registry is not
fully loaded causing potential error when trying to access model
existing in database (views, menitem, ...) since model added in last
loaded module does not exist in registry.

Thus, executing browser js test may lead to errors when executed during
an update on a database with other modules installed.

HTTPCase should be executed post_install to ensure that registry is
fully loaded to avoid this problem.

Since HTTPCase are slower than other test, it is also a good idea to
execute them at the end, in order to prioritize fast fail.

With this commit, a warning is isued if such a test class is tagged to
run at install time.

While at it, remove deprecated at_install and post_install helpers and
remove the deprecated phantom_js alias.

closes odoo/odoo#39462

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
2019-10-30 12:35:23 +00:00
Julien Castiaux 4f03a5f136 [FIX] *: remove old deprecated modules/functions
PEP-594 is deprecating a bunch of modules. As part of the cleanup, we
are also dealing with long deprecated modules, functions and aliases.

* `assert_` -> `assertTrue`
* `assertEquals` -> `assertEqual`
* `assertNotEquals` -> `assertNotEqual`
* `assertAlmostEquals` -> `assertAlmostEqual`
* `assertRaisesRegexp` -> `assertRaisesRegex`
* `assertRegexpMatches` -> `assertRegex`
* `base64.encodestring` -> `base64.encodebytes`
* `base64.decodestring` -> `base64.decodebytes`
* `inspect.getargspec` -> `inspect.signature`
* `inspect.formatargspec` -> `inspect.signature`
* `logging.warn` -> `logging.warning`

closes odoo/odoo#36863

Task: 2003936
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2019-09-17 11:36:42 +00:00
Raphael Collet 9920f20e4c [IMP] models: ORM speedup
This branch is the combination of several optimizations in the ORM:

* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;

* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;

* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);

* make method `modified` take advantage of inverse fields to inverse
dependencies;

* filter records by evaluating a domain on records in Python;

* a computed field with `readonly=False` behaves like a normal field
with an onchange method;

* computed fields are computed in superuser mode by default.

Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.

closes odoo/odoo#35659

Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
2019-08-20 12:43:59 +00:00
Martin Trigaux 85ee046b37 [IMP] *: use res.lang methods
get_installed and _lang_get_id are both ormcached and correctly check
the context

Retrieving a res.lang from a code is a frequent action that can be
achieved with _lang_get (cf previous commit).
Using _lang_get ensure the active_test in the context is correct and
is not poluted with another context propagation issue.
odoo/odoo#35490 discussion is an example of bad context propagation

closes odoo/odoo#35504

Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2019-08-07 13:06:17 +00:00
Adrian Torres 4b38cc6590 [REM] *: calls to @api.multi
Multi is the default api for methods, it is not necessary to explicitly
decorate methods with it, adds clutter and most people use it because
they see that the rest of the code uses it.

Done with `find . -type f -name '*.py' | xargs sed -i '/@api.multi/d'`
2019-07-17 14:13:12 +02:00
Xavier-Do fa45f88726 [FIX] core: avoid writting sources during tests
Future runbot improvement may share sources between build,
meaning that sources will be readonly to avoid any interaction.

_touch() was supposed to handle ro filesystem but
f7130556 introduced a new test that was not working
in this case.

Instead of trying to touch file and pseudomocking the result
if the filesystem is readonly, this commit add a real patch
on getmtime.

cherry-pick of b295723c99

closes odoo/odoo#34816

Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2019-07-12 08:45:04 +00:00
Christophe Simonis 9035fb8a2a [MERGE] forward port branch saas-12.4 up to 618ea1ab67 2019-06-18 10:24:25 +02:00
Lucas Lefèvre 6b93738299 [FIX] test_assetsbundle: Properly unpatch ir.attachment
`_patch_method` and `_revert_method` do not work properly.
This commit fixes a test which patches `create` and `unlink`
of `ir.attachment`.
The methods are not correctly unpatched.

One consequence:
if a module is installed after this test is executed and
the new module overrides `create` and `unlink`, the overritten
methods are never called.

closes odoo/odoo#33952

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2019-06-06 14:23:28 +00:00
Romain Derie 55e7b702bb [REM] base, website: remove deprecated code from qweb and bundle
The _get_asset method was deprecated since df4072b0b4
Last use of this deprecated method was removed with d7da1d648c

Also, remains and to_html were deprecated since 9cd982bcc8
test_assetsbundle was still using the deprecated method to test..

Closes #33432

Signed-off-by: Romain Derie (rde) <rde@odoo.com>
2019-06-12 14:02:34 +00:00
Christophe Monniez fec86404e7 [IMP] tests: replace browser_js success and failure strings
When evaluating js tests using the browser_js method, they are
considered a success when 'ok' is found in the console log and a failure
if 'error' is found instead.

This is not very robust and recently failed with this commit : 7410de111f
It adds a filter named 'Error' in a view, when the 'click_all' test
clicks on this filter, a message with the name of the filter is written
in the console, wrongly leading to a test failure.

With this commit, the browser_js method expects a log message in the
browser console with the explicit text "test successful".
On the other hand, any error in the console log will lead to a test
failure but a test can be forced to fail with the explicit error message
in the browser console "test failed".

As a bonus, the python logger should now also log browser js trace messages.

closes odoo/odoo#31158
2019-02-19 08:06:07 +00:00
Géry Debongnie a8cec47c93 [IMP] assets: remove css files split
Before this commit, our asset bundles had a special behaviour for
bundles of type css: because of an old limitation in IE9, css files had
to be split to make sure they had less than 4095 rules per stylesheets.

Since we no longer support IE9, this code can be removed, which should
slightly simplify the bundle code, and should slightly reduce the number
of requests required to properly load odoo.

closes odoo/odoo#27933
2018-10-19 09:18:28 +00:00
Martin Trigaux f7130556bb [REF] *: add support for RTL languages
מזל טוב

The direction parameter will now have a real impact on the web interface and
completly switch the direction of the interface.
Add the dependency rtlcss to switch the direction with a css postprocessor.

Two different assets will be generated for ltr or rtl
/web/content/<...>/web.assets_backend.0.css
and
/web/content/<...>/rtl/web.assets_backend.0.css
to be able to differentiate it easily

Adapt the css and views to be easier to switch direction

Render graph as it is, do not move y-axis right side, as per mathematical rule
value goes left from origin are considered negative, so keep graph as it is
for right to left language

add test cases for language direction specific assets bundle
if language direction is rtl then we also store css files in assets bundle
now we store two assets bundle, one for ltr and one for rtl language direction
test two assets bundles generated, also test two assets bundle generated for en_US and ar_SY
2018-08-10 16:32:50 +02:00
Christophe Simonis c8bb02141d [MERGE] forward port branch saas-11.3 up to 3cdcbce93c 2018-06-28 19:01:52 +02:00
Christophe Simonis 40836194c2 [MERGE] forward port branch saas-11.2 up to 12ddec3a76 2018-06-27 13:41:00 +02:00
Christophe Simonis 68326b15f9 [MERGE] forward port branch 11.0 up to c5ed372970 2018-06-21 12:18:40 +02:00
Christophe Matthieu 9cd982bcc8 [FIX] qweb: use cdn for asset files
Refactoring of qweb to call _post_processing_att for each nodes. This
change remove the crappy ovewrite in website module. The website overwrite
only the _post_processing_att to add the cdn parameters.

The static node (without t- attributes) can stay static (remove overwrite
of _is_static_node), the cdn is applied at the compile time for this
nodes instead of at the running time like the dynamic node.
2018-06-15 08:42:15 +02:00
qsm-odoo 2d63d2d75a [REF] test_assetsbundle: adapt bundle tests 2018-04-18 15:59:02 +02:00
Raphael Collet 7ea4f13f16 [REF] tests: TestCursor is now a proxy to a real Cursor
This allows rpc requests in `HttpCase` to use the cursor `self.cr`, which is
now shared between the Python test and the rpc requests.  This simplifies code
to prepare a JS test, and code to check the result of a JS tour.

Fixes #12237
2018-02-12 10:31:59 +01:00
Thibault Delavallée 1fbc29a641 [MOV] base: move ir_* models into models/ 2017-11-27 11:15:03 +01:00
Denis Ledoux 808c0e5dde [FIX] test_assetsbundle: handle readonly files
If the Odoo server runs with its sources read-only
for itself, this is not possible for the server
to touch the files contained in the assets.

In such a case, instead of touching the file,
we alter the attachment checksum, to simulate
a change within the modified date.

However, in such a case, we can only test that the
bundle is well generated again.
We can no longer check the change of version and modified date,
since the modified date of the files have not changed,
and therefore the checksum (which is based on the files
modified date, among others) neither.
2017-09-27 22:11:36 +02:00
Raphael Collet 9e64f9f951 [REF] openerp: move openerp to odoo 2016-09-02 17:28:12 +02:00