The main motivation is to be able to generate assets bundle outside
the t-call-assets call.
The need of an id in the url makes it mandatory to have an attachment
when adding the url in the page. Without this restriction, we can guess
the url without generating the assets.
This can also have other useful side effect:
There are corner case when a worked could have an invalid url in
cache because, if the transaction is rollbacked or if another request
generates the same attachment at the same time. This should be
partially solved by removing the id: The url remains valid even if the
attachment does not exist.
Note that the extra part of the url was made explicit, always there and
taking one / to remove complexity and ambiguity.
Note that an additional query appeared in .test_50_perf_sql_web_assets
because of the search, this but two of them were in _find_record. One of
them was an `exist`, not making much sense since we are not getting the
id from the attachment url anymore but from a search, and the other one
was prefetch of the "public field" since the call to _find_record does
not go in other cases (xmlid, website published, access token, ....). A
attachment of a asset is always public, and this part of the security
was moved to the search domain. The final result is one less query:
- one query to search
- one query to read the fields (_get_stream_from) (the prefetch could
actually be set to avoid prefetching everything)
Part-of: odoo/odoo#131353
Replace all the calls to get_resource_path to the better file_path or
directly use file_open when not needed
Doing both a get_resource_path and file_open means checking twice that
the file exists.
Doing a simple path concatenation before a file_open is safe.
If given to another method (e.g. etree.parse), calling file_path is
the prefered method.
Note that get_resource_path used to return False when the file does
not exists while file_path/file_open raises a FileNotFoundException
closesodoo/odoo#135607
Related: odoo/upgrade#5187
Related: odoo/enterprise#47475
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Before this commit, owl was in the linter's accepted global variables.
This allowed direct access to owl global object.
For instance, to use xml from owl, you could do :
`const { xml } = owl;`
or you could use it directly:
`owl.xml`
Now, owl is not accepted on linter's global variables anymore, so to
import xml, now you need to use a proper import:
`import { xml } from "@odoo/owl";`
task-id 3498859
closesodoo/odoo#137517
Related: odoo/enterprise#48364
Related: odoo/design-themes#709
Signed-off-by: Samuel Degueldre (sad) <sad@odoo.com>
As all the templates are now imported in the owl app, there is not need
anymore to specify the owl="1" attribute in the templates.
Part of task~3443861
Part-of: odoo/odoo#130467
One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.
Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.
The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.
Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.
The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.
We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.
Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.
Note that this poc is inspired from the long term cache but not all
use case where applie yet.
Part-of: odoo/odoo#119813
1. move cache to _get_asset_paths
The `_get_asset_content` cache has many cache key that are related to a
posprocessing of the `_get_asset_paths` result, the heavy part of this
method. Moving the cache to _get_asset_content will have the benefit
to create less duplicates entries in the ormcache as well as less cache
miss.
To simplify even further, the css and js parameters are removed since
they only filter the output of get_paths, the heavy part of globing the
file will be done before that. Anyway, they are both true when called
from _get_asset_content, and the only other call, in
`_get_related_bundle` don't really need to filter them since it is not
a critical part regarding performance, and the funtional result will
stay the same.
The initial orm cache key was using `_get_template_cache_keys`, a little
overkill and possibly creating duplicates entries again. The only
context key needed is website_id for `_get_related_assets`.
Note that it is not really enough, the orm cache key should actually
contain `request.session.get('force_website_id')` as well has
`request.httprequest.host`. This will be addressed latter since a nicer
solution would be to have website_id as a unique parameter computed
earlier.
2. better _get_asset_paths cache key
The orm cache key was simplified in previous point but there is still
one concern, the website_id depends on more parameters than that:
- request.session.get('force_website_id')
- request.httprequest.host
- existing websites
The idea here is to call `get_current_website` instead of using all
parameters that could define the webiste.
In the same spirit of `_get_template_cache_keys` `_assets_path_params`
can be overriden to give extra params that are usefull to list assets
path. Those params are computed before entering the method
`_get_asset_paths`. This may latter put at a higher level latter, in
get_asset_node, to simplify the _generate_asset_nodes_cache key.
3. better assets_node caches key
The main purpose of this part is to improve ormcache containing assets
nodes. The ormcache key contains
- to much context key
- missing session/host/env info
- unwanted boolean options.
- keys leading to the same cache value
The main goal being to reduce the size of the cache keys, decrease the
number of cache entries and improve the cache hit.
This will also make the behaviour more coherent and hopefully less bug
prone because of mismatch in parameters.
The main reason of the orm cache is the slowness of the validation of
the assets. This includes:
- listing files (dedicated orm cache)
- computing version
The cache key was depending on
- `debug`
The only relevant value for debug is "contains assets"
We dont need to differ between debug='', debug='1', debug='test',
and 'debug=assets', 'debug=tests,assets', ...
- `defer_load`, `lazy_load`, `media`
Those values are only useful to generate html node, a leightweight
operations that does not really needs to be in cache. `media` was also
used in the generation but it looks useless if we have the media on the
node. THIS NEEDS TO BE VALIDATED but in any case, since media is not
used to generate the url, it doesn't make sence to use it in the
generation.
The main idea to remove them from the ormcache key is simply to generate
the nodes outide the ormcached values.
-`async_load`
This one is similar to `defer_load` and `lazy_load` but it looks like
it wasn't used anymore. This was simply removed
- context.get('lang')
The only information needed is the direction, rtl or ltr. This means
en and fr languages, despite sharing the same css assets, will duplicate
the ormcache entries.
-`_get_template_cache_keys`
Only the lang and webiste where really relevant in this flow. Other
keys are actually useless in this flow.
Some information used in the generation where not in the orm cache key
- `self.env.user.lang` if there is no lang in the context
- `request.session.get('force_website_id')`
- `request.httprequest.host`
- ...
The proposed solution is to:
- extract any informùation needed from thecontext, request, environment
before entering the ormcache, reduce it to the minimal possible set of
values needed
```
rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl'
assets_params = self.env['ir.asset']._get_assets_params() # website_id
debug_assets = debug and 'assets' in debug
```
and remove a leightweight part of the logic
```
def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None):
links = self._get_asset_links(bundle, css=css, js=js, debug=debug)
return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media)
```
Where _get_asset_links is the cached part, and _links_to_nodes is the
lightweight part generating the nodes based on the `defer_load`, ....
Additionnal notes:
- data-asset-version and data-asset-bundle are removed from the node
since they don't seem to be used anymore since 65d70acdbf
- async_load is removed since there is no occurence of this in the code.
- a small hack is still needed to pass javascript content instead of
links, this is only to manage css compile error and will hopefully be
removed in the future.
- a context key is still in use to generate the bundle, the
`commit_assetsbundle` but it has no impact on content and will hopefully
be removed in the future.
4. Add test for ormcache hit/miss
In this context, hit/miss is about having the same cache key for the
same result. This test demonstrates the current state, were entries are
create in the ormcache only if the key is really different and will lead
to a different result.
5. remove cache invalidation
This cache invalidation is quite agressive since everytime an
assetbundle is updated, all workers will clear their cache.
The concerned cache by this clear_cache is `_generate_asset_nodes_cache`
throug `_get_asset_nodes`.
The cache is ignored, both in dev=xml and debug=assets.
This clear cache was made conditionnal in 553ea82f81 but this does
not solve an issue we can have in production.
Lets imagine a clean solution
- all sources are updated
- all workers are restarted.
The orm caches are all empty, but since the sources
changed, all bundles will be recomputed. This means that every bundle
updated in database with save_attachement will invalidate the cache of
all workers. Rendering a pdf report of any kind using a specific bundle
will invalidate all cache. Starting a debug=assets for the first time
will invalidate all cache, even if the cache is not used in this case.
But for a regenerated bundle we would expect the ormcache to be:
- empty (did not generate the same bundle yet)
- have the same value (concurrent generation of the same bundle)
Having a different value would mean that the bundle was generated with
another version of the sources. In this case it is maybe even better not
to invalidate the cache since it could lead to an invalidation war
between two workers.
The only case where invalidating this cache is useful is when a bundle
changes, Usually if an ir_asset is created, modified, ...
There is still another rare but possible possibility to have a 404 if
the transaction is rollbacked after populating the assets node cache.
In this case, we only need to clear the cache locally in case of
rollback.
Part-of: odoo/odoo#121376
One of the most costly part of a page loading when the ormcache is cold
is computing the assets node, the unique identifier of an attachment to
validate whether the existing attachment is still valid with the current
version of the static files.
This operation needs to glob assets path in the filesystem,
get the modification date, check attachments, ...
Right now this task is not really optimized and can take some time
because of an excessive number of glob on the filesystem, unnecessary
exists to define absolute path, double computation of file list and
modified times when getting js and css bundle separately, ...
A list of modifications mainly discussed in the pr message are made
with this commit to speedup things.
- split css and js unique
- prepare api for an in memory glob
- change api to propagate absolute path and meta information through
`ir.asset._get_paths`-> _get_asset_paths -> `_get_asset_content` ->
`AssetsBundle`
closesodoo/odoo#121159
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This commit makes the `dependencies` param of
`odoo.define` mandatory. It was optional and when
omitted, a regexp read the function to find the
dependencies. We can simplify it now almost all js
modules have been converted to esm.
The transpiler already adds the param for the es
modules except if the module has an alias.
task id: 3271352
closesodoo/odoo#119145
Related: odoo/enterprise#40040
Signed-off-by: Mathieu Duckerts-Antoine <dam@odoo.com>
This commit changes the js transpiler to write sync module.
The goal is to remove all async module and simplify the
module loader in the future.
task id: 3265979
closesodoo/odoo#119186
Signed-off-by: Géry Debongnie <ged@odoo.com>
Since saas 16.1, we added a more visible error information box when the
module system cannot find some dependencies, or if there is an error in
the JS code. This error box is very useful to understand a lot of
typical devlopment issues. However, it was occasionally displayed in
production code.
However, we sometimes observe that the error information box was
displayed, because the request to load an assets failed with a 404.
It is unclear in which circumstance this can happen, but it seems that
the following elements are involved:
- the user has an open tab with odoo, kept open for a while
- some js code is changed on the server, which causes the previous
assets to be deleted
- the user reload its open tabs, so the browser will load the /web page
from disk, which points to the old assets files
- then it will try to load all assets, with a 404 on one of the script
- the error information box is displayed
This commit intercepts the loading error and stop displaying the error
information box, which is basically the same behaviour as 16.0 and
before. We could force a reload in that case, but it seems dangerous,
since in case of errors, it could easily lead to an infinite loop.
closesodoo/odoo#117625
X-original-commit: 0a44b5b1e008b591a716efd26217c7db7360c37b
Signed-off-by: Géry Debongnie <ged@odoo.com>
After this commit, the transpiler will add the dependencies of a file
to the second argument of the odoo.define(...).
For example:
odoo.define("@test/test", ['@test/dep1', '@test/dep2'], async function (require)
With this change, the code for the module system in the browser will get
directly the dependencies instead of having to stringify the function
defining a module, and grep inside for the "require" words.
closesodoo/odoo#105651
Taskid: 3062390
Related: odoo/enterprise#34895
Signed-off-by: Géry Debongnie <ged@odoo.com>
The purpose of this improvement is to easily identify the module of a
test and to be able to run only the tests of a module.
closesodoo/odoo#105530
Task-id: 3062058
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
Commit [1] and [2] introduced an assets pregeneration to speed up the
testing suite on runbot.
It seems like it was chosen to always pregenerate the assets when a
module was installed/upgraded. This was done in the `test_assetsbundle`
module to only impact the tests flows, but it was not correct as it was
not only impacting the tests flows but also regular runbot usage.
Indeed, any `all` runbot (the default one) will have that module
installed.
It means that any module operation through regular flow in those DB will
trigger the assets pregeneration.
One of the problem is then that during a theme switch through the
website builder (or configurator), assets would be pregenerated, adding
~40 seconds to the processing time.
Since that theme install was already taking ~30 seconds on `all` runbot,
it sometimes now total over 60 seconds, triggering the `CPU Time Limit`.
You can find some benchmark and metrics on this commit PR.
Obviously, that's not fine because runbot, despite being our testing
tool/platform, is also one of our main demo tool.
That's especially true around the OXP period, where people are recording
video on it but will also use it to show demo to the customers.
Finally, there is no reason to not limit this assets pregeneration to
the test mode anyway.
Note that sometimes, the theme install is failing and responding with a
502 error after a long time, this is yet unsure if this was the cause
but it was during this investigation that this commit fix was found.
It is probably the cause though.
[1]: https://github.com/odoo/odoo/commit/c764c38d7caf300dce7509e87d2bd268eee0161d
[2]: https://github.com/odoo/odoo/commit/d47cf37b688117d6b2514b9b620787aa66312c61closesodoo/odoo#103105
X-original-commit: 5e77961e4768ef2d3d40f1600dd76a633a0e231b
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Co-authored-by: Xavier-Do <xdo@odoo.com>
The bundle name is irrelevant in the bundle content and will prevent
attachment to store the same file if the bundles are exactly the same.
closesodoo/odoo#102502
X-original-commit: b1d57adf6ff358aa79f41817f4f7bfe60f9e5ac0
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The current asset_bundle version uses the last modified date.
This can be problematic in some case.
Even if it is a long time issue, the problem was rediscovered on runbot
with a commit being in the future. Runbot will export all file and set
the write date of the file to the commit date. The main purpose is to
have deterministic bundle version between different builds. This also
allows to generate assets bundle once at install for all post install
subbuild.
The issue here is that the commit date was greater than now(), meaning
that some tests setting custom css in attachment won't trigger the
regeneration of assets bundle. This wasn't really noticeable before
assets pregeneration.
This is not the first time strange issues occurs because of the
last modified logic.
This commit combined all last_modified to generate the bundle
version.
The current adaptation is quick and dirty and this will be reworked in
another post 16.0 freeze assets refactoring and cleanup.
closesodoo/odoo#100160
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
XML files are now declared in python module manifests. During the qweb
't-call-asset' directive, assetbundle will fetch the declared xml files,
apply the inheritance (t-inherit) and create a javascript service (for
eg: 'web.assets_backend.bundle.xml') which is added at the end of the
*.js mimifier file.
When the debug mode is activated, comments are added in the template
indicating which file the template comes from as well as the
inheritances applied to it.
****
JavaScript:
assets.js (module @web/core/assets) takes care of loading libraries,
javascripts and styles.
`loadJS(url)` (loads the javascript and returns a resolved promise when
the templates are also loaded via the '*.bundle.xml' service)
`loadCSS(url)` (loads the style a resolved promise when the file is
loaded)
`loadXML(xml, app=assets.defaultApp)` (load template into
application/owl, used by the `*.bundle.xml` services)
`getBundle(bundleName)` (get the bundle descriptor)
`loadBundle(desc)` (load the files and bundle from a descriptor)
templates (XML element content all owl templates)
A new `ready(serviceName)` method on boot.js lets you know when a
service is loaded are the require.
The xmlDependencies attribute no longer exists.
Python:
The xmls taken into account by assetbundle.py, applying `t-inherit`
inheritances and adding an `name_of_the_bundle.bundle.xml` service in
the generated JavaScript file.
****
Every manifest changes is into the next commit, except 'web_tour' in
this current commit as example.
Part-of: odoo/odoo#95500
The pregenerate of assets_bundle should be done maximum two times
- at the end of an install/update if the module test_assetsbundle is
installed
- at the beginning of post_install tests
Currently, during the nightly (install and all tests are ran at the
same time), the pregeration of assets is triggered at the end of some
tests
File "/home/xdo/osrc/master/odoo/odoo/tests/common.py", line 319, in _tearDownPreviousClass
super()._tearDownPreviousClass(test, result)
File "/usr/lib/python3.8/unittest/suite.py", line 300, in _tearDownPreviousClass
previousClass.doClassCleanups()
File "/usr/lib/python3.8/unittest/case.py", line 731, in doClassCleanups
function(*args, **kwargs)
File "/home/xdo/osrc/master/odoo/odoo/modules/registry.py", line 704, in reset_changes
self.setup_models(cr)
File "/home/xdo/osrc/master/odoo/odoo/modules/registry.py", line 306, in setup_models
model._register_hook()
File "/home/xdo/osrc/master/odoo/odoo/addons/test_assetsbundle/models/ir_qweb.py", line 13, in _register_hook
The registry.updated_modules is set to allow post_install tests
to select the right tests to execute. This is a way to define if we
updated some modules loading this registry. This list is unfortunatelly
not emptied before running the test, meaning that if register hook are
called on the same registry again, pregenerate will be executed
a second time.
A solution here is to check if the registry is not ready,
meaning that this is the load_modules call to register_hook
and not the setup_models one
closesodoo/odoo#100079
Related: odoo/enterprise#31271
Signed-off-by: Raphael Collet <rco@odoo.com>
During tests on runbot, the main reason tours are slow to start is
because the first loading of "/web" need to generate assets bundles.
Generation can take up to 5 seconds time the number of tour.
Before this, the first requests to /web takes around 7 seconds on
runbot and the next ones less than 1 second.
After this commit, the first request to /web takes around 1.5 seconds
Note that the main difficulty is to choose when to generate the assets.
(With optimisations from following commits) the generation time is
arround 30 seconds (21 css + 12 js) the first time, for all modules.
If the attachements exists the generation time is arround 3 seconds
(2.5 js + 0.5 css) mainly because of globs to find usefull files.
In practice for runbot the ideal would be to generate them at
the end of the install so that it is shared for all post_install builds.
Doint it at the end of an install is not wanted in all cases, saas
pregenerated templates and upgrade may avoid doing that.
Doing it in a special runbot step (subcommand) is not possible for niglty
execution (no split, in one go). This needs to be in the code.
It is also not practical for devs wanting to have the same behaviour on
a local machine.
A solution to make the test conditionnal at install is to base the
condion on an existing test_module. test_assetsbundle is a good
candidate here. On runbot, the at_install step (befor split) will
automatically generate assets with this solution.
Assets are also generated before post_install tests. This should be fast
if they already exists and ensure that assets are up to date after
modifying sources locally or after downloading a database on runbot.
It should be fast enough for small database and may be even faster
in the future for small diffs.
Part-of: odoo/odoo#99176
Previously, when importing both the default export and named imports
with a trailing comma, we would transpile into an object where the first
keys were the named imports, including any trailing comma, and
concatenating with the default import with a leading comma, this means
that the trailing comma would transpile to two successive commas which
is not valid syntax.
This commit fixes that by putting the default import first, and adding
the named imports after, preserving the trailing comma if any, but
precluding the possibility that we get two commas back to back.
closesodoo/odoo#93693
X-original-commit: 798d2f2c6e8a496f4ce0cd3c4f1609693e692254
Signed-off-by: Géry Debongnie <ged@odoo.com>
Signed-off-by: Samuel Degueldre <sad@odoo.com>
Remove most values uselessly specified because giving the same value as
the default one (see _DEFAULT_MANIFEST in odoo/modules/module.py)
* auto_install is Falsy by default
* author is Odoo SA by default
* summary & description are empty strings by default
* application is False by default
* test, demo, depends and data are empty lists by default
This will reduce noise/inconsistencies between manifests specifications,
simplify analysis of manifests content, ...
closesodoo/odoo#90209
Related: odoo/enterprise#26807
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Previously, the useAssets hook would load the templates from the given
bundles into the qweb instance that was in the environment. In owl 2,
qweb is no longer in the environment, and templates need to be loaded in
the app.
This commit fixes that by exporting a function to load templates in the
main app from assets.js, when the main app is created, it should be
written on the function in a similar vein to renderToString, so that all
further calls to the function will load the templates in that app.
This commit also adds an actual lazy-loading test in the
test_assetsbundle module, as we don't want to pollute the manifest of
web with bundles that only exist for one specific test.
closesodoo/odoo#90122
Related: odoo/enterprise#26776
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
There were inconsistencies in the calls to `_render`.
* the view context could contain information that misled developers.
Indeed, the context and value of the view are not supposed to be found
in the rendering. Thus by calling `ir.qweb` with the name of the
template, we ensure that there is no unwanted information and in
addition the cache key is that of the name of the template which saves
a query.
* the context used for rendering was modified by a method on
`ir.ui.view`, except this is not information used by this model. There
is now a `_prepare_environment` method residing on `ir.qweb`. This
method allows to modify the value dictionary as well as the context in
which the rendering will be done. This preparation of the data as well
as my security check is done only once per rendering. This also saves
some queries
* Freeze options for rendering were inconsistent. It could be that
options on which rendering depends were not part of the cache key. Thus,
depending on the user who generated the generation of the rendering
function, there was or was not information in the template. For example
for automatic branding. This is no longer possible, because it is the
context that is used. The options serving as a cache key are only
recorded for information (for the profiling system for example). A
simplification of the `ir.qweb.field` models could be made.
The report rendering and call `ir.qweb` instead of `ir.ui.view`.
Part-of: odoo/odoo#85110
Previously, importing the default export on the same line as the named
exports was not supported, this commit adds support for that in order to
allow removing duplicate imports.
before:
import something from "path";
import { A } from "path";
after:
import something, { A } from "path";
Part-of: odoo/odoo#84530
QWeb is the primary templating engine used by Odoo. It is an XML
templating engine and used mostly to generate XML, HTML fragments and
pages.
To create new XML template, please see :doc:`QWeb Templates documentation
<https://www.odoo.com/documentation/15.0/developer/reference/frontend/qweb.html>`
In **input** you have an XML template giving the corresponding input
etree. Each etree input nodes are used to generate a python function.
This fonction is called and will give the XML **output**.
The ``_compile`` method is responsible to generate the function from the
etree, that function is a python generator that yield one output line at a
time. This generator is consumed by ``_render``. The generated function is
orm cached.
In the graphic below you can see theresume of the call of the methods
performed in the IrQweb class.
Odoo
┗━► _render (returns MarkupSafe)
┗━► _compile (returns function) ◄━━━━━━━━━┓
┗━► _compile_node (returns code string array) ◄━━━━━━━┓ ┃
┃ (add technical directives: t-inner-content, t-tag) ┃ ┃
┣━► _directives_eval_order (defined directive order) ┃ ┃
┃ ┃ ┃
┣━► _compile_directives (recursive) ◄━━━━┓ ┃ ┃
┃ ┣━► _compile_directive ┃ ┃ ┃
┃ ┃ ┗━► t-if ━━► _compile_directive_if ━┫ ┃ ┃
┃ ┃ ┗━► t-foreach ━━► _compile_directive_foreach ━┫ ┃ ┃
┃ ┃ ┗━► t-* ━━► ... ━┛ ┃ ┃
┃ ┃ ┗━► t-inner-content ━━► _compile_directive_inner_content ◄━━━━┓ ━┛ ┃
┃ ┃ ┗━► t-tag ━━► _compile_directive_tag ━┫ ┃
┃ ┃ ┗━► t-call ━━► _compile_directive_call ━┫ ━━━┛
┃ ┃ ┗━► t-out ━━► _compile_directive_out ◄━┓ ━┫
┃ ┃ ┗━► t-field ━━► _compile_directive_field ━┛ ┃
┃ ┃ ┃
┗━━┻━► _compile_static_node ━┛
Part-of: odoo/odoo#81024
When the assets are regenerated, the previous attachments are deleted.
This may happen during the execution of t-call-assets directives in a
qweb view.
However, some properties that have been accessed with a sudo() were
accessible in the cache. Clearing the cache during the view rendering
could lead to access errors, which wouldn't be present without this
directive.
This commit works around this problem by removing these attachments with
a SQL query, without relying on the classic unlink, so that the cache is
preserved in this case.
By the way, sanitize the filename when marking it for deletion.
closesodoo/odoo#83570
X-original-commit: 627d508edfb62449067a05b0e3c1a0004f0af32c
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Paul Morelle <pmo@odoo.com>
The http.addons_manifest is a map {module: manifest_dict} that is
populated upon the first http request. This map is basically a module
manifest cache with an extra `addons_path` key, the path of the module
on the file-system. This cache is eagerly populated upon the first http
request, the map is empty in non-http contextes (e.g. cron) which have
been a source of bugs (e.g. 50c8eb1).
A manifest cache is necessary because reading and parsing python files
from the file-system is not that cheap but there is no reason that cache
is located in `odoo.http`. A thin cache layer now wraps
`load_information_from_description_file()`/`load_manifest()` and is
lazily populated.
The `http.addons_manifest` have been removed. The extra `addons_path`
key is now present in the "normal" manifest. The `read_manifest()` was
hardly used so it has been deprecated. The only way to retrieve a
manifest is now `load_information_from_description_file()` which was
renamed `load_manifest()` (no cache) and `get_manifest()` (cache).
Side note about performances, the cache is necessary. Addons manifest
are read-only and reading + parsing python files from the file system is
not a cheap operation. Running the e-commerce tour
`@website_sale.test_04_admin_website_sale_tour` without cache on
`load_manifest()` requires 68,29 secs to complete on my laptop,
exceeding the default 1-minute time frame allowed in tests. Using a
cache the time is down to 36,53 secs. The performance impact is huge.
Part-of: odoo/odoo#79977
*: base_address_city, base_address_extended, bus, crm, im_livechat,
l10n_ae_pos, lunch, pos_restaurant_adyen, test_assetsbundle,
test_converter, test_lint.
It is spelled `auto_install`, the `complexity` key is long gone, `qweb`
has been moved to `assets: {'web.assets_qweb': []}`. `js` and `css` are
long gone too, `maintainer` is redundant with `author` which is
"Odoo S.A." by default already, the `certificate` key is long gone.
closesodoo/odoo#80988
Related: odoo/enterprise#22766
Signed-off-by: Julien Castiaux <juc@odoo.com>
After this commit, the js_transpiler will support exported hoisting
function. So it will be possible to define a hoisting function and
exported in an @odoo-module.
Example of exported hoisting function which wasn't supported before
this commit:
```
hoisted();
export function hoisted() {
...
};
```
closesodoo/odoo#80965
X-original-commit: 40ef9f7aeac8048cce8f002c84181abc64bcbe87
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
Previously, having a remove directive in a bundle that didn't match any
file of that bundle's type would raise an error, saying that the path
we're attempting to remove could not be located in the current bundle.
This basically breaks the remove directive inside of xml/js-css hybrid
bundles, because getting the assets path in xml-mode will cause all
remove directives that only target css/js files to fail and vice versa.
This commit fixes that by only raising an error when there were files
that match the extensions currently enabled, and those files were not
found in the bundle.
closesodoo/odoo#75790
Signed-off-by: Simon Genin (ges@odoo) <ges@odoo.com>
Regarding previous commit, the option parameter can be removed
from _get_asset_content api, followed by a nice snowball effect.
Part-of: odoo/odoo#75248
In order to limit encoding decoding, the _render method returns a
unicode string in the markup safe object instead of a MarkupSafeBytes
closesodoo/odoo#68299
Related: odoo/upgrade#2454
Related: odoo/enterprise#17270
Signed-off-by: Antony Lesuisse (al) <al@openerp.com>
Transpile odoo-modules into async function rather than simple functions.
The aim of this change is to support top-level await.
closesodoo/odoo#74629
Signed-off-by: Simon Genin (ges@odoo) <ges@odoo.com>
The license is missing in most enterprise manifest so
the decision was taken to make it explicit in all cases.
When not defined, a warning will be triggered starting from
14.0 when falling back on the default LGPL-3.
closesodoo/odoo#74245
Related: odoo/design-themes#48
Related: odoo/enterprise#19862
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This commit is to fix two problems when transpiling an @odoo-module file.
1. When a file exports an object with a long name, the regex EXPORT_FROM_RE will cause catastrophic backtracking. The regex will take a long time to resolve.
Example:
```js
export {a, aReallyVeryLongNameElement};
```
2. When a file exports an object without ending the line with ';'. The regex EXPORT_OBJECT_RE will remove spaces and line breaks that follow this kind of export.
Example:
Input:
```js
export {a}
xxx
```
Output:
```js
Object.assign(__export, {a})xxx
```
Output after this commit:
```js
Object.assign(__export, {a})
xxx
```
closesodoo/odoo#68868
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Rationale:
The majority of cases where an ir.asset is manually declared
outside of manifest files is to specifically add a single asset file.
This means developers are specifying a single asset *path*, and not a
glob expression. In this context, it seems better to name the filepath
field `path`, and document that it can be specified with a glob
expression when (seldom) needed, rather than making the exception appear
to be the norm - possibly puzzling many developers (What's a glob and
why do I need one?)
The doc is updated as well, and some spell-checking and wording
improvements were done too.
This required some adaptations to the existing `ir.asset` declarations:
- odoo/enterprise#17465
- odoo/design-themes#459closesodoo/odoo#68695
Related: odoo/upgrade#2348
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Conversion of all modules to the new manifest assets declaration.
Part of task: 2352566
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
This commit changes the way assets are declared in Odoo modules.
Before: assets were declared in template files. Template bundles were
generated from primary templates, so technically any qweb template could
have been called as an asset bundle, with the 't-call-assets' directive.
Being standard qweb templates, they had access to standard HTML tags
(script, link, with or without raw scripts or style definition), qweb
directives (t-call, t-raw, etc.) and could be inherited by other
templates.
Now: assets are defined in the module's manifest and generated by the
't-call-assets' directive.
More information on the new system can be found on the updated user
documentation (see the "JavaScript Reference" section).
Task: 2352566
Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
In a @odoo-module js file, importing a legacy module that did not have
a name respecting the "addon.name" format was not supported.
This commit will support all keys that do not start with a "." or "@".
This commit will also improve some regexes allowing to extract path.
Legacy module name supported:
import X from "some/path"
import X from "addon.name"
Legacy module name not supported:
(relative path and @module/path are only used to import @odoo-module js
files)
import X from "@some/path"
import X from "./some/path"
import X from "../../some/path"
closesodoo/odoo#68152
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
When you define an @odoo-module with an alias without any space between
the alias and the **/, the alias contains the **/.
The purpose of this commit is to no longer add the **/ to the alias.
Example:
/** @odoo-module alias=web.base**/
Before:
odoo.define(`web.base**/`, function(require) {
After:
odoo.define(`web.base`, function(require) {
closesodoo/odoo#66929
Signed-off-by: Géry Debongnie (ged) <ged@openerp.com>
The goal of this change is to simplify the code managing `create_date`
and `write_date` in methods `create()` and `write()`, and also to remove
weird behaviors caused by the way those fields were updated.
Assume we update a simple field on a record. This adds pending updates
for the field and `write_date`. However, the value of `write_date` is
not known yet: it will be updated as `NOW() AT TIME ZONE 'UTC'` in SQL.
So `write_date` is actually given a dummy value in pending updates, and
it is invalidated from cache, until its value is flushed to the database
and fetched again.
Now assume we access another field on the record, and that field is not
in cache. The prefetching mechanism will read all column fields,
including `write_date`, and flush them first.
# this adds pending updates foo: 42, write_uid: 1, write_date: False
record.foo = 42
# assume 'bar' is not in cache; this prefetches all column fields,
# which flushes the pending updates above before reading them back
result = record.bar
We can avoid flushing pending updates if the values read from database
do not overwrite existing values in cache. If you assume that the value
of a pending update is in cache (in the example, `foo: 42`), you don't
need to flush the corresponding field. Indeed, the value of `foo` will
remain 42 in cache, whatever its value in the database. This assumption
(pending updates are in cache) is true for all fields *except* for
`write_date`: it is invalidated from cache, and given a dummy value in
pending updates. This branch actually makes this assumption true for
all fields. The avoidance of flushing pending updates will be done in
another commit.
In order to directly assign `write_date` its value, we use a cache for
the value `NOW() AT TIME ZONE 'UTC'` from the database. This costs at
most one query per transaction, and potentially saves a few queries.
Co-authored-by: Victor Feyens <vfe@odoo.com>
Improve the development experience in debug=assets mode by reducing the
number of requests to the server. We are adapting the solution used for
the JS files to the CSS files. This solution consists of no longer
sending all the files separately, but sending only the bundles
associated with their sourcemap. This allows us to keep the same
debugging experience while drastically reducing the number of requests
to the server.
Benchmark:
saas 14.2 917 requests domcontentloaded after 3.76s
master (bundling du js) 299 requests domcontentloaded after 2.03s
branch (bundling js+css) 36 requests domcontentloaded after 1.01s
Task id : 2463840
closesodoo/odoo#66169
Related: odoo/design-themes#453
Signed-off-by: Géry Debongnie (ged) <ged@openerp.com>
This commit fixes two unrelated small issues in the JS transpiler regexps:
- handle special chars in file url (so, one can now use a "-" in a file url)
- issue with the @odoo-module detection: it required 2 spaces after the @odoo-module word, because of the way the regexp was constructed. It also required spaces at the start of the line. All those have been fixed.
closesodoo/odoo#66255
Signed-off-by: Géry Debongnie (ged) <ged@openerp.com>
Because of the way Odoo works at its core, we do not know before hand
which files will be loaded as an asset in the browser, because it
depends on the installed Odoo addons. This is why it is historically
difficult to integrate Odoo with standard JS tooling, and this is why
Odoo needs to use a custom javascript module system.
However, there is a way to use native JS modules (and gain all the
benefits from it: IDE autocompletion, ease of refactoring, intellisense,
...): we can write JS as native JS modules, but convert them at runtime
into Odoo custom modules. This is exactly the strategy applied by this
PR.
This has a lot of benefits, but there is a downside: we can no longer
serve statically JS files in debug=assets. This would be a dealbreaker,
if we did not have sourcemaps (implemented in the next commit).
This commit introduces the python code that will transpile native JS
modules into odoo JS modules.
Task ID: 2414902
PR: 63177
Co-authored-by: Francois (fge) <fge@odoo.com>
Use case:
When the server is restarted, the python is updated,
but some users may have an ongoing session in a browser tab
This may lead to code being unsynchronized and ultimately to some
odd bugs.
Purpose:
When we are in such a case, that is, the assets were recomputed
after a update of the code and a restart of the server by the request of another user,
notify connected users that assets have changed.
Then propose them to reload the page.
Known caveats:
- This is not a developer's feature.
Since assets computing is ORM cached, they have limited
opportunities to rebuild. Namely, the feature won't trigger
each time the JS has changed, rather, it will
when JS has changed AND the cache has been reset somehow (e.g. when the server is restarted).
- This not a portal/website feature either, but only in backend.
Business clients won't be notified that the JS has changed.
- While requests debug=assets do trigger a recomputing
of the *components* of bundles, they do not save a bundle
This means that the requests that sends the notification
cannot be debug=assets.
Task 2034462
closesodoo/odoo#39875
Signed-off-by: Mathieu Duckerts-Antoine <Polymorphe57@users.noreply.github.com>
Xavier-Do