The main motivation is to be able to generate assets bundle outside
the t-call-assets call.
The need of an id in the url makes it mandatory to have an attachment
when adding the url in the page. Without this restriction, we can guess
the url without generating the assets.
This can also have other useful side effect:
There are corner case when a worked could have an invalid url in
cache because, if the transaction is rollbacked or if another request
generates the same attachment at the same time. This should be
partially solved by removing the id: The url remains valid even if the
attachment does not exist.
Note that the extra part of the url was made explicit, always there and
taking one / to remove complexity and ambiguity.
Note that an additional query appeared in .test_50_perf_sql_web_assets
because of the search, this but two of them were in _find_record. One of
them was an `exist`, not making much sense since we are not getting the
id from the attachment url anymore but from a search, and the other one
was prefetch of the "public field" since the call to _find_record does
not go in other cases (xmlid, website published, access token, ....). A
attachment of a asset is always public, and this part of the security
was moved to the search domain. The final result is one less query:
- one query to search
- one query to read the fields (_get_stream_from) (the prefetch could
actually be set to avoid prefetching everything)
Part-of: odoo/odoo#131353
Fixes a large number of cases where strings are translated then
formatted, instead of letting `_()` do the formatting internally,
which allows it to recover from incorrect translations (missing,
broken, or extra placeholders).
Also
- removes translation markers entirely when there's nothing to
translate e.g. `_("%s - %s")` is not useful
- fixes a few messes which lead to only partial translatability
(DRY is generally a bad idea when translations are involved, even
more so when you don't make the variable part translatable)
- fixes a few nearby issues noticed at the same time
- replaces a few `"%s"` by `%r`, which should automatically quote
strings relatively appropriately
- fixes translated strings which use `\` to escape a newline (in order
to fill-paragraph): `\` escapes only the newline, if the
continuation string is indented this results in a bunch of spaces
ending in the string to translate, which is pretty garbage for the
translator, using implicit concatenation works much better
Note: some of the updates revert f-string parameters to %, because
babel (2.9) apparently has trouble with f-strings and blows up trying
to extract them.
Not in scope:
Helping translators fix translatable strings e.g. any translation
string with more than one placeholder probably should use keyword
placeholders
- Provides more context / data to the translator to make sense of the
sentence.
- Allows reordering the translated terms, which can be necessary
depending on the sentence and language.
closesodoo/odoo#139314
Related: odoo/enterprise#49311
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Replace all the calls to get_resource_path to the better file_path or
directly use file_open when not needed
Doing both a get_resource_path and file_open means checking twice that
the file exists.
Doing a simple path concatenation before a file_open is safe.
If given to another method (e.g. etree.parse), calling file_path is
the prefered method.
Note that get_resource_path used to return False when the file does
not exists while file_path/file_open raises a FileNotFoundException
closesodoo/odoo#135607
Related: odoo/upgrade#5187
Related: odoo/enterprise#47475
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
This commit reworks a little bit the backend assets to remove a
bundle and thus save a call at webclient startup. The bundle
"assets_backend_prod_only" existed only to allow to add files in
production, but not in the tests (typically, the file that spawns
the webclient).
This commit introduces a new bundle "web.assets_web" that contains
"assets_backend" and the few files that we only want in production.
In the /web page, we now load "assets_web" instead of
"assets_backend" and "assets_backend_prod_only". In the /web/tests
page, we keep loading "assets_backend", which is now directly
included into "web.tests_assets".
For the sake of consistency, this commit also renames the dark
mode bundle "dark_mode_assets_backend" into "assets_web_dark".
closesodoo/odoo#135204
Related: odoo/enterprise#47316
Signed-off-by: Samuel Degueldre (sad) <sad@odoo.com>
In this commit, the loadXML function has been removed. We use registry with
xml_templates to load XML templates for OWL Apps.
The goal of task is to remove loadXML and getBundle from assets to simplify
the understanding of assets api.
task-3266441
closesodoo/odoo#134520
Related: odoo/enterprise#47001
Signed-off-by: Michaël Mattiello (mcm) <mcm@odoo.com>
Changes in default can be bothersome, embed a full baseline
configuration for reliability.
closesodoo/odoo#27926
Signed-off-by: Pierre Masereel <pim@odoo.com>
*: auth_password_policy, bus, event, im_livechat, mail, mass_mailing,
mrp_subcontracting, point_of_sale, pos_self_order, project, stock,
survey, web, web_editor, web_tour, website, website_event,
website_forum, website_sale, website_slides, base
Historically, the web.assets_common bundle was used to contain assets
that were needed by both the frontend and the backend. In practice, this
caused a bunch of issues where people would add things in assets common
that were not needed by both, and it was also abused as a way to get
bootstrap working in unrelated places by only using that bundle's css.
Because of this, as a first step, the assets_common stop being used in
the frontend, but was left everywhere else.
This commit removes the bundle completely, and moves the files that used
to be in that bundle in the other bundles that need them, this will
allow those bundles to evolve independently going forward.
in im_livechat and mail, some of the unneeded legacy code was removed, this
allows us to avoind including all of the legacy code from web in the
livechat embed bundle and in the dicuss public bundle respectively.
closesodoo/odoo#132190
Related: odoo/enterprise#45884
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.
Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.
The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.
Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.
The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.
We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.
Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.
Note that this poc is inspired from the long term cache but not all
use case where applie yet.
Part-of: odoo/odoo#119813
This is a proposal to try to find a similar asset before generating one.
If get_attachments fails, the next step will be to generate the
attachments from scratch, a slow operations.
When creating a new website, all assetsbundle would actually be
similar to their version without website, but the url is different.
This can be visible because the first loading of /web is slow after
creating a new website: the website_id is forced in the session
and the assets_backend are regnerated, identical to the original ones.
This commit proposes to try to find an attachments with differents extra
but the same uniquifier when possible and copy it's content.
Note that just returning the other attachement url may work, but it
would be confusing to randomly have links to assets comming from another
website_id. This would also be a problem if the original attachment is
unlinked, forcing to recompute it for other websites.
Good to know, since the content is the same, no duplication of the
content should appear in the filestore, just an entry in the database.
Part-of: odoo/odoo#121376
1. move cache to _get_asset_paths
The `_get_asset_content` cache has many cache key that are related to a
posprocessing of the `_get_asset_paths` result, the heavy part of this
method. Moving the cache to _get_asset_content will have the benefit
to create less duplicates entries in the ormcache as well as less cache
miss.
To simplify even further, the css and js parameters are removed since
they only filter the output of get_paths, the heavy part of globing the
file will be done before that. Anyway, they are both true when called
from _get_asset_content, and the only other call, in
`_get_related_bundle` don't really need to filter them since it is not
a critical part regarding performance, and the funtional result will
stay the same.
The initial orm cache key was using `_get_template_cache_keys`, a little
overkill and possibly creating duplicates entries again. The only
context key needed is website_id for `_get_related_assets`.
Note that it is not really enough, the orm cache key should actually
contain `request.session.get('force_website_id')` as well has
`request.httprequest.host`. This will be addressed latter since a nicer
solution would be to have website_id as a unique parameter computed
earlier.
2. better _get_asset_paths cache key
The orm cache key was simplified in previous point but there is still
one concern, the website_id depends on more parameters than that:
- request.session.get('force_website_id')
- request.httprequest.host
- existing websites
The idea here is to call `get_current_website` instead of using all
parameters that could define the webiste.
In the same spirit of `_get_template_cache_keys` `_assets_path_params`
can be overriden to give extra params that are usefull to list assets
path. Those params are computed before entering the method
`_get_asset_paths`. This may latter put at a higher level latter, in
get_asset_node, to simplify the _generate_asset_nodes_cache key.
3. better assets_node caches key
The main purpose of this part is to improve ormcache containing assets
nodes. The ormcache key contains
- to much context key
- missing session/host/env info
- unwanted boolean options.
- keys leading to the same cache value
The main goal being to reduce the size of the cache keys, decrease the
number of cache entries and improve the cache hit.
This will also make the behaviour more coherent and hopefully less bug
prone because of mismatch in parameters.
The main reason of the orm cache is the slowness of the validation of
the assets. This includes:
- listing files (dedicated orm cache)
- computing version
The cache key was depending on
- `debug`
The only relevant value for debug is "contains assets"
We dont need to differ between debug='', debug='1', debug='test',
and 'debug=assets', 'debug=tests,assets', ...
- `defer_load`, `lazy_load`, `media`
Those values are only useful to generate html node, a leightweight
operations that does not really needs to be in cache. `media` was also
used in the generation but it looks useless if we have the media on the
node. THIS NEEDS TO BE VALIDATED but in any case, since media is not
used to generate the url, it doesn't make sence to use it in the
generation.
The main idea to remove them from the ormcache key is simply to generate
the nodes outide the ormcached values.
-`async_load`
This one is similar to `defer_load` and `lazy_load` but it looks like
it wasn't used anymore. This was simply removed
- context.get('lang')
The only information needed is the direction, rtl or ltr. This means
en and fr languages, despite sharing the same css assets, will duplicate
the ormcache entries.
-`_get_template_cache_keys`
Only the lang and webiste where really relevant in this flow. Other
keys are actually useless in this flow.
Some information used in the generation where not in the orm cache key
- `self.env.user.lang` if there is no lang in the context
- `request.session.get('force_website_id')`
- `request.httprequest.host`
- ...
The proposed solution is to:
- extract any informùation needed from thecontext, request, environment
before entering the ormcache, reduce it to the minimal possible set of
values needed
```
rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl'
assets_params = self.env['ir.asset']._get_assets_params() # website_id
debug_assets = debug and 'assets' in debug
```
and remove a leightweight part of the logic
```
def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None):
links = self._get_asset_links(bundle, css=css, js=js, debug=debug)
return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media)
```
Where _get_asset_links is the cached part, and _links_to_nodes is the
lightweight part generating the nodes based on the `defer_load`, ....
Additionnal notes:
- data-asset-version and data-asset-bundle are removed from the node
since they don't seem to be used anymore since 65d70acdbf
- async_load is removed since there is no occurence of this in the code.
- a small hack is still needed to pass javascript content instead of
links, this is only to manage css compile error and will hopefully be
removed in the future.
- a context key is still in use to generate the bundle, the
`commit_assetsbundle` but it has no impact on content and will hopefully
be removed in the future.
4. Add test for ormcache hit/miss
In this context, hit/miss is about having the same cache key for the
same result. This test demonstrates the current state, were entries are
create in the ormcache only if the key is really different and will lead
to a different result.
5. remove cache invalidation
This cache invalidation is quite agressive since everytime an
assetbundle is updated, all workers will clear their cache.
The concerned cache by this clear_cache is `_generate_asset_nodes_cache`
throug `_get_asset_nodes`.
The cache is ignored, both in dev=xml and debug=assets.
This clear cache was made conditionnal in 553ea82f81 but this does
not solve an issue we can have in production.
Lets imagine a clean solution
- all sources are updated
- all workers are restarted.
The orm caches are all empty, but since the sources
changed, all bundles will be recomputed. This means that every bundle
updated in database with save_attachement will invalidate the cache of
all workers. Rendering a pdf report of any kind using a specific bundle
will invalidate all cache. Starting a debug=assets for the first time
will invalidate all cache, even if the cache is not used in this case.
But for a regenerated bundle we would expect the ormcache to be:
- empty (did not generate the same bundle yet)
- have the same value (concurrent generation of the same bundle)
Having a different value would mean that the bundle was generated with
another version of the sources. In this case it is maybe even better not
to invalidate the cache since it could lead to an invalidation war
between two workers.
The only case where invalidating this cache is useful is when a bundle
changes, Usually if an ir_asset is created, modified, ...
There is still another rare but possible possibility to have a 404 if
the transaction is rollbacked after populating the assets node cache.
In this case, we only need to clear the cache locally in case of
rollback.
Part-of: odoo/odoo#121376
The C implementation of the JS minification gives speedups between 6
and 55 times faster than the regex-based Python port, depending on
how compressed the input it (which is what our default implementation
does).
This is measurable when generating compiled assets bundle from scratch,
e.g. after installing/updating modules or source code.
As an illustration, the minification of a 2MB JS bundle can be 50x
faster:
```py
import rjsmin
from odoo.addons.base.models.assetsbundle import rjsmin as rjsm
js_source = open("web.assets_common_lazy.js").read() # 2MB JS
%timeit rjsm(js_source)
# -> 339 ms ± 495 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit rjsmin.jsmin(js_source)
# -> 6.88 ms ± 213 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
It's also a drop-in replacement, as long as you rjsmin 1.1.0 or better
is available (to support format strings properly, a.o.).
See also the documentation of rjsmin: http://opensource.perlig.de/rjsmin/closesodoo/odoo#104283
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This error was intoduced in #121159
The javascript unique should also be based on templates.
closesodoo/odoo#121842
X-original-commit: 2a7c6640357b4d6fc0c539b5d8d5e6dd80882d5b
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
One of the most costly part of a page loading when the ormcache is cold
is computing the assets node, the unique identifier of an attachment to
validate whether the existing attachment is still valid with the current
version of the static files.
This operation needs to glob assets path in the filesystem,
get the modification date, check attachments, ...
Right now this task is not really optimized and can take some time
because of an excessive number of glob on the filesystem, unnecessary
exists to define absolute path, double computation of file list and
modified times when getting js and css bundle separately, ...
A list of modifications mainly discussed in the pr message are made
with this commit to speedup things.
- split css and js unique
- prepare api for an in memory glob
- change api to propagate absolute path and meta information through
`ir.asset._get_paths`-> _get_asset_paths -> `_get_asset_content` ->
`AssetsBundle`
closesodoo/odoo#121159
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The assets bundle system allows to generate `<script>` elements whose
resource is meant to be lazy loaded. In that case, a "data-src" is used
instead of "src" directly as attribute. This is valid HTML code: "src"
is not required. However... this lazy loading system also automatically
added `defer="defer"` which makes the "src" attribute required, thus
failing W3C validation.
Now, we only add "data-src" and the `defer="defer"` part is added on the
client-side, once the "data-src" is switched to "src". Note that this
"defer" attribute may not be needed in this case at all, but it cannot
hurt.
Part-of: odoo/odoo#120311
Before this commit an ir_assets generated automaticaly by the web editor
will generate an url ending with ...custom.addon.bundle_name.ext
After this commit the url will start with /_custom/addon.bundle_name/...
This will make it easier to spot at immediately if it is a custom asset
and thus it is useless to apply the glob. Actually, it will fail on the
/_custom when trying to glob, making it faster.
This is mainly useful to clarify and debug but in a case with mainly
customised ir_asset for one bundle, it may have an impact on speed.
An upgrade script was created for this change.
closesodoo/odoo#120699
Related: odoo/upgrade#4636
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This commit makes the `dependencies` param of
`odoo.define` mandatory. It was optional and when
omitted, a regexp read the function to find the
dependencies. We can simplify it now almost all js
modules have been converted to esm.
The transpiler already adds the param for the es
modules except if the module has an alias.
task id: 3271352
closesodoo/odoo#119145
Related: odoo/enterprise#40040
Signed-off-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Since saas 16.1, we added a more visible error information box when the
module system cannot find some dependencies, or if there is an error in
the JS code. This error box is very useful to understand a lot of
typical devlopment issues. However, it was occasionally displayed in
production code.
However, we sometimes observe that the error information box was
displayed, because the request to load an assets failed with a 404.
It is unclear in which circumstance this can happen, but it seems that
the following elements are involved:
- the user has an open tab with odoo, kept open for a while
- some js code is changed on the server, which causes the previous
assets to be deleted
- the user reload its open tabs, so the browser will load the /web page
from disk, which points to the old assets files
- then it will try to load all assets, with a 404 on one of the script
- the error information box is displayed
This commit intercepts the loading error and stop displaying the error
information box, which is basically the same behaviour as 16.0 and
before. We could force a reload in that case, but it seems dangerous,
since in case of errors, it could easily lead to an infinite loop.
closesodoo/odoo#117625
X-original-commit: 0a44b5b1e008b591a716efd26217c7db7360c37b
Signed-off-by: Géry Debongnie <ged@odoo.com>
is_transpiled is only needed when generating a asset bundle while
JavascriptAsset can be generated to compute the version hash.
is_transpiled needs to read the content to be defined which is quite
slow. Transforming is_transpiled into a lazy property will speedup the
cold loading of generate_assets_node, especially when attachment already
exists.
Locally:
- /web with all modules in debug=assets goes from ~350 to ~150 ms
- generate_assets_nodes part goes from ~230 to ~55 ms
closesodoo/odoo#116123
X-original-commit: 47b44e1ae61583a881bb2ede06cc02f4b9423cbc
Signed-off-by: Simon Genin (ges@odoo) <ges@odoo.com>
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The main goal of this commit is to reduce the size of the registry by
removing the (almost) useless __last_update field.
Statistics # of fields with all modules installed:
before 30184 fields, 1299x last_update (4.30%)
Before this commit, the computed field __last_update was added on every model.
The idea behind this field was to have a computed field that had either
the write_date or the create_date if the write_date was empty. However,
the write_date is always written, even on creation, making it useless
to have the computed field __last_update
After this update, we completely remove from BaseModel:
* __last_update
* CONCURRENCY_CHECK_FIELD that was always defined as "__last_update"
* _compute_concurrency_field that was the compute function for __last_update
closesodoo/odoo#105739
Task-id: 3062140 (part of 3062137 improve registry load time)
Related: odoo/upgrade#4038
Related: odoo/enterprise#33939
Signed-off-by: Raphael Collet <rco@odoo.com>
In c989ff339d, odoo module errors now
block the interface with an error message. Currenltly, in order to
support lazy-loading scenarios, the way that scss compilation errors are
displayed to the user is by attempting to import a module that may or
may not be present, and doing nothing when it's not present.
This commit fixes that by instead, writing the compilation error message
to a global variable, logging it in the console, and creating a service
that will display that error on start.
closesodoo/odoo#104671
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
in b1d57adf6ff3 the size of the header added to source files while
concatenating them was reduced, but the corresponding offset use when
generating source maps was not adapted, causing source maps to drift
further and further from the actual source location the further in the
compiled asset the line is.
This commit fixes that by decrementing the header offset as needed.
closesodoo/odoo#102651
X-original-commit: 4d4585ee6c905c2a3adc4692098c1a3544417629
Signed-off-by: Géry Debongnie <ged@odoo.com>
Signed-off-by: Samuel Degueldre <sad@odoo.com>
The bundle name is irrelevant in the bundle content and will prevent
attachment to store the same file if the bundles are exactly the same.
closesodoo/odoo#102502
X-original-commit: b1d57adf6ff358aa79f41817f4f7bfe60f9e5ac0
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The current asset_bundle version uses the last modified date.
This can be problematic in some case.
Even if it is a long time issue, the problem was rediscovered on runbot
with a commit being in the future. Runbot will export all file and set
the write date of the file to the commit date. The main purpose is to
have deterministic bundle version between different builds. This also
allows to generate assets bundle once at install for all post install
subbuild.
The issue here is that the commit date was greater than now(), meaning
that some tests setting custom css in attachment won't trigger the
regeneration of assets bundle. This wasn't really noticeable before
assets pregeneration.
This is not the first time strange issues occurs because of the
last modified logic.
This commit combined all last_modified to generate the bundle
version.
The current adaptation is quick and dirty and this will be reworked in
another post 16.0 freeze assets refactoring and cleanup.
closesodoo/odoo#100160
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
XML files are now declared in python module manifests. During the qweb
't-call-asset' directive, assetbundle will fetch the declared xml files,
apply the inheritance (t-inherit) and create a javascript service (for
eg: 'web.assets_backend.bundle.xml') which is added at the end of the
*.js mimifier file.
When the debug mode is activated, comments are added in the template
indicating which file the template comes from as well as the
inheritances applied to it.
****
JavaScript:
assets.js (module @web/core/assets) takes care of loading libraries,
javascripts and styles.
`loadJS(url)` (loads the javascript and returns a resolved promise when
the templates are also loaded via the '*.bundle.xml' service)
`loadCSS(url)` (loads the style a resolved promise when the file is
loaded)
`loadXML(xml, app=assets.defaultApp)` (load template into
application/owl, used by the `*.bundle.xml` services)
`getBundle(bundleName)` (get the bundle descriptor)
`loadBundle(desc)` (load the files and bundle from a descriptor)
templates (XML element content all owl templates)
A new `ready(serviceName)` method on boot.js lets you know when a
service is loaded are the require.
The xmlDependencies attribute no longer exists.
Python:
The xmls taken into account by assetbundle.py, applying `t-inherit`
inheritances and adding an `name_of_the_bundle.bundle.xml` service in
the generated JavaScript file.
****
Every manifest changes is into the next commit, except 'web_tour' in
this current commit as example.
Part-of: odoo/odoo#95500
* = base,http_routing,hw_drivers
- Update Bootstrap from 4.3.1 to 5.1.3
- Update PopperJS to version 2 for Bootstrap 5 (JS part)
Some code was for PopperJS V1, but it's not compatible anymore.
- Remove some BS5 classes utilities backport
- Fix path for BS5
Task ID: 2766483
Part-of: odoo/odoo#95450
- Fixes the RegEx that insert our internal structure path
for some assets like image font...
BS5 uses inline SVG and the function `escape-svg` to escape some
characters in HTML. But we can't add the path as the image is inline.
- Don't include Bootstrap in LESS assets;
Since Bootstrap 4 (and also in 5) there are no more LESS files for
Bootstrap. So, there is no reason to include the path when we compile
LESS assets.
Task ID: 2766483
Part-of: odoo/odoo#95450
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
Assets bundle are wrongly regenerated on the following circumstance.
The use of multiples tag `t-call-assets` in some xml template(s) with:
- at least one `t-call-assets` with `t-js` or `t-css` to false
- at least one `t-call-assets` without `t-js` nor `t-css`
In mass_mailing, the buggy regeneration of assets happens at a critical moment
and break the url of a previously generated asset. That url was however saved
in a javascript cache and therefore the bundle is inaccessible. No css style is
therefore applied to the page.
Note 1: the bug for the previous specific case in mass_mailing only
appears if within all the files defined for the bundle
`web_editor.wysiwyg_assets`, the last modified file (mtime) was of
type javascript. No visible bug would appear if the last modified file
was of type stylesheet (css, scss, ...). However, the asset would be
regenerated at a time where the cache should be used (invisible for the user).
Note 2: in order to reproduce the bug systematically without this commit,
it is necessary to disable the browser cache. Otherwise, if the
browser has the broken url in cache, no http request will be made and the bug
will not appears.
The bundles are wrongly regenerated because of how the url is
generated. Part of the url is generated using the "version" of the
bundle. The version is defined using a checksum. The checksum uses
information about the *last modified* file of the bundle. And this is
where bug arises.
If the bundle was called with `t-js="false"` the list of file didn't
include js files.
- The version for the `css` bundle will be A
If the bundle was called with `t-css="false"` the list of file didn't
include css files.
- The version for the `js` bundle will be B
If the bundle was called without `t-js` nor `t-css` the list includes
all js and css files.
- The version for the `js` bundle **and** the `css` bundle will be A
if the last modified file type is javascript or B if it is a stylesheet.
This commit change the behavior to always consider all the files last
modification within bundles in order to derive the version.
Thus:
If the bundle was called with `t-js="false"` the list of file didn't
include js files.
- The version for the `css` bundle will be A
If the bundle was called with `t-css="false"` the list of file didn't
include css files.
- The version for the `js` bundle will be A
If the bundle was called without `t-js` nor `t-css` the list of file
includes.
- The version for the `js` bundle **and** the `css` bundle will be A.
Example:
assetsA has two files:
```
file1.css
file2.js
```
file1.css was last modified at time A
file2.js was last modified at time B
bundle A
```xml
<template name="template_a">
<t t-call-assets="assetsA" t-js="false"/>
</template>
<template name="template_b">
<t t-call-assets="assetsA" t-css="false"/>
</template>
<template name="template_c">
<t t-call-assets="assetsA"/>
</template>
```
In a javascript file:
```js
const ajax = require('web.ajax');
await ajax.loadAsset({assetLibs: ['template_a']});
await ajax.loadAsset({assetLibs: ['template_b']});
await ajax.loadAsset({assetLibs: ['template_c']});
await ajax.loadAsset({assetLibs: ['template_a']});
```
Whenever loading a assets of a template, the urls for the css and javascript
files are saved into a cache until the next browser reload.
In this example, without this commit:
- the first `ajax.loadAsset` `template_a` generate a css asset with version X
-> a css file is generated and the url of the css file is
`/web/assets/version_X...min.css` and is saved in the javascript
cache for `template_a`
- the `ajax.loadAsset` `template_b` generate a js asset with version Y
-> a js file is generated and the url of the js file is
`/web/assets/version_Y...js`
- the `ajax.loadAsset` `template_c` generate a css **and** js asset
with version Y if time A (a css file) is bigger than time B (a js
file):
-> a css file is generated and the url of the css file is
`/web/assets/version_X...min.css` and is saved in the
javascript cache for `template_c`
-> a js file is generated and the url of the js file is
`/web/assets/version_X...js`
if time B (a js file) is bigger than time a (a css file)
-> a css file is generated and the url of the css file
is`/web/assets/version_Y...min.css` and is saved in the
javascript cache for `template_c`
-> a js file is generated and the url of the css file is
`/web/assets/version_Y...js`
--> because the css file is re-generated (the version changed from
X to Y), the url in the javascript cache is no longer valid.
- the last `ajax.loadAsset` `template_a` use the url in its cache and
either the css url or the js url would no longer will be available
and a http error will trigger if the code tries to access it.
Task-2778413
closesodoo/odoo#86155
X-original-commit: 2267cfc947eff9fc4f00468610a82c8fbbca36ad
Signed-off-by: David Monjoie (dmo) <dmo@odoo.com>
When the assets are regenerated, the previous attachments are deleted.
This may happen during the execution of t-call-assets directives in a
qweb view.
However, some properties that have been accessed with a sudo() were
accessible in the cache. Clearing the cache during the view rendering
could lead to access errors, which wouldn't be present without this
directive.
This commit works around this problem by removing these attachments with
a SQL query, without relying on the classic unlink, so that the cache is
preserved in this case.
By the way, sanitize the filename when marking it for deletion.
closesodoo/odoo#83570
X-original-commit: 627d508edfb62449067a05b0e3c1a0004f0af32c
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Paul Morelle <pmo@odoo.com>
Before this commit, the server version used to detect whether the client
assets are outdated relied on the `odoo.release.version` string directly
imported at the root of the `assetsbundle.py` file.
The problem is that this string is altered by web_enterprise and we must
wait for these changes before using the string.
This commit dynamically imports the `release.version` string to use its
final version.
closesodoo/odoo#82705
X-original-commit: 9eccbd884853fddab8a17d28779d670e7d48bbc3
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Julien Mougenot (jum) <jum@odoo.com>
Before this commit: a notification asking to reload the current window
appeared as soon as the server detected a change in one of the assets
bundles, even on first load.
To fix this problem and make the feature more meaningful, it has been
decided to only notify the client when the server version (not the
bundle version) is outdated (i.e. on database upgrades, when the changes
in the code are actually relevant).
closesodoo/odoo#82032
X-original-commit: a3b5a9d715be6a93c7f2859074b916f3249f97c3
Signed-off-by: Antony Lesuisse <al@odoo.com>
Signed-off-by: Julien Mougenot (jum) <jum@odoo.com>
>>> u1 = self.sudo(False).browse(1)
>>> u2 = self.sudo().browse(2)
>>> (u1 + u2).env.su
False
>>> (u2 + u1).env.su
True
Ensure all attachments are always in sudo
Before this commit, a portal user could not go in debug asset
Introduced at 3a98996eed
closesodoo/odoo#80807
X-original-commit: 9f0ce611b2acf5927a7703a7811081c4adbc3f41
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
When detecting if the assets where already preprocessed, the method
only the last asset type matching was returned and deleted.
In standard, it should not have any impact as all assets are of the
same type but it may be possible to mix it with external modules.
Sign CLA
closesodoo/odoo#79786
X-original-commit: 0c2f83f4a6c4603d107007ac77c94e6a2805cd81
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
* = auth_signup, calendar, im_livechat, snailmail_account, survey, test_mail,
web_editor, website_crm_iap_reveal, website_livechat
The aim of this PR is to improve/fix various flaws and limitation of the current
API, to make it easier to use and more efficient.
Notification are now defined with 3 distinct parts:
- the channel determines which client(s) should receive it
- the type determines how it should be handled
- the payload determines any extra information helpful for handling it
Channel
=======
Business code
-------------
- Record channel is introduced for ease of subscribing to and sending
notifications to specific partners, channels, documents, ...
- String channel is still supported (but it is converted internally to the tuple
channel).
- Tuple channel is still supported without any change (but should be avoided
whenever possible due to its complex syntax).
The channel is no longer sent to the client. When the channel was used for
business purpose, the information it contained has been moved into either the
new type, or the payload itself.
Technical note
--------------
All channels are now internally converted to the tuple (db, ...) channel, which
is necessary for the platform code (saas/sh).
Internally, the bus.bus table is not changed, type and payload are grouped
together into what was (and still is) called message.
Type
====
Type is introduced to uniformize the way notifications are sent and handled.
All existing notifications already had some kind of manually-built type in them.
This is now officially supported at the bus API.
In client code this will allow (to be done in future commits) to register one
handler per specific type, instead of having to iterate and to filter all
received notifications on every handler.
Payload
=======
Payload (ex message) did not change, it can still be anything depending on
business needs.
Few adaptations:
- When the type was included on the payload, the type has been moved to the new
type parameter.
- When the channel was used in business code, its data has been copied into the
payload.
task-1891151
closesodoo/odoo#79201
X-original-commit: 543af27c7d6836ffac9e80ff8490b6ddbd849221
Related: odoo/enterprise#21998
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Before this commit: the "Asset changed" log message would display
an incorrect version and name. This commit fixes that.
closesodoo/odoo#78175
X-original-commit: d4350ddace643fefa5951655e53f701b13b45333
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
This commit proposes a mechanism to manually add a frame before a
blocking c-call and triggers it before calling libsaas.compile.
closesodoo/odoo#75519
X-original-commit: f836ff3a67d446cd8b3ab2cf573fab649153e6da
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
* Remove AST in favor of pure Pyhon. This should make it easier for
developers to understand and create new directives because they do not
need to know AST.
* Remove `t-call-options` as it has been merged into `t-options` for more
consistency. Support for t-call-options is retained.
* Use generators for lists. This increases performances as the rendering
can be sent directly without having to wait for the creation of the
entire list.
* Optimize expressions runtime computation by pre-computing the static
parts.
Example:
'<' + 'div' + '>' + '<' + dynamic_value + '>'
Now compiles as:
'<div><' + dynamic_value + '>'
Add a big fat warning when the qweb compiler finds a `t-raw`.
`t-esc` should now be used everywhere, the use-case for `t-raw` should
be handled by converting the corresponding values to `Markup`
objects. Even though it's convenient, this constructor *should never
be made available in the qweb rendering context* (maybe that should be
checked for explicitely?).
Replace `werkzeug.escape` by `markupsafe.escape` in
`odoo.tools.html_escape`, this means the output of `html_escape` is
markup-safe.
Updated qweb to work correctly with escaping and `Markup`, amongst
other things QWeb bodies should be markup-safe internally (so that a
`t-set` value can be fed into a `t-esc`). See at the bottom for the
attributes handling as it's a bit complicated.
`to_text` needed updating: `markupsafe.Markup` is a subclass of `str`,
but `str` is not a passthrough for strings. So `Markup` instances
going through would be converted to normal `str`, losing their safety
flag. Since qweb internally uses `to_text` on pretty much
everything (in order to handle None / False), this would then cause
almost every `Markup` to get mistakenly double-escaped.
Also mark a bunch of APIs as markup-safe by default
* html_sanitize output.
* HTML fields content, sanitization is applied on intake (so stripped
by the trip through the database) and if the field is unsanitised
the injection is very much intentional, probably. Note: this
includes automatically decoding bytes as a number of default values
& computes yield bytes, which Markup will happily accept... by
repr-ing them which is useless. This is hard to notice without `-b`.
* Script-safe json, it's rather the point (though it uses a
non-standard escaping scheme).
* Note that `nl2br`, kinda: it should work correctly whether or not
the input is markup-safe, this means we should not need to escape
values fed to `nl2br`, but it doesn't hurt either.
Update some qweb field serialisations to mark their output as
markup-safe when necessary (e.g. monetary, barcode,
contact). Otherwise either using proper escaping internally or doing
nothing should do the trick.
Also update qweb to return markup-safe bytes: we want qweb to return
markup-safe contents as a common use-case is to render something with
one template, and inject its content in an other one (with Python code
inbetween, as `t-call` works a bit differently and does not go through
the external rendering interface).
However qweb returns `bytes` while `Markup` extends `str`. After a
quick experiment with changing qweb rendering to return `str` (rather
unmitigated failure I fear), it looks like the safest tack is to add a
somewhat similar bytes-based type, which decodes to a `Markup` but
keeps to bytes semantics.
For debugging and convenience reasons, MarkupSafeBytes does *not*
stringify and raises an error instead (`__repr__` works fine). This is
to avoid implicit stringifications which do the wrong thing (namely
create a string `"b'foo'"`).
Also add some configuration around BytesWarning (which still has to be
enabled at the interpreter level via `-b`, there's no way to enable it
programmatically smh), and monkeypatch `showwarning` to show warning
tracebacks, as it's common for warnings to be triggered in the bowels
of the application, and hard to relate to business logic without the
complete traceback.
`t-out`
=======
`t-esc` is a bit confusing for the new behaviour of "maybe escape
maybe not", so add a `t-out` alias with the same behaviour.
Unlike `t-raw`, `t-esc` is only soft-deprecated for now: there are
thousands of instances, so editing all the templates is not
great. Eventually we'll add a `ci/style` to prevent addition of new
ones, and eventually we might do a bulk-replace and hard-deprecate.
Attributes handling
===================
There are a few issues with respect to attributes. The first issue is
that markup-safe content is not necessarily attributes-safe
e.g. markup-safe content can contain unescaped `<` or double-quotes
while attributes can not. So we must forcefully escape the input, even
if it's supposedly markup-safe already.
This causes a problem for script-safe JSON: it's markup-safe but
really does its own thing. So instead of escaping it up-front and
wrapping it in Markup, make script-safe JSON its own type which
applies JSON-escaping *during the `__html__` call.
This way if a script-safe JSON object goes through `markupsafe.escape`
we'll apply script-safe escaping, otherwise it'll be treated as a
regular strings and eventually escaped the normal way.
A second issue was the processing of format-valued
attributes (`t-attf`): literal segments should always be markup-safe,
while non-literal may or may not be. This turns out to be an issue if
the non-literal segment *is* markup-safe: in that case when the
literal and non-literal segments get concatenated the literal segments
will get escaped, then attributes serialization will escape
them *again* leading to doubly-escaped content in attributes.
The most visible instance of this was the `snippet_options` template,
specifically:
<t t-set="so_content_addition_selector" t-translation="off">blockquote, ...</t>
<div id="so_content_addition"
t-att-data-selector="so_content_addition_selector"
t-attf-data-drop-near="p, h1, h2, h3, .row > div > img, #{so_content_addition_selector}"
data-drop-in=".content, nav"/>
Here `so_content_addition_selector` is a qweb body therefore
markup-safe, When concatenated with the literal part of
`t-atff-data-drop-near` it would cause the HTML-escaping of that
yielding a new Markup object. Normal attributes processing would then
strip the markup flag (using `str()`) and escape it again, leading to
doubly-escaped literals.
The original hack around was to unescape() `Markup` content before
stringifying it and escaping it again, in the attribute serialization
method (`_append_attributes`).
That's pretty disgusting, after some more consideration & testing it
looks like a much better and safer fix is to ensure the
expression (non-literal) segments of format strings always result in
`str`, never `Markup`, which is easy enough: just all `str()` on the
output of strexpr. We could also have concatenated all the bits using
`''.join` instead of repeated concatenation (`+`).
Also add a check on the type of the format string for safety, I think
it should always be a proper str and the bytes thing is only when
running in py2 (where lxml uses bytestrings as a space optimization
for ascii-only values) but it should not hurt too much to perform a
single typecheck assertion on the value... instead of performing one
per literal segment.
Note: we may need to implement unescape anyway, because it's still
possible to get double-escaping with the current scheme: given an
explicitly escape-ed `foo` and `t-att-foo="foo"`, `foo` will be
re-escaped.
fixup! [CHG] core, web: deprecate t-raw
This commit changes the way assets are declared in Odoo modules.
Before: assets were declared in template files. Template bundles were
generated from primary templates, so technically any qweb template could
have been called as an asset bundle, with the 't-call-assets' directive.
Being standard qweb templates, they had access to standard HTML tags
(script, link, with or without raw scripts or style definition), qweb
directives (t-call, t-raw, etc.) and could be inherited by other
templates.
Now: assets are defined in the module's manifest and generated by the
't-call-assets' directive.
More information on the new system can be found on the updated user
documentation (see the "JavaScript Reference" section).
Task: 2352566
Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
Improve the development experience in debug=assets mode by reducing the
number of requests to the server. We are adapting the solution used for
the JS files to the CSS files. This solution consists of no longer
sending all the files separately, but sending only the bundles
associated with their sourcemap. This allows us to keep the same
debugging experience while drastically reducing the number of requests
to the server.
Benchmark:
saas 14.2 917 requests domcontentloaded after 3.76s
master (bundling du js) 299 requests domcontentloaded after 2.03s
branch (bundling js+css) 36 requests domcontentloaded after 1.01s
Task id : 2463840
closesodoo/odoo#66169
Related: odoo/design-themes#453
Signed-off-by: Géry Debongnie (ged) <ged@openerp.com>
Because of the way Odoo works at its core, we do not know before hand
which files will be loaded as an asset in the browser, because it
depends on the installed Odoo addons. This is why it is historically
difficult to integrate Odoo with standard JS tooling, and this is why
Odoo needs to use a custom javascript module system.
However, there is a way to use native JS modules (and gain all the
benefits from it: IDE autocompletion, ease of refactoring, intellisense,
...): we can write JS as native JS modules, but convert them at runtime
into Odoo custom modules. This is exactly the strategy applied by this
PR.
This has a lot of benefits, but there is a downside: we can no longer
serve statically JS files in debug=assets. This would be a dealbreaker,
if we did not have sourcemaps (implemented in the next commit).
This commit introduces the python code that will transpile native JS
modules into odoo JS modules.
Task ID: 2414902
PR: 63177
Co-authored-by: Francois (fge) <fge@odoo.com>
Use case:
When the server is restarted, the python is updated,
but some users may have an ongoing session in a browser tab
This may lead to code being unsynchronized and ultimately to some
odd bugs.
Purpose:
When we are in such a case, that is, the assets were recomputed
after a update of the code and a restart of the server by the request of another user,
notify connected users that assets have changed.
Then propose them to reload the page.
Known caveats:
- This is not a developer's feature.
Since assets computing is ORM cached, they have limited
opportunities to rebuild. Namely, the feature won't trigger
each time the JS has changed, rather, it will
when JS has changed AND the cache has been reset somehow (e.g. when the server is restarted).
- This not a portal/website feature either, but only in backend.
Business clients won't be notified that the JS has changed.
- While requests debug=assets do trigger a recomputing
of the *components* of bundles, they do not save a bundle
This means that the requests that sends the notification
cannot be debug=assets.
Task 2034462
closesodoo/odoo#39875
Signed-off-by: Mathieu Duckerts-Antoine <Polymorphe57@users.noreply.github.com>
* add a `raw` computed field, though only update some of base to use
it (addons for which that makes sense can be migrated progressively)
* avoid working with base64 data when it's possible to work with the
actual data
* improve datas (base64 encoded content): should depend on bin_size
closesodoo/odoo#47212
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Xavier-Do