The C implementation of the JS minification gives speedups between 6
and 55 times faster than the regex-based Python port, depending on
how compressed the input it (which is what our default implementation
does).
This is measurable when generating compiled assets bundle from scratch,
e.g. after installing/updating modules or source code.
As an illustration, the minification of a 2MB JS bundle can be 50x
faster:
```py
import rjsmin
from odoo.addons.base.models.assetsbundle import rjsmin as rjsm
js_source = open("web.assets_common_lazy.js").read() # 2MB JS
%timeit rjsm(js_source)
# -> 339 ms ± 495 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit rjsmin.jsmin(js_source)
# -> 6.88 ms ± 213 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
It's also a drop-in replacement, as long as you rjsmin 1.1.0 or better
is available (to support format strings properly, a.o.).
See also the documentation of rjsmin: http://opensource.perlig.de/rjsmin/closesodoo/odoo#104283
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This error was intoduced in #121159
The javascript unique should also be based on templates.
closesodoo/odoo#121842
X-original-commit: 2a7c6640357b4d6fc0c539b5d8d5e6dd80882d5b
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
One of the most costly part of a page loading when the ormcache is cold
is computing the assets node, the unique identifier of an attachment to
validate whether the existing attachment is still valid with the current
version of the static files.
This operation needs to glob assets path in the filesystem,
get the modification date, check attachments, ...
Right now this task is not really optimized and can take some time
because of an excessive number of glob on the filesystem, unnecessary
exists to define absolute path, double computation of file list and
modified times when getting js and css bundle separately, ...
A list of modifications mainly discussed in the pr message are made
with this commit to speedup things.
- split css and js unique
- prepare api for an in memory glob
- change api to propagate absolute path and meta information through
`ir.asset._get_paths`-> _get_asset_paths -> `_get_asset_content` ->
`AssetsBundle`
closesodoo/odoo#121159
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The assets bundle system allows to generate `<script>` elements whose
resource is meant to be lazy loaded. In that case, a "data-src" is used
instead of "src" directly as attribute. This is valid HTML code: "src"
is not required. However... this lazy loading system also automatically
added `defer="defer"` which makes the "src" attribute required, thus
failing W3C validation.
Now, we only add "data-src" and the `defer="defer"` part is added on the
client-side, once the "data-src" is switched to "src". Note that this
"defer" attribute may not be needed in this case at all, but it cannot
hurt.
Part-of: odoo/odoo#120311
Before this commit an ir_assets generated automaticaly by the web editor
will generate an url ending with ...custom.addon.bundle_name.ext
After this commit the url will start with /_custom/addon.bundle_name/...
This will make it easier to spot at immediately if it is a custom asset
and thus it is useless to apply the glob. Actually, it will fail on the
/_custom when trying to glob, making it faster.
This is mainly useful to clarify and debug but in a case with mainly
customised ir_asset for one bundle, it may have an impact on speed.
An upgrade script was created for this change.
closesodoo/odoo#120699
Related: odoo/upgrade#4636
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This commit makes the `dependencies` param of
`odoo.define` mandatory. It was optional and when
omitted, a regexp read the function to find the
dependencies. We can simplify it now almost all js
modules have been converted to esm.
The transpiler already adds the param for the es
modules except if the module has an alias.
task id: 3271352
closesodoo/odoo#119145
Related: odoo/enterprise#40040
Signed-off-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Since saas 16.1, we added a more visible error information box when the
module system cannot find some dependencies, or if there is an error in
the JS code. This error box is very useful to understand a lot of
typical devlopment issues. However, it was occasionally displayed in
production code.
However, we sometimes observe that the error information box was
displayed, because the request to load an assets failed with a 404.
It is unclear in which circumstance this can happen, but it seems that
the following elements are involved:
- the user has an open tab with odoo, kept open for a while
- some js code is changed on the server, which causes the previous
assets to be deleted
- the user reload its open tabs, so the browser will load the /web page
from disk, which points to the old assets files
- then it will try to load all assets, with a 404 on one of the script
- the error information box is displayed
This commit intercepts the loading error and stop displaying the error
information box, which is basically the same behaviour as 16.0 and
before. We could force a reload in that case, but it seems dangerous,
since in case of errors, it could easily lead to an infinite loop.
closesodoo/odoo#117625
X-original-commit: 0a44b5b1e008b591a716efd26217c7db7360c37b
Signed-off-by: Géry Debongnie <ged@odoo.com>
is_transpiled is only needed when generating a asset bundle while
JavascriptAsset can be generated to compute the version hash.
is_transpiled needs to read the content to be defined which is quite
slow. Transforming is_transpiled into a lazy property will speedup the
cold loading of generate_assets_node, especially when attachment already
exists.
Locally:
- /web with all modules in debug=assets goes from ~350 to ~150 ms
- generate_assets_nodes part goes from ~230 to ~55 ms
closesodoo/odoo#116123
X-original-commit: 47b44e1ae61583a881bb2ede06cc02f4b9423cbc
Signed-off-by: Simon Genin (ges@odoo) <ges@odoo.com>
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The main goal of this commit is to reduce the size of the registry by
removing the (almost) useless __last_update field.
Statistics # of fields with all modules installed:
before 30184 fields, 1299x last_update (4.30%)
Before this commit, the computed field __last_update was added on every model.
The idea behind this field was to have a computed field that had either
the write_date or the create_date if the write_date was empty. However,
the write_date is always written, even on creation, making it useless
to have the computed field __last_update
After this update, we completely remove from BaseModel:
* __last_update
* CONCURRENCY_CHECK_FIELD that was always defined as "__last_update"
* _compute_concurrency_field that was the compute function for __last_update
closesodoo/odoo#105739
Task-id: 3062140 (part of 3062137 improve registry load time)
Related: odoo/upgrade#4038
Related: odoo/enterprise#33939
Signed-off-by: Raphael Collet <rco@odoo.com>
In c989ff339d, odoo module errors now
block the interface with an error message. Currenltly, in order to
support lazy-loading scenarios, the way that scss compilation errors are
displayed to the user is by attempting to import a module that may or
may not be present, and doing nothing when it's not present.
This commit fixes that by instead, writing the compilation error message
to a global variable, logging it in the console, and creating a service
that will display that error on start.
closesodoo/odoo#104671
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
in b1d57adf6ff3 the size of the header added to source files while
concatenating them was reduced, but the corresponding offset use when
generating source maps was not adapted, causing source maps to drift
further and further from the actual source location the further in the
compiled asset the line is.
This commit fixes that by decrementing the header offset as needed.
closesodoo/odoo#102651
X-original-commit: 4d4585ee6c905c2a3adc4692098c1a3544417629
Signed-off-by: Géry Debongnie <ged@odoo.com>
Signed-off-by: Samuel Degueldre <sad@odoo.com>
The bundle name is irrelevant in the bundle content and will prevent
attachment to store the same file if the bundles are exactly the same.
closesodoo/odoo#102502
X-original-commit: b1d57adf6ff358aa79f41817f4f7bfe60f9e5ac0
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The current asset_bundle version uses the last modified date.
This can be problematic in some case.
Even if it is a long time issue, the problem was rediscovered on runbot
with a commit being in the future. Runbot will export all file and set
the write date of the file to the commit date. The main purpose is to
have deterministic bundle version between different builds. This also
allows to generate assets bundle once at install for all post install
subbuild.
The issue here is that the commit date was greater than now(), meaning
that some tests setting custom css in attachment won't trigger the
regeneration of assets bundle. This wasn't really noticeable before
assets pregeneration.
This is not the first time strange issues occurs because of the
last modified logic.
This commit combined all last_modified to generate the bundle
version.
The current adaptation is quick and dirty and this will be reworked in
another post 16.0 freeze assets refactoring and cleanup.
closesodoo/odoo#100160
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
XML files are now declared in python module manifests. During the qweb
't-call-asset' directive, assetbundle will fetch the declared xml files,
apply the inheritance (t-inherit) and create a javascript service (for
eg: 'web.assets_backend.bundle.xml') which is added at the end of the
*.js mimifier file.
When the debug mode is activated, comments are added in the template
indicating which file the template comes from as well as the
inheritances applied to it.
****
JavaScript:
assets.js (module @web/core/assets) takes care of loading libraries,
javascripts and styles.
`loadJS(url)` (loads the javascript and returns a resolved promise when
the templates are also loaded via the '*.bundle.xml' service)
`loadCSS(url)` (loads the style a resolved promise when the file is
loaded)
`loadXML(xml, app=assets.defaultApp)` (load template into
application/owl, used by the `*.bundle.xml` services)
`getBundle(bundleName)` (get the bundle descriptor)
`loadBundle(desc)` (load the files and bundle from a descriptor)
templates (XML element content all owl templates)
A new `ready(serviceName)` method on boot.js lets you know when a
service is loaded are the require.
The xmlDependencies attribute no longer exists.
Python:
The xmls taken into account by assetbundle.py, applying `t-inherit`
inheritances and adding an `name_of_the_bundle.bundle.xml` service in
the generated JavaScript file.
****
Every manifest changes is into the next commit, except 'web_tour' in
this current commit as example.
Part-of: odoo/odoo#95500
* = base,http_routing,hw_drivers
- Update Bootstrap from 4.3.1 to 5.1.3
- Update PopperJS to version 2 for Bootstrap 5 (JS part)
Some code was for PopperJS V1, but it's not compatible anymore.
- Remove some BS5 classes utilities backport
- Fix path for BS5
Task ID: 2766483
Part-of: odoo/odoo#95450
- Fixes the RegEx that insert our internal structure path
for some assets like image font...
BS5 uses inline SVG and the function `escape-svg` to escape some
characters in HTML. But we can't add the path as the image is inline.
- Don't include Bootstrap in LESS assets;
Since Bootstrap 4 (and also in 5) there are no more LESS files for
Bootstrap. So, there is no reason to include the path when we compile
LESS assets.
Task ID: 2766483
Part-of: odoo/odoo#95450
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
Assets bundle are wrongly regenerated on the following circumstance.
The use of multiples tag `t-call-assets` in some xml template(s) with:
- at least one `t-call-assets` with `t-js` or `t-css` to false
- at least one `t-call-assets` without `t-js` nor `t-css`
In mass_mailing, the buggy regeneration of assets happens at a critical moment
and break the url of a previously generated asset. That url was however saved
in a javascript cache and therefore the bundle is inaccessible. No css style is
therefore applied to the page.
Note 1: the bug for the previous specific case in mass_mailing only
appears if within all the files defined for the bundle
`web_editor.wysiwyg_assets`, the last modified file (mtime) was of
type javascript. No visible bug would appear if the last modified file
was of type stylesheet (css, scss, ...). However, the asset would be
regenerated at a time where the cache should be used (invisible for the user).
Note 2: in order to reproduce the bug systematically without this commit,
it is necessary to disable the browser cache. Otherwise, if the
browser has the broken url in cache, no http request will be made and the bug
will not appears.
The bundles are wrongly regenerated because of how the url is
generated. Part of the url is generated using the "version" of the
bundle. The version is defined using a checksum. The checksum uses
information about the *last modified* file of the bundle. And this is
where bug arises.
If the bundle was called with `t-js="false"` the list of file didn't
include js files.
- The version for the `css` bundle will be A
If the bundle was called with `t-css="false"` the list of file didn't
include css files.
- The version for the `js` bundle will be B
If the bundle was called without `t-js` nor `t-css` the list includes
all js and css files.
- The version for the `js` bundle **and** the `css` bundle will be A
if the last modified file type is javascript or B if it is a stylesheet.
This commit change the behavior to always consider all the files last
modification within bundles in order to derive the version.
Thus:
If the bundle was called with `t-js="false"` the list of file didn't
include js files.
- The version for the `css` bundle will be A
If the bundle was called with `t-css="false"` the list of file didn't
include css files.
- The version for the `js` bundle will be A
If the bundle was called without `t-js` nor `t-css` the list of file
includes.
- The version for the `js` bundle **and** the `css` bundle will be A.
Example:
assetsA has two files:
```
file1.css
file2.js
```
file1.css was last modified at time A
file2.js was last modified at time B
bundle A
```xml
<template name="template_a">
<t t-call-assets="assetsA" t-js="false"/>
</template>
<template name="template_b">
<t t-call-assets="assetsA" t-css="false"/>
</template>
<template name="template_c">
<t t-call-assets="assetsA"/>
</template>
```
In a javascript file:
```js
const ajax = require('web.ajax');
await ajax.loadAsset({assetLibs: ['template_a']});
await ajax.loadAsset({assetLibs: ['template_b']});
await ajax.loadAsset({assetLibs: ['template_c']});
await ajax.loadAsset({assetLibs: ['template_a']});
```
Whenever loading a assets of a template, the urls for the css and javascript
files are saved into a cache until the next browser reload.
In this example, without this commit:
- the first `ajax.loadAsset` `template_a` generate a css asset with version X
-> a css file is generated and the url of the css file is
`/web/assets/version_X...min.css` and is saved in the javascript
cache for `template_a`
- the `ajax.loadAsset` `template_b` generate a js asset with version Y
-> a js file is generated and the url of the js file is
`/web/assets/version_Y...js`
- the `ajax.loadAsset` `template_c` generate a css **and** js asset
with version Y if time A (a css file) is bigger than time B (a js
file):
-> a css file is generated and the url of the css file is
`/web/assets/version_X...min.css` and is saved in the
javascript cache for `template_c`
-> a js file is generated and the url of the js file is
`/web/assets/version_X...js`
if time B (a js file) is bigger than time a (a css file)
-> a css file is generated and the url of the css file
is`/web/assets/version_Y...min.css` and is saved in the
javascript cache for `template_c`
-> a js file is generated and the url of the css file is
`/web/assets/version_Y...js`
--> because the css file is re-generated (the version changed from
X to Y), the url in the javascript cache is no longer valid.
- the last `ajax.loadAsset` `template_a` use the url in its cache and
either the css url or the js url would no longer will be available
and a http error will trigger if the code tries to access it.
Task-2778413
closesodoo/odoo#86155
X-original-commit: 2267cfc947eff9fc4f00468610a82c8fbbca36ad
Signed-off-by: David Monjoie (dmo) <dmo@odoo.com>
When the assets are regenerated, the previous attachments are deleted.
This may happen during the execution of t-call-assets directives in a
qweb view.
However, some properties that have been accessed with a sudo() were
accessible in the cache. Clearing the cache during the view rendering
could lead to access errors, which wouldn't be present without this
directive.
This commit works around this problem by removing these attachments with
a SQL query, without relying on the classic unlink, so that the cache is
preserved in this case.
By the way, sanitize the filename when marking it for deletion.
closesodoo/odoo#83570
X-original-commit: 627d508edfb62449067a05b0e3c1a0004f0af32c
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Paul Morelle <pmo@odoo.com>
Before this commit, the server version used to detect whether the client
assets are outdated relied on the `odoo.release.version` string directly
imported at the root of the `assetsbundle.py` file.
The problem is that this string is altered by web_enterprise and we must
wait for these changes before using the string.
This commit dynamically imports the `release.version` string to use its
final version.
closesodoo/odoo#82705
X-original-commit: 9eccbd884853fddab8a17d28779d670e7d48bbc3
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Julien Mougenot (jum) <jum@odoo.com>
Before this commit: a notification asking to reload the current window
appeared as soon as the server detected a change in one of the assets
bundles, even on first load.
To fix this problem and make the feature more meaningful, it has been
decided to only notify the client when the server version (not the
bundle version) is outdated (i.e. on database upgrades, when the changes
in the code are actually relevant).
closesodoo/odoo#82032
X-original-commit: a3b5a9d715be6a93c7f2859074b916f3249f97c3
Signed-off-by: Antony Lesuisse <al@odoo.com>
Signed-off-by: Julien Mougenot (jum) <jum@odoo.com>
>>> u1 = self.sudo(False).browse(1)
>>> u2 = self.sudo().browse(2)
>>> (u1 + u2).env.su
False
>>> (u2 + u1).env.su
True
Ensure all attachments are always in sudo
Before this commit, a portal user could not go in debug asset
Introduced at 3a98996eed
closesodoo/odoo#80807
X-original-commit: 9f0ce611b2acf5927a7703a7811081c4adbc3f41
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
When detecting if the assets where already preprocessed, the method
only the last asset type matching was returned and deleted.
In standard, it should not have any impact as all assets are of the
same type but it may be possible to mix it with external modules.
Sign CLA
closesodoo/odoo#79786
X-original-commit: 0c2f83f4a6c4603d107007ac77c94e6a2805cd81
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
* = auth_signup, calendar, im_livechat, snailmail_account, survey, test_mail,
web_editor, website_crm_iap_reveal, website_livechat
The aim of this PR is to improve/fix various flaws and limitation of the current
API, to make it easier to use and more efficient.
Notification are now defined with 3 distinct parts:
- the channel determines which client(s) should receive it
- the type determines how it should be handled
- the payload determines any extra information helpful for handling it
Channel
=======
Business code
-------------
- Record channel is introduced for ease of subscribing to and sending
notifications to specific partners, channels, documents, ...
- String channel is still supported (but it is converted internally to the tuple
channel).
- Tuple channel is still supported without any change (but should be avoided
whenever possible due to its complex syntax).
The channel is no longer sent to the client. When the channel was used for
business purpose, the information it contained has been moved into either the
new type, or the payload itself.
Technical note
--------------
All channels are now internally converted to the tuple (db, ...) channel, which
is necessary for the platform code (saas/sh).
Internally, the bus.bus table is not changed, type and payload are grouped
together into what was (and still is) called message.
Type
====
Type is introduced to uniformize the way notifications are sent and handled.
All existing notifications already had some kind of manually-built type in them.
This is now officially supported at the bus API.
In client code this will allow (to be done in future commits) to register one
handler per specific type, instead of having to iterate and to filter all
received notifications on every handler.
Payload
=======
Payload (ex message) did not change, it can still be anything depending on
business needs.
Few adaptations:
- When the type was included on the payload, the type has been moved to the new
type parameter.
- When the channel was used in business code, its data has been copied into the
payload.
task-1891151
closesodoo/odoo#79201
X-original-commit: 543af27c7d6836ffac9e80ff8490b6ddbd849221
Related: odoo/enterprise#21998
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Before this commit: the "Asset changed" log message would display
an incorrect version and name. This commit fixes that.
closesodoo/odoo#78175
X-original-commit: d4350ddace643fefa5951655e53f701b13b45333
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
This commit proposes a mechanism to manually add a frame before a
blocking c-call and triggers it before calling libsaas.compile.
closesodoo/odoo#75519
X-original-commit: f836ff3a67d446cd8b3ab2cf573fab649153e6da
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
* Remove AST in favor of pure Pyhon. This should make it easier for
developers to understand and create new directives because they do not
need to know AST.
* Remove `t-call-options` as it has been merged into `t-options` for more
consistency. Support for t-call-options is retained.
* Use generators for lists. This increases performances as the rendering
can be sent directly without having to wait for the creation of the
entire list.
* Optimize expressions runtime computation by pre-computing the static
parts.
Example:
'<' + 'div' + '>' + '<' + dynamic_value + '>'
Now compiles as:
'<div><' + dynamic_value + '>'
Add a big fat warning when the qweb compiler finds a `t-raw`.
`t-esc` should now be used everywhere, the use-case for `t-raw` should
be handled by converting the corresponding values to `Markup`
objects. Even though it's convenient, this constructor *should never
be made available in the qweb rendering context* (maybe that should be
checked for explicitely?).
Replace `werkzeug.escape` by `markupsafe.escape` in
`odoo.tools.html_escape`, this means the output of `html_escape` is
markup-safe.
Updated qweb to work correctly with escaping and `Markup`, amongst
other things QWeb bodies should be markup-safe internally (so that a
`t-set` value can be fed into a `t-esc`). See at the bottom for the
attributes handling as it's a bit complicated.
`to_text` needed updating: `markupsafe.Markup` is a subclass of `str`,
but `str` is not a passthrough for strings. So `Markup` instances
going through would be converted to normal `str`, losing their safety
flag. Since qweb internally uses `to_text` on pretty much
everything (in order to handle None / False), this would then cause
almost every `Markup` to get mistakenly double-escaped.
Also mark a bunch of APIs as markup-safe by default
* html_sanitize output.
* HTML fields content, sanitization is applied on intake (so stripped
by the trip through the database) and if the field is unsanitised
the injection is very much intentional, probably. Note: this
includes automatically decoding bytes as a number of default values
& computes yield bytes, which Markup will happily accept... by
repr-ing them which is useless. This is hard to notice without `-b`.
* Script-safe json, it's rather the point (though it uses a
non-standard escaping scheme).
* Note that `nl2br`, kinda: it should work correctly whether or not
the input is markup-safe, this means we should not need to escape
values fed to `nl2br`, but it doesn't hurt either.
Update some qweb field serialisations to mark their output as
markup-safe when necessary (e.g. monetary, barcode,
contact). Otherwise either using proper escaping internally or doing
nothing should do the trick.
Also update qweb to return markup-safe bytes: we want qweb to return
markup-safe contents as a common use-case is to render something with
one template, and inject its content in an other one (with Python code
inbetween, as `t-call` works a bit differently and does not go through
the external rendering interface).
However qweb returns `bytes` while `Markup` extends `str`. After a
quick experiment with changing qweb rendering to return `str` (rather
unmitigated failure I fear), it looks like the safest tack is to add a
somewhat similar bytes-based type, which decodes to a `Markup` but
keeps to bytes semantics.
For debugging and convenience reasons, MarkupSafeBytes does *not*
stringify and raises an error instead (`__repr__` works fine). This is
to avoid implicit stringifications which do the wrong thing (namely
create a string `"b'foo'"`).
Also add some configuration around BytesWarning (which still has to be
enabled at the interpreter level via `-b`, there's no way to enable it
programmatically smh), and monkeypatch `showwarning` to show warning
tracebacks, as it's common for warnings to be triggered in the bowels
of the application, and hard to relate to business logic without the
complete traceback.
`t-out`
=======
`t-esc` is a bit confusing for the new behaviour of "maybe escape
maybe not", so add a `t-out` alias with the same behaviour.
Unlike `t-raw`, `t-esc` is only soft-deprecated for now: there are
thousands of instances, so editing all the templates is not
great. Eventually we'll add a `ci/style` to prevent addition of new
ones, and eventually we might do a bulk-replace and hard-deprecate.
Attributes handling
===================
There are a few issues with respect to attributes. The first issue is
that markup-safe content is not necessarily attributes-safe
e.g. markup-safe content can contain unescaped `<` or double-quotes
while attributes can not. So we must forcefully escape the input, even
if it's supposedly markup-safe already.
This causes a problem for script-safe JSON: it's markup-safe but
really does its own thing. So instead of escaping it up-front and
wrapping it in Markup, make script-safe JSON its own type which
applies JSON-escaping *during the `__html__` call.
This way if a script-safe JSON object goes through `markupsafe.escape`
we'll apply script-safe escaping, otherwise it'll be treated as a
regular strings and eventually escaped the normal way.
A second issue was the processing of format-valued
attributes (`t-attf`): literal segments should always be markup-safe,
while non-literal may or may not be. This turns out to be an issue if
the non-literal segment *is* markup-safe: in that case when the
literal and non-literal segments get concatenated the literal segments
will get escaped, then attributes serialization will escape
them *again* leading to doubly-escaped content in attributes.
The most visible instance of this was the `snippet_options` template,
specifically:
<t t-set="so_content_addition_selector" t-translation="off">blockquote, ...</t>
<div id="so_content_addition"
t-att-data-selector="so_content_addition_selector"
t-attf-data-drop-near="p, h1, h2, h3, .row > div > img, #{so_content_addition_selector}"
data-drop-in=".content, nav"/>
Here `so_content_addition_selector` is a qweb body therefore
markup-safe, When concatenated with the literal part of
`t-atff-data-drop-near` it would cause the HTML-escaping of that
yielding a new Markup object. Normal attributes processing would then
strip the markup flag (using `str()`) and escape it again, leading to
doubly-escaped literals.
The original hack around was to unescape() `Markup` content before
stringifying it and escaping it again, in the attribute serialization
method (`_append_attributes`).
That's pretty disgusting, after some more consideration & testing it
looks like a much better and safer fix is to ensure the
expression (non-literal) segments of format strings always result in
`str`, never `Markup`, which is easy enough: just all `str()` on the
output of strexpr. We could also have concatenated all the bits using
`''.join` instead of repeated concatenation (`+`).
Also add a check on the type of the format string for safety, I think
it should always be a proper str and the bytes thing is only when
running in py2 (where lxml uses bytestrings as a space optimization
for ascii-only values) but it should not hurt too much to perform a
single typecheck assertion on the value... instead of performing one
per literal segment.
Note: we may need to implement unescape anyway, because it's still
possible to get double-escaping with the current scheme: given an
explicitly escape-ed `foo` and `t-att-foo="foo"`, `foo` will be
re-escaped.
fixup! [CHG] core, web: deprecate t-raw
This commit changes the way assets are declared in Odoo modules.
Before: assets were declared in template files. Template bundles were
generated from primary templates, so technically any qweb template could
have been called as an asset bundle, with the 't-call-assets' directive.
Being standard qweb templates, they had access to standard HTML tags
(script, link, with or without raw scripts or style definition), qweb
directives (t-call, t-raw, etc.) and could be inherited by other
templates.
Now: assets are defined in the module's manifest and generated by the
't-call-assets' directive.
More information on the new system can be found on the updated user
documentation (see the "JavaScript Reference" section).
Task: 2352566
Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
Improve the development experience in debug=assets mode by reducing the
number of requests to the server. We are adapting the solution used for
the JS files to the CSS files. This solution consists of no longer
sending all the files separately, but sending only the bundles
associated with their sourcemap. This allows us to keep the same
debugging experience while drastically reducing the number of requests
to the server.
Benchmark:
saas 14.2 917 requests domcontentloaded after 3.76s
master (bundling du js) 299 requests domcontentloaded after 2.03s
branch (bundling js+css) 36 requests domcontentloaded after 1.01s
Task id : 2463840
closesodoo/odoo#66169
Related: odoo/design-themes#453
Signed-off-by: Géry Debongnie (ged) <ged@openerp.com>
Because of the way Odoo works at its core, we do not know before hand
which files will be loaded as an asset in the browser, because it
depends on the installed Odoo addons. This is why it is historically
difficult to integrate Odoo with standard JS tooling, and this is why
Odoo needs to use a custom javascript module system.
However, there is a way to use native JS modules (and gain all the
benefits from it: IDE autocompletion, ease of refactoring, intellisense,
...): we can write JS as native JS modules, but convert them at runtime
into Odoo custom modules. This is exactly the strategy applied by this
PR.
This has a lot of benefits, but there is a downside: we can no longer
serve statically JS files in debug=assets. This would be a dealbreaker,
if we did not have sourcemaps (implemented in the next commit).
This commit introduces the python code that will transpile native JS
modules into odoo JS modules.
Task ID: 2414902
PR: 63177
Co-authored-by: Francois (fge) <fge@odoo.com>
Use case:
When the server is restarted, the python is updated,
but some users may have an ongoing session in a browser tab
This may lead to code being unsynchronized and ultimately to some
odd bugs.
Purpose:
When we are in such a case, that is, the assets were recomputed
after a update of the code and a restart of the server by the request of another user,
notify connected users that assets have changed.
Then propose them to reload the page.
Known caveats:
- This is not a developer's feature.
Since assets computing is ORM cached, they have limited
opportunities to rebuild. Namely, the feature won't trigger
each time the JS has changed, rather, it will
when JS has changed AND the cache has been reset somehow (e.g. when the server is restarted).
- This not a portal/website feature either, but only in backend.
Business clients won't be notified that the JS has changed.
- While requests debug=assets do trigger a recomputing
of the *components* of bundles, they do not save a bundle
This means that the requests that sends the notification
cannot be debug=assets.
Task 2034462
closesodoo/odoo#39875
Signed-off-by: Mathieu Duckerts-Antoine <Polymorphe57@users.noreply.github.com>
* add a `raw` computed field, though only update some of base to use
it (addons for which that makes sense can be migrated progressively)
* avoid working with base64 data when it's possible to work with the
actual data
* improve datas (base64 encoded content): should depend on bin_size
closesodoo/odoo#47212
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Since we now use native promises, the code in charge of alerting the
user of bundle compilation errors was opening both an alert and a
dialog (fwp of https://github.com/odoo/odoo/pull/46437).
In 13.0, the mention fwp is not enough. On the website, the JS is also
lazy loaded now so we need to wait for that lazy loading before being
able to open a modal.
The whole alerting code should be improved to not be defined in python.
Fixes https://github.com/odoo/odoo/issues/45447closesodoo/odoo#46482
X-original-commit: d5a2469b1edd0d22b366b44086d08c6f0b2247b2
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Fine-tunning of 42eec88e5da164e4930c48113109f2c8d4b73da3
Before this commit, the assets were not found when the user is connected
in mode debug assets in a multi-website and multi-company. This occurs
because the attachments (assets) were created only for the first website
accessed, and cannot be loaded when trying to access the second website.
Now, the attachments are created for each website accessed and can be
loaded withtout problems.
closesodoo/odoo#42593
X-original-commit: 95077a8cdd9f2cd99c2ef85bdf7ca5d58e006b71
Signed-off-by: Jorge Pinna Puissant (jpp) <jpp@odoo.com>
Assets bundle creation will try to find previous version of assets and unlink them.
This needs to invalidate ormcache to notify other workers.
Since assets are never commited during test, assets will be generate at each request if request is
not commited (cursor closed) whtch is the case with 404.
Invalidating the orm cache only when a previous attachement was existing will help to
mitigate the problem, and slightly improve HTTPCases test speed.
closesodoo/odoo#38979
X-original-commit: d4d63549e1b31e3f3c97a2b91155023dbac62847
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Before this commit, the assets were not found when the user is connected
in mode debug assets in a multi-website and multi-company. This occurs
because the attachments (assets) were created with the number of the
first website accessed, and cannot be loaded when trying to access the
second website.
The website_id is added in the creation of the views see:
092870270e
Now, the attachments are created without the website number and is
shared between all the websites.
opw-2076768
closesodoo/odoo#37789
X-original-commit: 42eec88e5da164e4930c48113109f2c8d4b73da3
Signed-off-by: Jorge Pinna Puissant (jpp) <jpp@odoo.com>
SHA-1 is a cryptographic hash function that have weaknesses known since
2005, it has been deprecated by the NIST [1] about 10 years ago in 2011
and Google [2] have been able to perform a collision attack in 2017.
We use SHA-1 in order to generate unique URL for resources that can be
cached by the browser: assets bundle, translations, qweb templates and
qweb images.
Although practical attacks still requires quite a lot of computational
resources, it is time to upgrade SHA-1 to SHA-2.
We have selected the SHA-512/256 variant of the SHA-2 algorithm as
replacement for SHA-1 for the following reasons:
* On 64 bits platform, SHA-512 is the fastest SHA-2 variant, it is only
~1.5x slower than SHA-1. [3]
* Keeping only the 256 foremost bits protects against both collision
attacks and length extension attacks.
* The hexadecimal digest is only 24 chars longer than SHA-1 which is
nice to have somewhat short URLs.
We have not used SHA-3 because:
* At the moment of writing, it is too slow (~3x slower than SHA-1) [3]
* It is not guaranteed to be available with the Python 3.5 `hashlib`
module.
* One of the author of SHA-3 is Belgian.
[1] https://csrc.nist.gov/projects/hash-functions/nist-policy-on-hash-functions
[2] https://shattered.io/
[3] http://bench.cr.yp.to/results-hash.html
[4] http://www.commitstrip.com/en/2017/02/27/the-sha-1-alternative/
This version adds full support of string templates.
Before this version, rjsmin may produce wrong code with string
templates. For example:
```js
const url = `https://odoo.com`;
console.log(url);
```
was minified to:
```js
const url=`https:console.log(url);
```
Because `//` was wrongly handled as a comment inside the string
template.
closesodoo/odoo#35867
Signed-off-by: Géry Debongnie (ged) <ged@openerp.com>
get_installed and _lang_get_id are both ormcached and correctly check
the context
Retrieving a res.lang from a code is a frequent action that can be
achieved with _lang_get (cf previous commit).
Using _lang_get ensure the active_test in the context is correct and
is not poluted with another context propagation issue.
odoo/odoo#35490 discussion is an example of bad context propagation
closesodoo/odoo#35504
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
This commit removes the field `datas_fname` from `ir.attachment` as
it was unnecessary and most of the time the duplicate of `name` or
`url`.
Task #1909865closesodoo/odoo#32976
Signed-off-by: Martin Geubelle (mge) <mge@openerp.com>
Olivier Dony