From 6d5d234f15e458c4a92c240fc743aee941b460e2 Mon Sep 17 00:00:00 2001 From: Xavier-Do Date: Thu, 20 Apr 2023 09:22:32 +0000 Subject: [PATCH] [IMP] base: better ormcache management MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. move cache to _get_asset_paths The `_get_asset_content` cache has many cache key that are related to a posprocessing of the `_get_asset_paths` result, the heavy part of this method. Moving the cache to _get_asset_content will have the benefit to create less duplicates entries in the ormcache as well as less cache miss. To simplify even further, the css and js parameters are removed since they only filter the output of get_paths, the heavy part of globing the file will be done before that. Anyway, they are both true when called from _get_asset_content, and the only other call, in `_get_related_bundle` don't really need to filter them since it is not a critical part regarding performance, and the funtional result will stay the same. The initial orm cache key was using `_get_template_cache_keys`, a little overkill and possibly creating duplicates entries again. The only context key needed is website_id for `_get_related_assets`. Note that it is not really enough, the orm cache key should actually contain `request.session.get('force_website_id')` as well has `request.httprequest.host`. This will be addressed latter since a nicer solution would be to have website_id as a unique parameter computed earlier. 2. better _get_asset_paths cache key The orm cache key was simplified in previous point but there is still one concern, the website_id depends on more parameters than that: - request.session.get('force_website_id') - request.httprequest.host - existing websites The idea here is to call `get_current_website` instead of using all parameters that could define the webiste. In the same spirit of `_get_template_cache_keys` `_assets_path_params` can be overriden to give extra params that are usefull to list assets path. Those params are computed before entering the method `_get_asset_paths`. This may latter put at a higher level latter, in get_asset_node, to simplify the _generate_asset_nodes_cache key. 3. better assets_node caches key The main purpose of this part is to improve ormcache containing assets nodes. The ormcache key contains - to much context key - missing session/host/env info - unwanted boolean options. - keys leading to the same cache value The main goal being to reduce the size of the cache keys, decrease the number of cache entries and improve the cache hit. This will also make the behaviour more coherent and hopefully less bug prone because of mismatch in parameters. The main reason of the orm cache is the slowness of the validation of the assets. This includes: - listing files (dedicated orm cache) - computing version The cache key was depending on - `debug` The only relevant value for debug is "contains assets" We dont need to differ between debug='', debug='1', debug='test', and 'debug=assets', 'debug=tests,assets', ... - `defer_load`, `lazy_load`, `media` Those values are only useful to generate html node, a leightweight operations that does not really needs to be in cache. `media` was also used in the generation but it looks useless if we have the media on the node. THIS NEEDS TO BE VALIDATED but in any case, since media is not used to generate the url, it doesn't make sence to use it in the generation. The main idea to remove them from the ormcache key is simply to generate the nodes outide the ormcached values. -`async_load` This one is similar to `defer_load` and `lazy_load` but it looks like it wasn't used anymore. This was simply removed - context.get('lang') The only information needed is the direction, rtl or ltr. This means en and fr languages, despite sharing the same css assets, will duplicate the ormcache entries. -`_get_template_cache_keys` Only the lang and webiste where really relevant in this flow. Other keys are actually useless in this flow. Some information used in the generation where not in the orm cache key - `self.env.user.lang` if there is no lang in the context - `request.session.get('force_website_id')` - `request.httprequest.host` - ... The proposed solution is to: - extract any informùation needed from thecontext, request, environment before entering the ormcache, reduce it to the minimal possible set of values needed ``` rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl' assets_params = self.env['ir.asset']._get_assets_params() # website_id debug_assets = debug and 'assets' in debug ``` and remove a leightweight part of the logic ``` def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None): links = self._get_asset_links(bundle, css=css, js=js, debug=debug) return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media) ``` Where _get_asset_links is the cached part, and _links_to_nodes is the lightweight part generating the nodes based on the `defer_load`, .... Additionnal notes: - data-asset-version and data-asset-bundle are removed from the node since they don't seem to be used anymore since 65d70acdbf - async_load is removed since there is no occurence of this in the code. - a small hack is still needed to pass javascript content instead of links, this is only to manage css compile error and will hopefully be removed in the future. - a context key is still in use to generate the bundle, the `commit_assetsbundle` but it has no impact on content and will hopefully be removed in the future. 4. Add test for ormcache hit/miss In this context, hit/miss is about having the same cache key for the same result. This test demonstrates the current state, were entries are create in the ormcache only if the key is really different and will lead to a different result. 5. remove cache invalidation This cache invalidation is quite agressive since everytime an assetbundle is updated, all workers will clear their cache. The concerned cache by this clear_cache is `_generate_asset_nodes_cache` throug `_get_asset_nodes`. The cache is ignored, both in dev=xml and debug=assets. This clear cache was made conditionnal in 553ea82f81354 but this does not solve an issue we can have in production. Lets imagine a clean solution - all sources are updated - all workers are restarted. The orm caches are all empty, but since the sources changed, all bundles will be recomputed. This means that every bundle updated in database with save_attachement will invalidate the cache of all workers. Rendering a pdf report of any kind using a specific bundle will invalidate all cache. Starting a debug=assets for the first time will invalidate all cache, even if the cache is not used in this case. But for a regenerated bundle we would expect the ormcache to be: - empty (did not generate the same bundle yet) - have the same value (concurrent generation of the same bundle) Having a different value would mean that the bundle was generated with another version of the sources. In this case it is maybe even better not to invalidate the cache since it could lead to an invalidation war between two workers. The only case where invalidating this cache is useful is when a bundle changes, Usually if an ir_asset is created, modified, ... There is still another rare but possible possibility to have a 404 if the transaction is rollbacked after populating the assets node cache. In this case, we only need to clear the cache locally in case of rollback. Part-of: odoo/odoo#121376 --- addons/bus/controllers/websocket.py | 5 +- addons/test_website/tests/test_qweb.py | 4 +- addons/web/tests/test_assets.py | 2 +- addons/web/tests/test_assets_xml.py | 8 +- addons/web/tests/test_js.py | 2 +- addons/web_editor/controllers/main.py | 11 +- addons/website/models/ir_asset.py | 32 +- addons/website/models/ir_qweb.py | 19 +- addons/website/tests/test_ir_asset.py | 4 +- addons/website/tests/test_performance.py | 2 +- addons/website/tests/test_ui.py | 17 +- odoo/addons/base/models/assetsbundle.py | 175 +++------- odoo/addons/base/models/ir_asset.py | 74 ++--- odoo/addons/base/models/ir_qweb.py | 161 +++++---- .../tests/test_assetsbundle.py | 311 ++++++++++++------ odoo/tests/common.py | 1 + 16 files changed, 445 insertions(+), 383 deletions(-) diff --git a/addons/bus/controllers/websocket.py b/addons/bus/controllers/websocket.py index 8c828094fd7..e97871ff120 100644 --- a/addons/bus/controllers/websocket.py +++ b/addons/bus/controllers/websocket.py @@ -64,8 +64,9 @@ class WebsocketController(Controller): prevent new worker versions to be loaded from the browser cache. """ bundle = 'bus.websocket_worker_assets' - files, _ = request.env["ir.qweb"]._get_asset_content(bundle) - asset = AssetsBundle(bundle, files) + assets_params = request.env["ir.asset"]._get_assets_params() + files, _ = request.env["ir.qweb"]._get_asset_content(bundle, assets_params=assets_params) + asset = AssetsBundle(bundle, files, assets_params=assets_params) stream = request.env['ir.binary']._get_stream_from(asset.js( is_minified="assets" not in request.session.debug )) diff --git a/addons/test_website/tests/test_qweb.py b/addons/test_website/tests/test_qweb.py index 6f165f50d6f..c66615a6b02 100644 --- a/addons/test_website/tests/test_qweb.py +++ b/addons/test_website/tests/test_qweb.py @@ -65,11 +65,11 @@ class TestQweb(TransactionCaseWithUserDemo): - + - + diff --git a/addons/web/tests/test_assets.py b/addons/web/tests/test_assets.py index b4261b2b917..dd2c4349148 100644 --- a/addons/web/tests/test_assets.py +++ b/addons/web/tests/test_assets.py @@ -34,7 +34,7 @@ class TestAssetsGenerateTimeCommon(odoo.tests.TransactionCase): start_t = time.time() css = assets_type == 'css' js = assets_type == 'js' - self.env['ir.qweb']._generate_asset_nodes(bundle, css=css, js=js) + self.env['ir.qweb']._generate_asset_links(bundle, css=css, js=js) yield (f'{bundle}.{assets_type}', time.time() - start_t) except ValueError: _logger.info('Error detected while generating bundle %r %s', bundle, assets_type) diff --git a/addons/web/tests/test_assets_xml.py b/addons/web/tests/test_assets_xml.py index be1f995d583..fe46a970a7a 100644 --- a/addons/web/tests/test_assets_xml.py +++ b/addons/web/tests/test_assets_xml.py @@ -63,19 +63,13 @@ class TestStaticInheritanceCommon(odoo.tests.TransactionCase): def renderBundle(self, debug=False): files = [] for url in self.template_files: - atype = 'text/xml' - if '.js' in url: - atype = 'text/javascript' files.append({ - 'atype': atype, 'url': url, 'filename': url, 'content': None, 'media': None, }) - asset = AssetsBundle('web.test_bundle', files, env=self.env, css=False, js=True, debug='assets' if debug else '') - # to_node return the files descriptions and generate attachments. - asset.to_node(css=False, js=False) + asset = AssetsBundle('web.test_bundle', files, env=self.env, css=False, js=True, debug_assets=debug) content = asset.xml(show_inherit_info=debug) return f'\n{content}\n' diff --git a/addons/web/tests/test_js.py b/addons/web/tests/test_js.py index c30cc17203b..4cef8ec460c 100644 --- a/addons/web/tests/test_js.py +++ b/addons/web/tests/test_js.py @@ -45,7 +45,7 @@ class WebSuite(odoo.tests.HttpCase): for asset in assets: filename = asset['filename'] - if not filename or asset['atype'] != 'text/javascript': + if not filename or not filename.endswith('.js'): continue with open(filename, 'rb') as fp: if RE_ONLY.search(fp.read().decode('utf-8')): diff --git a/addons/web_editor/controllers/main.py b/addons/web_editor/controllers/main.py index 967648991d8..c0a98a41d10 100644 --- a/addons/web_editor/controllers/main.py +++ b/addons/web_editor/controllers/main.py @@ -448,9 +448,9 @@ class Web_Editor(http.Controller): AssetsUtils = request.env['web_editor.assets'] files_data_by_bundle = [] - resources_type_info = {'t_call_assets_attribute': 't-js', 'mimetype': 'text/javascript'} + t_call_assets_attribute = 't-js' if file_type == 'scss': - resources_type_info = {'t_call_assets_attribute': 't-css', 'mimetype': 'text/scss'} + t_call_assets_attribute = 't-css' # Compile regex outside of the loop # This will used to exclude library scss files from the result @@ -460,7 +460,7 @@ class Web_Editor(http.Controller): url_infos = dict() for v in views: for asset_call_node in etree.fromstring(v["arch"]).xpath("//t[@t-call-assets]"): - attr = asset_call_node.get(resources_type_info['t_call_assets_attribute']) + attr = asset_call_node.get(t_call_assets_attribute) if attr and not json.loads(attr.lower()): continue asset_name = asset_call_node.get("t-call-assets") @@ -468,7 +468,7 @@ class Web_Editor(http.Controller): # Loop through bundle files to search for file info files_data = [] for file_info in request.env["ir.qweb"]._get_asset_content(asset_name)[0]: - if file_info["atype"] != resources_type_info['mimetype']: + if file_info["url"].rpartition('.')[2] != file_type: continue url = file_info["url"] @@ -608,7 +608,8 @@ class Web_Editor(http.Controller): if not bundle_css: bundle = 'web.assets_frontend' files, _ = request.env["ir.qweb"]._get_asset_content(bundle) - asset = AssetsBundle(bundle, files) + assets_params = request.env["ir.asset"]._get_assets_params() + asset = AssetsBundle(bundle, files, assets_params=assets_params) bundle_css = asset.css().index_content color_search = re.search(r'(?i)--%s:\s+(%s|%s)' % (css_color_value, regex_hex, regex_rgba), bundle_css) if not color_search: diff --git a/addons/website/models/ir_asset.py b/addons/website/models/ir_asset.py index f0d6adfada4..889d7ce9a47 100644 --- a/addons/website/models/ir_asset.py +++ b/addons/website/models/ir_asset.py @@ -10,35 +10,41 @@ class IrAsset(models.Model): key = fields.Char(copy=False) # used to resolve multiple assets in a multi-website environment website_id = fields.Many2one('website', ondelete='cascade') - def _get_related_assets(self, domain): - website = self.env['website'].get_current_website(fallback=False) - if website: - domain += website.website_domain() - assets = super()._get_related_assets(domain) - return assets.filter_duplicate() + def _get_assets_params(self): + params = super()._get_assets_params() + params['website_id'] = self.env['website'].get_current_website(fallback=False).id + return params - def _get_active_addons_list(self): + def _get_related_assets(self, domain, website_id=None, **params): + if website_id: + domain += self.env['website'].website_domain(website_id) + assets = super()._get_related_assets(domain, **params) + return assets.filter_duplicate(website_id) + + def _get_active_addons_list(self, website_id=None, **params): """Overridden to discard inactive themes.""" - addons_list = super()._get_active_addons_list() - website = self.env['website'].get_current_website(fallback=False) + addons_list = super()._get_active_addons_list(**params) - if not website: + if not website_id: return addons_list IrModule = self.env['ir.module.module'].sudo() # discard all theme modules except website.theme_id - themes = IrModule.search(IrModule.get_themes_domain()) - website.theme_id + themes = IrModule.search(IrModule.get_themes_domain()) - self.env["website"].browse(website_id).theme_id to_remove = set(themes.mapped('name')) return [name for name in addons_list if name not in to_remove] - def filter_duplicate(self): + def filter_duplicate(self, website_id=None): """ Filter current recordset only keeping the most suitable asset per distinct name. Every non-accessible asset will be removed from the set: * In non website context, every asset with a website will be removed * In a website context, every asset from another website """ - current_website = self.env['website'].get_current_website(fallback=False) + if website_id is not None: + current_website = self.env['website'].browse(website_id) + else: + current_website = self.env['website'].get_current_website(fallback=False) if not current_website: return self.filtered(lambda asset: not asset.website_id) diff --git a/addons/website/models/ir_qweb.py b/addons/website/models/ir_qweb.py index 476c3f8e3c3..87e04678a11 100644 --- a/addons/website/models/ir_qweb.py +++ b/addons/website/models/ir_qweb.py @@ -20,14 +20,14 @@ re_background_image = re.compile(r"(background-image\s*:\s*url\(\s*['\"]?\s*)([^ class AssetsBundleMultiWebsite(AssetsBundle): def _get_asset_url_values(self, id, unique, extra, name, sep, extension): - website_id = self.env.context.get('website_id') + website_id = self.assets_params.get('website_id') website_id_path = website_id and ('%s/' % website_id) or '' extra = website_id_path + extra res = super(AssetsBundleMultiWebsite, self)._get_asset_url_values(id, unique, extra, name, sep, extension) return res def get_debug_asset_url(self, extra='', name='%', extension='%'): - website_id = self.env.context.get('website_id') + website_id = self.assets_params.get('website_id') website_id_path = website_id and ('%s/' % website_id) or '' extra = website_id_path + extra return super(AssetsBundleMultiWebsite, self).get_debug_asset_url(extra, name, extension) @@ -106,8 +106,8 @@ class IrQWeb(models.AbstractModel): return irQweb - def _get_asset_bundle(self, xmlid, files, env=None, css=True, js=True, debug=False): - return AssetsBundleMultiWebsite(xmlid, files, env=env, css=css, js=js, debug=debug) + def _get_asset_bundle(self, xmlid, files=None, env=None, css=True, js=True, debug_assets=False, rtl=False, assets_params=None): + return AssetsBundleMultiWebsite(xmlid, files, env=env, css=css, js=js, debug_assets=debug_assets, rtl=rtl, assets_params=assets_params) def _post_processing_att(self, tagName, atts): if atts.get('data-no-post-process'): @@ -161,14 +161,13 @@ class IrQWeb(models.AbstractModel): # version part combine with the initial extra (rtl) should be enough to ensure they are identical. # we dont expect to have any pregenerated rtl/website attachment so we don't manage assets with extra - nodes = super()._pregenerate_assets_bundles() + links = super()._pregenerate_assets_bundles() website = self.env['website'].search([], order='id', limit=1) if not website: - return nodes + return links nb_created = 0 - for node in nodes: - bundle_info = node[1] - bundle_url = bundle_info.get('src', '') or bundle_info.get('href', '') + for link in links: + bundle_url = link[0] if bundle_url.startswith('/web/assets/'): # example: "/web/assets/2152-ee56665/web.assets_frontend_lazy.min.js" _, _, _, id_unique, name = bundle_url.split('/') @@ -189,4 +188,4 @@ class IrQWeb(models.AbstractModel): if nb_created: _logger.runbot('%s bundle(s) were copied for website %s', nb_created, website.id) - return nodes + return links diff --git a/addons/website/tests/test_ir_asset.py b/addons/website/tests/test_ir_asset.py index 320ea6c5d8e..c7adb5f4b3b 100644 --- a/addons/website/tests/test_ir_asset.py +++ b/addons/website/tests/test_ir_asset.py @@ -39,12 +39,12 @@ class TestIrAsset(odoo.tests.HttpCase): 'active': False, }) - files = IrAsset.with_context(website_id=website_1.id)._get_asset_paths('test_bundle.irasset', css=True) + files = IrAsset._get_asset_paths('test_bundle.irasset', {'website_id': website_1.id}) self.assertEqual(len(files), 2, "There should be two assets in the specific website.") self.assertEqual(files[0][0], '/website/test/base0.css', "First asset should be the same as the base one.") self.assertEqual(files[1][0], '/website/test/specific1.css', "Second asset should be the specific one.") - files = IrAsset.with_context(website_id=website_2.id)._get_asset_paths('test_bundle.irasset', css=True) + files = IrAsset._get_asset_paths('test_bundle.irasset', {'website_id': website_2.id}) self.assertEqual(len(files), 3, "All three assets should be in the unmodified website.") self.assertEqual(files[0][0], '/website/test/base0.css', "First asset should be the base one.") self.assertEqual(files[1][0], '/website/test/base1.css', "Second asset should be the base one.") diff --git a/addons/website/tests/test_performance.py b/addons/website/tests/test_performance.py index 0fd0c5f7be6..6cf3ca95147 100644 --- a/addons/website/tests/test_performance.py +++ b/addons/website/tests/test_performance.py @@ -146,7 +146,7 @@ class TestWebsitePerformancePost(UtilPerf): @mute_logger('odoo.http') def test_50_perf_sql_web_assets(self): # assets route /web/assets/.. - self.env['ir.qweb']._generate_asset_nodes('web.assets_frontend_lazy', css=False, js=True) + self.env['ir.qweb']._generate_asset_links('web.assets_frontend_lazy', css=False, js=True) assets_url = self.env['ir.attachment'].search([('url', '=like', '/web/assets/%/web.assets_frontend_lazy%.js')], limit=1).url self.assertEqual(self._get_url_hot_query(assets_url), 4) self.assertEqual(self._get_url_hot_query(assets_url, cache=False), 4) diff --git a/addons/website/tests/test_ui.py b/addons/website/tests/test_ui.py index fd54f1c0171..0cce294033d 100644 --- a/addons/website/tests/test_ui.py +++ b/addons/website/tests/test_ui.py @@ -222,7 +222,9 @@ class TestUi(odoo.tests.HttpCase): self.start_tour("/", 'website_navbar_menu') def test_05_specific_website_editor(self): + asset_bundle_xmlid = 'website.assets_wysiwyg' website_default = self.env['website'].search([], limit=1) + new_website = self.env['website'].create({'name': 'New Website'}) code = b"document.body.dataset.hello = 'world';" @@ -231,7 +233,7 @@ class TestUi(odoo.tests.HttpCase): 'mimetype': 'text/javascript', 'datas': base64.b64encode(code), }) - custom_url = '/web/content/%s/%s' % (attach.id, attach.name) + custom_url = '/_custom/web/content/%s/%s' % (attach.id, attach.name) attach.url = custom_url self.env['ir.asset'].create({ @@ -240,6 +242,19 @@ class TestUi(odoo.tests.HttpCase): 'path': custom_url, 'website_id': new_website.id, }) + + files, _ = self.env['ir.qweb']._get_asset_content(asset_bundle_xmlid, assets_params={'website_id': website_default.id}) + self.assertNotIn(custom_url, [f['url'] for f in files]) + base_website_bundle = self.env['ir.qweb']._get_asset_bundle(asset_bundle_xmlid, files, env=self.env, assets_params={'website_id': website_default.id}) + base_website_css_version = base_website_bundle.get_version('css') + base_website_js_version = base_website_bundle.get_version('js') + + files, _ = self.env['ir.qweb']._get_asset_content('website.assets_wysiwyg', assets_params={'website_id': new_website.id}) + self.assertIn(custom_url, [f['url'] for f in files]) + new_website_bundle_modified = self.env['ir.qweb']._get_asset_bundle(asset_bundle_xmlid, files, env=self.env, assets_params={'website_id': new_website.id}) + self.assertEqual(new_website_bundle_modified.get_version('css'), base_website_css_version) + self.assertNotEqual(new_website_bundle_modified.get_version('js'), base_website_js_version, "js version for new website should now have been changed") + url_params = url_encode({'path': '/@/'}) self.start_tour(f'/website/force/{website_default.id}?{url_params}', "generic_website_editor", login='admin') self.start_tour(f'/website/force/{new_website.id}?{url_params}', "specific_website_editor", login='admin') diff --git a/odoo/addons/base/models/assetsbundle.py b/odoo/addons/base/models/assetsbundle.py index 6359bcaf09a..cebc1bd7637 100644 --- a/odoo/addons/base/models/assetsbundle.py +++ b/odoo/addons/base/models/assetsbundle.py @@ -57,7 +57,7 @@ class AssetsBundle(object): TRACKED_BUNDLES = ['web.assets_common', 'web.assets_backend'] - def __init__(self, name, files, env=None, css=True, js=True, debug=None): + def __init__(self, name, files, env=None, css=True, js=True, debug_assets=False, rtl=False, assets_params=None): """ :param name: bundle name :param files: files to be added to the bundle @@ -71,16 +71,16 @@ class AssetsBundle(object): self.stylesheets = [] self.css_errors = [] self.files = files - self.user_direction = self.env['res.lang']._lang_get( - self.env.context.get('lang') or self.env.user.lang - ).direction + self.rtl = rtl + self.assets_params = assets_params or {} self.has_css = css self.has_js = js self._checksum_cache = {} - self.is_debug_assets = debug and 'assets' in debug + self.is_debug_assets = debug_assets # asset-wide html "media" attribute for f in files: + extension = f['url'].rpartition('.')[2] params = { 'url': f['url'], 'filename': f['filename'], @@ -89,45 +89,37 @@ class AssetsBundle(object): } if css: css_params = { - 'media': f['media'], - 'direction': self.user_direction, + 'rtl': self.rtl, } - if f['atype'] == 'text/sass': + if extension == 'sass': self.stylesheets.append(SassStylesheetAsset(self, **params, **css_params)) - elif f['atype'] == 'text/scss': + elif extension == 'scss': self.stylesheets.append(ScssStylesheetAsset(self, **params, **css_params)) - elif f['atype'] == 'text/less': + elif extension == 'less': self.stylesheets.append(LessStylesheetAsset(self, **params, **css_params)) - elif f['atype'] == 'text/css': + elif extension == 'css': self.stylesheets.append(StylesheetAsset(self, **params, **css_params)) if js: - if f['atype'] == 'text/javascript': + if extension == 'js': self.javascripts.append(JavascriptAsset(self, **params)) - elif f['atype'] == 'text/xml': + elif extension == 'xml': self.templates.append(XMLAsset(self, **params)) - def to_node(self, css=True, js=True, async_load=False, defer_load=False, lazy_load=False): + def get_links(self): """ - :returns [(tagName, attributes, content)] if the tag is auto close + :returns a list of tuple. a tuple can be (url, None) or (None, inlineContent) """ response = [] - if css and self.stylesheets: + if self.has_css and self.stylesheets: css_attachments = self.css(is_minified=not self.is_debug_assets) or [] for attachment in css_attachments: if self.is_debug_assets: - href = self.get_debug_asset_url(extra='rtl/' if self.user_direction == 'rtl' else '', + href = self.get_debug_asset_url(extra='rtl/' if self.rtl else '', name=css_attachments.name, extension='') else: href = attachment.url - attr = dict([ - ["type", "text/css"], - ["rel", "stylesheet"], - ["href", href], - ['data-asset-bundle', self.name], - ['data-asset-version', self.get_version('css')], - ]) - response.append(("link", attr, None)) + response.append((href, None)) if self.css_errors: msg = '\n'.join(self.css_errors).replace('"', '\\"').replace('\n', '\\n') js_error = """ @@ -139,24 +131,13 @@ class AssetsBundle(object): } }); """ % msg - response.append(JavascriptAsset(self, inline=js_error).to_node()) - response.append(StylesheetAsset(self, url="/web/static/lib/bootstrap/dist/css/bootstrap.css").to_node()) + response.append((None, js_error)) + response.append(('/web/static/lib/bootstrap/dist/css/bootstrap.css', None)) - if js and self.javascripts: + if self.has_js and self.javascripts: js_attachment = self.js(is_minified=not self.is_debug_assets) src = self.get_debug_asset_url(name=js_attachment.name, extension='') if self.is_debug_assets else js_attachment[0].url - attr = dict([ - ["async", "async" if async_load else None], - # lazy_load will add defer in JS otherwise this is not W3C valid - # (defer is probably not needed there anyways) - ["defer", "defer" if defer_load else None], - ["type", "text/javascript"], - ["data-src" if lazy_load else "src", src], - ['data-asset-bundle', self.name], - ['data-asset-version', self.get_version('js')], - ['onerror', '__odooAssetError=1'] - ]) - response.append(("script", attr, None)) + response.append((src, None)) return response @@ -192,6 +173,7 @@ class AssetsBundle(object): 'name': name, 'sep': sep, 'extension': extension, + 'params': self.assets_params, } def get_asset_url(self, id='%', unique='%', extra='', name='%', sep="%", extension='%'): @@ -229,10 +211,10 @@ class AssetsBundle(object): ira = self.env['ir.attachment'] is_css = extension in ['css', 'min.css', 'css.map'] url = self.get_asset_url( - extra='%s' % ('rtl/' if is_css and self.user_direction == 'rtl' else ''), + extra='%s' % ('rtl/' if is_css and self.rtl else ''), name=self.name, - sep='', - extension='.%s' % extension + sep='.', + extension=extension, ) domain = [ @@ -244,8 +226,7 @@ class AssetsBundle(object): if attachments: self._unlink_attachments(attachments) - # force bundle invalidation on other workers - self.env['ir.qweb'].clear_caches() + # clear_cache was removed return True @@ -268,10 +249,10 @@ class AssetsBundle(object): url_pattern = self.get_asset_url( unique=unique, - extra='%s' % ('rtl/' if is_css and self.user_direction == 'rtl' else ''), # not sure about css.map + extra='%s' % ('rtl/' if is_css and self.rtl else ''), # not sure about css.map name=self.name, - sep='', - extension='.%s' % extension + sep='.', + extension=extension, ) self.env.cr.execute(""" SELECT max(id) @@ -285,6 +266,14 @@ class AssetsBundle(object): attachment_ids = [r[0] for r in self.env.cr.fetchall()] return self.env['ir.attachment'].sudo().browse(attachment_ids) + def add_post_rollback(self): + """ + In some rare cases it is possible that an attachment is created during a transaction, added to the ormcache + but the transaction is rollbacked, leading to 404 when getting the attachments. + This postrollback hook will help fix this issue by clearing the cache if it isn't commited. + """ + self.env.cr.postrollback.add(self.env.registry._Registry__cache.clear) + def save_attachment(self, extension, content): """Record the given bundle in an ir.attachment and delete all other ir.attachments referring to this bundle (with the same name and extension). @@ -318,19 +307,17 @@ class AssetsBundle(object): 'public': True, 'raw': content.encode('utf8'), } + self.add_post_rollback() attachment = ira.with_user(SUPERUSER_ID).create(values) url = self.get_asset_url( id=attachment.id, unique=self.get_version('css' if is_css else 'js'), - extra='%s' % ('rtl/' if extension in ['css', 'min.css'] and self.user_direction == 'rtl' else ''), - name=fname, - sep='', # included in fname - extension='' + extra='%s' % ('rtl/' if extension in ['css', 'min.css'] and self.rtl else ''), + name=self.name, + sep='.', # included in fname + extension=extension, ) - values = { - 'url': url, - } - attachment.write(values) + attachment.url = url if self.env.context.get('commit_assetsbundle') is True: self.env.cr.commit() @@ -588,7 +575,7 @@ class AssetsBundle(object): sourcemap_attachment = self.get_attachments('css.map') \ or self.save_attachment('css.map', '') debug_asset_url = self.get_debug_asset_url(name=self.name, - extra='rtl/' if self.user_direction == 'rtl' else '') + extra='rtl/' if self.rtl else '') generator = SourceMapGenerator( source_root="/".join( [".." for i in range(0, len(debug_asset_url.split("/")) - 2)] @@ -634,7 +621,7 @@ class AssetsBundle(object): compiled += self.compile_css(assets[0].compile, source) # We want to run rtlcss on normal css, so merge it in compiled - if self.user_direction == 'rtl': + if self.rtl: stylesheet_assets = [asset for asset in self.stylesheets if not isinstance(asset, (SassStylesheetAsset, ScssStylesheetAsset, LessStylesheetAsset))] compiled += '\n'.join([asset.get_source() for asset in stylesheet_assets]) compiled = self.run_rtlcss(compiled) @@ -798,9 +785,6 @@ class WebAsset(object): except ValueError: raise AssetNotFound("Could not find %s" % self.name) - def to_node(self): - raise NotImplementedError() - @property def last_modified(self): if self._last_modified is None: @@ -882,21 +866,6 @@ class JavascriptAsset(WebAsset): except AssetError as e: return u"console.error(%s);" % json.dumps(to_text(e)) - def to_node(self): - if self.url: - return ("script", dict([ - ["type", "text/javascript"], - ["src", self.html_url], - ['data-asset-bundle', self.bundle.name], - ['data-asset-version', self.bundle_version], - ]), None) - else: - return ("script", dict([ - ["type", "text/javascript"], - ["charset", "utf-8"], - ['data-asset-bundle', self.bundle.name], - ['data-asset-version', self.bundle_version], - ]), self.with_header()) def with_header(self, content=None, minimal=True): if minimal: @@ -927,7 +896,7 @@ class XMLAsset(WebAsset): try: content = super()._fetch_content() except AssetError as e: - return f'{json.dumps(to_text(e))}' + return u"console.error(%s);" % json.dumps(to_text(e)) parser = etree.XMLParser(ns_clean=True, recover=True, remove_comments=True) root = etree.parse(io.BytesIO(content.encode('utf-8')), parser=parser).getroot() @@ -939,17 +908,6 @@ class XMLAsset(WebAsset): def bundle_version(self): return self.bundle.get_version('js') - def to_node(self): - attributes = { - 'async': 'async', - 'defer': 'defer', - 'type': 'text/xml', - 'data-src': self.html_url, - 'data-asset-bundle': self.bundle.name, - 'data-asset-version': self.bundle_version, - } - return ("script", attributes, None) - def with_header(self, content=None): if content is None: content = self.content @@ -981,31 +939,22 @@ class StylesheetAsset(WebAsset): rx_sourceMap = re.compile(r'(/\*# sourceMappingURL=.*)', re.U) rx_charset = re.compile(r'(@charset "[^"]+";)', re.U) - def __init__(self, *args, media=None, direction=None, **kw): - self.media = media - self.direction = direction + def __init__(self, *args, rtl=False, **kw): + self.rtl = rtl super().__init__(*args, **kw) - if self.direction == 'rtl' and self.url: + if self.rtl and self.url: self.html_url_args = self.url.rsplit('.', 1) self.html_url_format = '%%s/%s/%s.%%s' % ('rtl', self.bundle.name) self.html_url_args = tuple(self.html_url_args) - @property - def content(self): - content = super().content - if self.media: - content = '@media %s { %s }' % (self.media, content) # this is not good! - return content - @property def bundle_version(self): return self.bundle.get_version('css') @func.lazy_property def unique_descriptor(self): - direction = self.direction or '' - media = self.media or '' - return f'{self.url or self.inline},{self.last_modified},{direction},{media}' # note that media shouldn't be used in unique since it isn't in extra + direction = (self.rtl and 'rtl') or 'ltr' + return f'{self.url or self.inline},{self.last_modified},{direction}' def _fetch_content(self): try: @@ -1047,26 +996,6 @@ class StylesheetAsset(WebAsset): content = re.sub(r' *([{}]) *', r'\1', content) return self.with_header(content) - def to_node(self): - if self.url: - attr = dict([ - ["type", "text/css"], - ["rel", "stylesheet"], - ["href", self.html_url], - ["media", escape(to_text(self.media)) if self.media else None], - ['data-asset-bundle', self.bundle.name], - ['data-asset-version', self.bundle_version], - ]) - return ("link", attr, None) - else: - attr = dict([ - ["type", "text/css"], - ["media", escape(to_text(self.media)) if self.media else None], - ['data-asset-bundle', self.bundle.name], - ['data-asset-version', self.bundle_version], - ]) - return ("style", attr, self.with_header()) - class PreprocessedCSS(StylesheetAsset): rx_import = None @@ -1074,7 +1003,7 @@ class PreprocessedCSS(StylesheetAsset): def __init__(self, *args, **kw): super().__init__(*args, **kw) self.html_url_args = tuple(self.url.rsplit('/', 1)) - self.html_url_format = '%%s/%s%s/%%s.css' % ('rtl/' if self.direction == 'rtl' else '', self.bundle.name) + self.html_url_format = '%%s/%s%s/%%s.css' % ('rtl/' if self.rtl else '', self.bundle.name) def get_command(self): raise NotImplementedError diff --git a/odoo/addons/base/models/ir_asset.py b/odoo/addons/base/models/ir_asset.py index d74a30671a5..1c68e368661 100644 --- a/odoo/addons/base/models/ir_asset.py +++ b/odoo/addons/base/models/ir_asset.py @@ -8,8 +8,7 @@ import odoo import odoo.modules.module # get_manifest, don't from-import it from odoo import api, fields, models, tools from odoo.tools import misc -from odoo.tools.constants import SCRIPT_EXTENSIONS, STYLE_EXTENSIONS, TEMPLATE_EXTENSIONS, ASSET_EXTENSIONS, EXTERNAL_ASSET - +from odoo.tools.constants import ASSET_EXTENSIONS, EXTERNAL_ASSET _logger = getLogger(__name__) @@ -91,7 +90,18 @@ class IrAsset(models.Model): active = fields.Boolean(string='active', default=True) sequence = fields.Integer(string="Sequence", default=DEFAULT_SEQUENCE, required=True) - def _get_asset_paths(self, bundle, css=False, js=False): + def _get_assets_params(self): + """ + This method can be overriden to add param _get_asset_paths call. + Those params will be part of the orm cache key + """ + return {} + + @tools.conditional( + 'xml' not in tools.config['dev_mode'], + tools.ormcache('bundle', 'tuple(sorted(assets_params.items()))'), + ) + def _get_asset_paths(self, bundle, assets_params): """ Fetches all asset file paths from a given list of addons matching a certain bundle. The returned list is composed of tuples containing the @@ -109,27 +119,21 @@ class IrAsset(models.Model): records matching the bundle are also applied to the current list. :param bundle: name of the bundle from which to fetch the file paths - :param css: boolean: whether or not to include style files - :param js: boolean: whether or not to include script files and template - files + :param assets_params: parameters needed by overrides, mainly website_id + see _get_assets_params :returns: the list of tuples (path, addon, bundle) """ installed = self._get_installed_addons_list() - addons = self._get_active_addons_list() + addons = self._get_active_addons_list(**assets_params) asset_paths = AssetPaths() - exts = [] - if js: - exts += SCRIPT_EXTENSIONS - exts += TEMPLATE_EXTENSIONS - if css: - exts += STYLE_EXTENSIONS addons = self._topological_sort(tuple(addons)) - self._fill_asset_paths(bundle, addons, installed, exts, asset_paths, []) + + self._fill_asset_paths(bundle, asset_paths, [], addons, installed, **assets_params) return asset_paths.list - def _fill_asset_paths(self, bundle, addons, installed, exts, asset_paths, seen): + def _fill_asset_paths(self, bundle, asset_paths, seen, addons, installed, **assets_params): """ Fills the given AssetPaths instance by applying the operations found in the matching bundle of the given addons manifests. @@ -150,22 +154,22 @@ class IrAsset(models.Model): # of the CURRENT bundle. bundle_start_index = len(asset_paths.list) - assets = self._get_related_assets([('bundle', '=', bundle)]).filtered('active') + assets = self._get_related_assets([('bundle', '=', bundle)], **assets_params).filtered('active') # 1. Process the first sequence of 'ir.asset' records for asset in assets.filtered(lambda a: a.sequence < DEFAULT_SEQUENCE): - self._process_path(bundle, asset.directive, asset.target, asset.path, addons, installed, exts, asset_paths, seen, bundle_start_index) + self._process_path(bundle, asset.directive, asset.target, asset.path, asset_paths, seen, addons, installed, bundle_start_index, **assets_params) # 2. Process all addons' manifests. for addon in addons: for command in odoo.modules.module.get_manifest(addon)['assets'].get(bundle, ()): directive, target, path_def = self._process_command(command) - self._process_path(bundle, directive, target, path_def, addons, installed, exts, asset_paths, seen, bundle_start_index) + self._process_path(bundle, directive, target, path_def, asset_paths, seen, addons, installed, bundle_start_index, **assets_params) # 3. Process the rest of 'ir.asset' records for asset in assets.filtered(lambda a: a.sequence >= DEFAULT_SEQUENCE): - self._process_path(bundle, asset.directive, asset.target, asset.path, addons, installed, exts, asset_paths, seen, bundle_start_index) + self._process_path(bundle, asset.directive, asset.target, asset.path, asset_paths, seen, addons, installed, bundle_start_index, **assets_params) - def _process_path(self, bundle, directive, target, path_def, addons, installed, exts, asset_paths, seen, bundle_start_index): + def _process_path(self, bundle, directive, target, path_def, asset_paths, seen, addons, installed, bundle_start_index, **assets_params): """ This sub function is meant to take a directive and a set of arguments and apply them to the current asset_paths list @@ -180,18 +184,17 @@ class IrAsset(models.Model): """ if directive == INCLUDE_DIRECTIVE: # recursively call this function for each INCLUDE_DIRECTIVE directive. - self._fill_asset_paths(path_def, addons, installed, exts, asset_paths, seen + [bundle]) + self._fill_asset_paths(path_def, asset_paths, seen + [bundle], addons, installed, **assets_params) return - if can_aggregate(path_def): - paths = self._get_paths(path_def, installed, exts) + paths = self._get_paths(path_def, installed) else: paths = [(path_def, EXTERNAL_ASSET, -1)] # external urls # retrieve target index when it applies if directive in DIRECTIVES_WITH_TARGET: - target_paths = self._get_paths(target, installed, exts) - if not target_paths and target.rpartition('.')[2] not in exts: + target_paths = self._get_paths(target, installed) + if not target_paths and target.rpartition('.')[2] not in ASSET_EXTENSIONS: # nothing to do: the extension of the target is wrong return if target_paths: @@ -237,14 +240,10 @@ class IrAsset(models.Model): :root_bundle: string: bundle from which to initiate the search. :returns: the first matching bundle or None """ - ext = target_path_def.split('.')[-1] installed = self._get_installed_addons_list() target_path, _full_path, _modified = self._get_paths(target_path_def, installed)[0] - - css = ext in STYLE_EXTENSIONS - js = ext in SCRIPT_EXTENSIONS or ext in TEMPLATE_EXTENSIONS - - asset_paths = self._get_asset_paths(root_bundle, css=css, js=js) + assets_params = self._get_assets_params() + asset_paths = self._get_asset_paths(root_bundle, assets_params) for path, _full_path, bundle, _modified in asset_paths: if path == target_path: @@ -280,7 +279,7 @@ class IrAsset(models.Model): return misc.topological_sort({manif['name']: manif['depends'] for manif in manifs}) @api.model - @tools.ormcache_context(keys='install_module') + @tools.ormcache() def _get_installed_addons_list(self): """ Returns the list of all installed addons. @@ -288,10 +287,9 @@ class IrAsset(models.Model): """ # Main source: the current registry list # Second source of modules: server wide modules - # Third source: the currently loading module from the context (similar to ir_ui_view) - return self.env.registry._init_modules.union(odoo.conf.server_wide_modules or []).union(self.env.context.get('install_module', [])) + return self.env.registry._init_modules.union(odoo.conf.server_wide_modules or []) - def _get_paths(self, path_def, installed, extensions=None): + def _get_paths(self, path_def, installed): """ Returns a list of tuple (path, full_path, modified) matching a given glob (path_def). The glob can only occur in the static direcory of an installed addon. @@ -352,11 +350,7 @@ class IrAsset(models.Model): msg += " It may be due to security reasons." _logger.warning(msg) # Paths are filtered on the extensions (if any). - return [ - path - for path in paths - if not extensions or path[0].rsplit('.', maxsplit=1)[-1] in extensions - ] + return paths def _process_command(self, command): """Parses a given command to return its directive, target and path definition.""" diff --git a/odoo/addons/base/models/ir_qweb.py b/odoo/addons/base/models/ir_qweb.py index 3e7eae8a7dc..dfd22c05ae9 100644 --- a/odoo/addons/base/models/ir_qweb.py +++ b/odoo/addons/base/models/ir_qweb.py @@ -2154,7 +2154,7 @@ class IrQWeb(models.AbstractModel): xmlid = el.attrib.pop('t-call-assets') css = self._compile_bool(el.attrib.pop('t-css', True)) js = self._compile_bool(el.attrib.pop('t-js', True)) - async_load = self._compile_bool(el.attrib.pop('async_load', False)) + # async_load support was removed defer_load = self._compile_bool(el.attrib.pop('defer_load', False)) lazy_load = self._compile_bool(el.attrib.pop('lazy_load', False)) media = el.attrib.pop('media', False) @@ -2164,7 +2164,6 @@ class IrQWeb(models.AbstractModel): css={css}, js={js}, debug=values.get("debug"), - async_load={async_load}, defer_load={defer_load}, lazy_load={lazy_load}, media={media!r}, @@ -2412,15 +2411,28 @@ class IrQWeb(models.AbstractModel): return (attributes, content, inherit_branding) - def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, async_load=False, defer_load=False, lazy_load=False, media=None): + def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None): """Generates asset nodes. If debug=assets, the assets will be regenerated when a file which composes them has been modified. Else, the assets will be generated only once and then stored in cache. """ - if debug and 'assets' in debug: - return self._generate_asset_nodes(bundle, css, js, debug, async_load, defer_load, lazy_load, media) + media = css and media or None + links = self._get_asset_links(bundle, css=css, js=js, debug=debug) + return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media) + + def _get_asset_links(self, bundle, css=True, js=True, debug=None): + """Generates asset nodes. + If debug=assets, the assets will be regenerated when a file which composes them has been modified. + Else, the assets will be generated only once and then stored in cache. + """ + rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl' + assets_params = self.env['ir.asset']._get_assets_params() # website_id + debug_assets = debug and 'assets' in debug + + if debug_assets: + return self._generate_asset_links(bundle, css=css, js=js, debug_assets=True, assets_params=assets_params, rtl=rtl) else: - return self._generate_asset_nodes_cache(bundle, css, js, debug, async_load, defer_load, lazy_load, media) + return self._generate_asset_links_cache(bundle, css=css, js=js, assets_params=assets_params, rtl=rtl) # qweb cache feature @@ -2472,80 +2484,91 @@ class IrQWeb(models.AbstractModel): # in non-xml-debug mode we want assets to be cached forever, and the admin can force a cache clear # by restarting the server after updating the source code (or using the "Clear server cache" in debug tools) 'xml' not in tools.config['dev_mode'], - tools.ormcache('bundle', 'css', 'js', 'debug', 'async_load', 'defer_load', 'lazy_load', 'media', 'tuple(self.env.context.get(k) for k in self._get_template_cache_keys())'), + tools.ormcache('bundle', 'css', 'js', 'tuple(sorted(assets_params.items()))', 'rtl'), ) - def _generate_asset_nodes_cache(self, bundle, css=True, js=True, debug=False, async_load=False, defer_load=False, lazy_load=False, media=None): - return self._generate_asset_nodes(bundle, css, js, debug, async_load, defer_load, lazy_load, media) + def _generate_asset_links_cache(self, bundle, css=True, js=True, assets_params=None, rtl=False): + return self._generate_asset_links(bundle, css, js, False, assets_params, rtl) - @tools.ormcache('bundle', 'defer_load', 'lazy_load', 'media', 'tuple(self.env.context.get(k) for k in self._get_template_cache_keys())') - def _get_asset_content(self, bundle, defer_load=False, lazy_load=False, media=None): - asset_paths = self.env['ir.asset']._get_asset_paths(bundle=bundle, css=True, js=True) + def _get_asset_content(self, bundle, assets_params=None): + if assets_params is None: + assets_params = self.env['ir.asset']._get_assets_params() # website_id + asset_paths = self.env['ir.asset']._get_asset_paths(bundle=bundle, assets_params=assets_params) files = [] - remains = [] + external_asset = [] for path, full_path, _bundle, last_modified in asset_paths: - ext = path.rpartition('.')[2] - is_js = ext in SCRIPT_EXTENSIONS - is_xml = ext in TEMPLATE_EXTENSIONS - is_css = ext in STYLE_EXTENSIONS - if not is_js and not is_xml and not is_css: - continue - - mimetype = None - if is_js: - mimetype = 'text/javascript' - elif is_css: - mimetype = f'text/{ext}' - elif is_xml: - mimetype = 'text/xml' - if full_path is not EXTERNAL_ASSET: files.append({ - 'atype': mimetype, 'url': path, 'filename': full_path, 'content': '', - 'media': media, 'last_modified': last_modified, }) else: - if is_js: - tag = 'script' - attributes = { - "type": mimetype, - } - attributes["data-src" if lazy_load else "src"] = path - if defer_load: - # lazy_load will add defer in JS otherwise this is not - # W3C valid (defer is probably not needed there anyways) - attributes["defer"] = "defer" - elif is_css: - tag = 'link' - attributes = { - "type": mimetype, - "rel": "stylesheet", - "href": path, - 'media': media, - } - elif is_xml: - tag = 'script' - attributes = { - "type": mimetype, - "async": "async", - "rel": "prefetch", - "data-src": path, - } - remains.append((tag, attributes, None)) + external_asset.append((path, None)) + return (files, external_asset) - return (files, remains) + def _get_asset_bundle(self, bundle_name, files=None, env=None, css=True, js=True, debug_assets=False, rtl=False, assets_params=None): + return AssetsBundle(bundle_name, files, env=env, css=css, js=js, debug_assets=debug_assets, rtl=rtl, assets_params=assets_params) - def _get_asset_bundle(self, bundle_name, files, env=None, css=True, js=True, debug=False): - return AssetsBundle(bundle_name, files, env=env, css=css, js=js, debug=debug) + def _links_to_nodes(self, paths, defer_load=False, lazy_load=False, media=None): + return [self._link_to_node(path, defer_load=defer_load, lazy_load=lazy_load, media=media) for path in paths] - def _generate_asset_nodes(self, bundle, css=True, js=True, debug=False, async_load=False, defer_load=False, lazy_load=False, media=None): - files, remains = self._get_asset_content(bundle, defer_load=defer_load, lazy_load=lazy_load, media=css and media or None) - asset = self._get_asset_bundle(bundle, files, env=self.env, css=css, js=js, debug=debug) - remains = [node for node in remains if (css and node[0] == 'link') or (js and node[0] == 'script')] - return remains + asset.to_node(css=css, js=js, async_load=async_load, defer_load=defer_load, lazy_load=lazy_load) + def _link_to_node(self, descriptor, defer_load=False, lazy_load=False, media=None): + path, js_content = descriptor # small hack to manage css compilation error, to transform into a full css solution or saved js error + ext = path.rsplit('.', maxsplit=1)[-1] if path else 'js' + is_js = ext in SCRIPT_EXTENSIONS + is_xml = ext in TEMPLATE_EXTENSIONS + is_css = ext in STYLE_EXTENSIONS + if not is_js and not is_xml and not is_css: + return + + if is_js: + is_asset_bundle = path and path.startswith('/web/assets/') + attributes = { + 'type': 'text/javascript', + } + + if (defer_load or lazy_load): + attributes['defer'] = 'defer' + if path: + if lazy_load: + attributes['data-src'] = path + else: + attributes['src'] = path + + if is_asset_bundle: + attributes['onerror'] = "__odooAssetError=1" + + if js_content: + attributes['charset'] = 'utf-8' + + return ('script', attributes, js_content) + + + if is_css: + attributes = { + 'type': f'text/{ext}', # we don't really expect to have anything else than pure css here + 'rel': 'stylesheet', + 'href': path, + 'media': media, + } + return ('link', attributes, None) + + if is_xml: + attributes = { + 'type': 'text/xml', + 'async': 'async', + 'rel': 'prefetch', + 'data-src': path, + } + return ('script', attributes, None) + + def _generate_asset_links(self, bundle, css=True, js=True, debug_assets=False, assets_params=None, rtl=False): + aggregable_files, external_assets = self._get_asset_content(bundle, assets_params=assets_params) + external_assets = [node for node in external_assets if (css and node[0].rpartition('.')[2] in STYLE_EXTENSIONS) or (js and node[0].rpartition('.')[2] in SCRIPT_EXTENSIONS)] + asset_bundle = self._get_asset_bundle(bundle, aggregable_files, env=self.env, css=css, js=js, debug_assets=debug_assets, rtl=rtl, assets_params=assets_params) + bundle_links = asset_bundle.get_links() + return external_assets + bundle_links def _get_asset_link_urls(self, bundle, debug=False): asset_nodes = self._get_asset_nodes(bundle, js=False, debug=debug) @@ -2575,16 +2598,16 @@ class IrQWeb(models.AbstractModel): js_bundles.add(asset) if css: css_bundles.add(asset) - nodes = [] + links = [] start = time.time() for bundle in sorted(js_bundles): - nodes += self._generate_asset_nodes(bundle, css=False, js=True) + links += self._generate_asset_links(bundle, css=False, js=True) _logger.info('JS Assets bundles generated in %s seconds', time.time()-start) start = time.time() for bundle in sorted(css_bundles): - nodes += self._generate_asset_nodes(bundle, css=True, js=False) + links += self._generate_asset_links(bundle, css=True, js=False) _logger.info('CSS Assets bundles generated in %s seconds', time.time()-start) - return nodes + return links def render(template_name, values, load, **options): diff --git a/odoo/addons/test_assetsbundle/tests/test_assetsbundle.py b/odoo/addons/test_assetsbundle/tests/test_assetsbundle.py index 1217dace5c6..136de75af5f 100644 --- a/odoo/addons/test_assetsbundle/tests/test_assetsbundle.py +++ b/odoo/addons/test_assetsbundle/tests/test_assetsbundle.py @@ -122,47 +122,57 @@ class FileTouchable(AddonManifestPatched): self.touches[filepath] = touch_time or time.time() return patch('os.path.getmtime', lambda filename: self.touches.get(filename) or GETMTINE(filename)) +@tagged('-at_install', 'post_install') +class Test404(HttpCase): + def get_assets_node_cache(self): + return [e for e in self.env.registry._Registry__cache.d if e[0] == 'ir.qweb' and '_generate_' in str(e[1])] #_generate_asset_nodes_cache + + def test_rollback(self): + """ + In some rare case, attachments can be generatted, added to the cache but the transaction is rollbacked leading to 404 + this actually tests the add_post_rollback behaviour + """ + self.env['ir.attachment'].search([('url', '=like', '/web/assets/%web.assets_frontend%')]).unlink() + self.env.registry._clear_cache() + self.url_open('/some_404_routes') + self.assertFalse(self.get_assets_node_cache(), 'orm_cache should be emptied by rollback') + self.assertFalse(self.env['ir.attachment'].search([('url', '=like', '/web/assets/%web.assets_frontend%%')]), "attachments should have been rollbacked") + self.url_open('/') + self.assertTrue(self.env['ir.attachment'].search([('url', '=like', '/web/assets/%web.assets_frontend%%')]), "attachments should have been generated as expected") + self.assertTrue(self.get_assets_node_cache(), 'orm_cache should remain populated') + self.env.registry._clear_cache() class TestJavascriptAssetsBundle(FileTouchable): @classmethod def setUpClass(cls): super().setUpClass() # this is mainly to avoid tests breaking when executed after pre-generate + cls.maxDiff = 10000 cls.env['ir.attachment'].search([('url', '=like', '/web/assets/%test_assetsbundle%')]).unlink() def setUp(self): super(TestJavascriptAssetsBundle, self).setUp() self.jsbundle_name = 'test_assetsbundle.bundle1' self.cssbundle_name = 'test_assetsbundle.bundle2' - self.env['res.lang']._activate_lang('ar_SY') - def _get_asset(self, bundle, env=None, debug=''): - env = (env or self.env) - files, _ = env['ir.qweb']._get_asset_content(bundle) - return AssetsBundle(bundle, files, env=env, debug=debug) + def _get_asset(self, bundle, rtl=False, debug_assets=False): + files, _ = self.env['ir.qweb']._get_asset_content(bundle) + return AssetsBundle(bundle, files, env=self.env, debug_assets=debug_assets, rtl=rtl) - def _any_ira_for_bundle(self, extension, lang=None): + def _any_ira_for_bundle(self, extension, rtl=False): """ Returns all ir.attachments associated to a bundle, regardless of the verion. """ - user_direction = self.env['res.lang']._lang_get(lang or self.env.user.lang).direction bundle = self.jsbundle_name if extension in ['js', 'min.js'] else self.cssbundle_name - rtl = 'rtl/' if extension in ['css', 'min.css'] and user_direction == 'rtl' else '' + rtl = 'rtl/' if rtl and extension in ['css', 'min.css'] else '' url = f'/web/assets/%-%/{rtl}{bundle}.{extension}' domain = [('url', '=like', url)] return self.env['ir.attachment'].search(domain) - def _node_to_list(self, nodes): - res = [] - for index, (tagName, t_attrs, content) in enumerate(nodes): - for name, value in t_attrs.items(): - res.append(value) - return res - def test_01_generation(self): """ Checks that a bundle creates an ir.attachment record when its `js` method is called for the first time and this ir.attachment is different depending on `is_minified` param. """ - self.bundle = self._get_asset(self.jsbundle_name, env=self.env) + self.bundle = self._get_asset(self.jsbundle_name) # there shouldn't be any minified attachment associated to this bundle self.assertEqual(len(self._any_ira_for_bundle('min.js')), 0, @@ -226,13 +236,13 @@ class TestJavascriptAssetsBundle(FileTouchable): def test_03_date_invalidation(self): """ Checks that a bundle is invalidated when one of its assets' modification date is changed. """ - bundle0 = self._get_asset(self.jsbundle_name, debug="assets") + bundle0 = self._get_asset(self.jsbundle_name, debug_assets=True) bundle0.js() last_modified0 = bundle0.get_checksum('js') version0 = bundle0.get_version('js') path = get_resource_path('test_assetsbundle', 'static', 'src', 'js', 'test_jsfile1.js') - bundle1 = self._get_asset(self.jsbundle_name, debug="assets") + bundle1 = self._get_asset(self.jsbundle_name, debug_assets=True) with self._touch(path): bundle1.js() @@ -284,10 +294,9 @@ class TestJavascriptAssetsBundle(FileTouchable): and create a minified ir.attachment. """ debug_bundle = self._get_asset(self.jsbundle_name) - nodes = debug_bundle.to_node() - content = self._node_to_list(nodes) + content = debug_bundle.get_links() # there should be a minified file - self.assertEqual(content[3].count('test_assetsbundle.bundle1.min.js'), 1) + self.assertIn('test_assetsbundle.bundle1.min.js', content[0][0]) # there should be one minified assets created in normal mode self.assertEqual(len(self._any_ira_for_bundle('min.js')), 1, @@ -297,35 +306,14 @@ class TestJavascriptAssetsBundle(FileTouchable): self.assertEqual(len(self._any_ira_for_bundle('js')), 0, "there shouldn't be any non-minified assets created in normal mode") - def test_06_debug(self): - """ Checks that a bundle rendered in debug 1 mode outputs non-minified assets - and create an non-minified ir.attachment. - """ - debug_bundle = self._get_asset(self.jsbundle_name, debug='1') - nodes = debug_bundle.to_node() - content = self._node_to_list(nodes) - # there should be a minified file - self.assertEqual(content[3].count('test_assetsbundle.bundle1.min.js'), 1, - "there should be one minified assets created in debug mode") - - # there should be one minified assets created in debug mode - self.assertEqual(len(self._any_ira_for_bundle('min.js')), 1, - "there should be one minified assets created in debug mode") - - # there shouldn't be any non-minified assets created in debug mode - self.assertEqual(len(self._any_ira_for_bundle('js')), 0, - "there shouldn't be any non-minified assets created in debug mode") - def test_07_debug_assets(self): """ Checks that a bundle rendered in debug assets mode outputs non-minified assets and create an non-minified ir.attachment at the . """ - debug_bundle = self._get_asset(self.jsbundle_name, debug='assets') - nodes = debug_bundle.to_node() - content = self._node_to_list(nodes) - # there should be a non-minified file (not .min.js) - self.assertEqual(content[3].count('test_assetsbundle.bundle1.js'), 1, - "there should be one non-minified assets created in debug assets mode") + debug_bundle = self._get_asset(self.jsbundle_name, debug_assets=True) + content = debug_bundle.get_links() + # there should be a minified file + self.assertIn('test_assetsbundle.bundle1.js', content[0][0], "there should be one non-minified assets created in debug assets mode") # there shouldn't be any minified assets created in debug mode self.assertEqual(len(self._any_ira_for_bundle('min.js')), 0, @@ -398,11 +386,10 @@ class TestJavascriptAssetsBundle(FileTouchable): def test_12_css_debug(self): """ Check that a bundle in debug mode outputs non-minified assets. """ - debug_bundle = self._get_asset(self.cssbundle_name, debug='assets') - nodes = debug_bundle.to_node() - content = self._node_to_list(nodes) - # find back one of the original asset file - self.assertIn('/web/assets/debug/test_assetsbundle.bundle2.css', content) + debug_bundle = self._get_asset(self.cssbundle_name, debug_assets=True) + content = debug_bundle.get_links() + # there should be a minified file + self.assertEqual(content[0][0], '/web/assets/debug/test_assetsbundle.bundle2.css') # there should be one css asset created in debug mode self.assertEqual(len(self._any_ira_for_bundle('css')), 1, @@ -423,9 +410,8 @@ class TestJavascriptAssetsBundle(FileTouchable): self.assertEqual(ira0.store_fname, ira1.store_fname) # the ir.attachment records should be deduplicated in the bundle's content - nodes = bundle0.to_node() - content = self._node_to_list(nodes) - self.assertEqual(content[2].count('test_assetsbundle.bundle2.min.css'), 1) + content = bundle0.get_links() + self.assertIn('test_assetsbundle.bundle2.min.css', content[0][0]) # Language direction specific tests @@ -433,17 +419,17 @@ class TestJavascriptAssetsBundle(FileTouchable): """ Checks that a bundle creates an ir.attachment record when its `css` method is called for the first time for language with different direction and separate bundle is created for rtl direction. """ - self.bundle = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'})) + self.bundle = self._get_asset(self.cssbundle_name, rtl=True) # there shouldn't be any attachment associated to this bundle - self.assertEqual(len(self._any_ira_for_bundle('min.css', lang='ar_SY')), 0) + self.assertEqual(len(self._any_ira_for_bundle('min.css', rtl=True)), 0) self.assertEqual(len(self.bundle.get_attachments('min.css')), 0) # trigger the first generation and, thus, the first save in database self.bundle.css() # there should be one attachment associated to this bundle - self.assertEqual(len(self._any_ira_for_bundle('min.css', lang='ar_SY')), 1) + self.assertEqual(len(self._any_ira_for_bundle('min.css', rtl=True)), 1) self.assertEqual(len(self.bundle.get_attachments('min.css')), 1) def test_16_ltr_and_rtl_css_access(self): @@ -473,23 +459,22 @@ class TestJavascriptAssetsBundle(FileTouchable): self.assertEqual(ltr_version0, ltr_version1) self.assertEqual(ltr_date0, ltr_date1) - # Assets access for ar_SY language - rtl_bundle0 = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'})) + rtl_bundle0 = self._get_asset(self.cssbundle_name, rtl=True) rtl_bundle0.css() - self.assertEqual(len(self._any_ira_for_bundle('min.css', lang='ar_SY')), 1) + self.assertEqual(len(self._any_ira_for_bundle('min.css', rtl=True)), 1) rtl_version0 = rtl_bundle0.get_version('css') - rtl_ira0 = self._any_ira_for_bundle('min.css', lang='ar_SY') + rtl_ira0 = self._any_ira_for_bundle('min.css', rtl=True) rtl_date0 = rtl_ira0.create_date - rtl_bundle1 = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'})) + rtl_bundle1 = self._get_asset(self.cssbundle_name, rtl=True) rtl_bundle1.css() - self.assertEqual(len(self._any_ira_for_bundle('min.css', lang='ar_SY')), 1) + self.assertEqual(len(self._any_ira_for_bundle('min.css', rtl=True)), 1) rtl_version1 = rtl_bundle1.get_version('css') - rtl_ira1 = self._any_ira_for_bundle('min.css', lang='ar_SY') + rtl_ira1 = self._any_ira_for_bundle('min.css', rtl=True) rtl_date1 = rtl_ira1.create_date self.assertEqual(rtl_version0, rtl_version1) @@ -507,14 +492,12 @@ class TestJavascriptAssetsBundle(FileTouchable): def test_17_css_bundle_date_invalidation(self): """ Checks that both css bundles are invalidated when one of its assets' modification date is changed """ - # Assets access for en_US language - ltr_bundle0 = self._get_asset(self.cssbundle_name, debug='assets') + ltr_bundle0 = self._get_asset(self.cssbundle_name, debug_assets=True) ltr_bundle0.css() ltr_last_modified0 = ltr_bundle0.get_checksum('css') ltr_version0 = ltr_bundle0.get_version('css') - # Assets access for ar_SY language - rtl_bundle0 = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'}), debug='assets') + rtl_bundle0 = self._get_asset(self.cssbundle_name, rtl=True, debug_assets=True) rtl_bundle0.css() rtl_last_modified0 = rtl_bundle0.get_checksum('css') rtl_version0 = rtl_bundle0.get_version('css') @@ -522,7 +505,7 @@ class TestJavascriptAssetsBundle(FileTouchable): # Touch test_cssfile1.css # Note: No lang specific context given while calling _get_asset so it will load assets for en_US path = get_resource_path('test_assetsbundle', 'static', 'src', 'css', 'test_cssfile1.css') - ltr_bundle1 = self._get_asset(self.cssbundle_name, debug='assets') + ltr_bundle1 = self._get_asset(self.cssbundle_name, debug_assets=True) with self._touch(path): ltr_bundle1.css() @@ -532,12 +515,12 @@ class TestJavascriptAssetsBundle(FileTouchable): self.assertNotEqual(ltr_last_modified0, ltr_last_modified1) self.assertNotEqual(ltr_version0, ltr_version1) - rtl_bundle1 = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'}), debug='assets') + rtl_bundle1 = self._get_asset(self.cssbundle_name, rtl=True, debug_assets=True) rtl_bundle1.css() rtl_last_modified1 = rtl_bundle1.get_checksum('css') rtl_version1 = rtl_bundle1.get_version('css') - rtl_ira1 = self._any_ira_for_bundle('min.css', lang='ar_SY') + rtl_ira1 = self._any_ira_for_bundle('min.css', rtl=True) self.assertNotEqual(rtl_last_modified0, rtl_last_modified1) self.assertNotEqual(rtl_version0, rtl_version1) @@ -560,7 +543,7 @@ class TestJavascriptAssetsBundle(FileTouchable): ltr_files0 = ltr_bundle0.files ltr_version0 = ltr_bundle0.get_version('css') - rtl_bundle0 = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'})) + rtl_bundle0 = self._get_asset(self.cssbundle_name, rtl=True) rtl_bundle0.css() rtl_files0 = rtl_bundle0.files rtl_version0 = rtl_bundle0.get_version('css') @@ -585,11 +568,11 @@ class TestJavascriptAssetsBundle(FileTouchable): self.assertNotEqual(ltr_files0, ltr_files1) self.assertNotEqual(ltr_version0, ltr_version1) - rtl_bundle1 = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'})) + rtl_bundle1 = self._get_asset(self.cssbundle_name, rtl=True) rtl_bundle1.css() rtl_files1 = rtl_bundle1.files rtl_version1 = rtl_bundle1.get_version('css') - rtl_ira1 = self._any_ira_for_bundle('min.css', lang='ar_SY') + rtl_ira1 = self._any_ira_for_bundle('min.css', rtl=True) self.assertNotEqual(rtl_files0, rtl_files1) self.assertNotEqual(rtl_version0, rtl_version1) @@ -606,12 +589,11 @@ class TestJavascriptAssetsBundle(FileTouchable): def test_19_css_in_debug_assets(self): """ Checks that a bundle rendered in debug mode(assets) with right to left language direction stores css files in assets bundle. """ - debug_bundle = self._get_asset(self.cssbundle_name, env=self.env(context={'lang': 'ar_SY'}), debug='assets') - nodes = debug_bundle.to_node() - content = self._node_to_list(nodes) + debug_bundle = self._get_asset(self.cssbundle_name, rtl=True, debug_assets=True) + content = debug_bundle.get_links() # there should be an css assets bundle in /debug/rtl if user's lang direction is rtl and debug=assets - self.assertIn('/web/assets/debug/rtl/{0}.css'.format(self.cssbundle_name), content, + self.assertEqual('/web/assets/debug/rtl/{0}.css'.format(self.cssbundle_name), content[0][0], "there should be an css assets bundle in /debug/rtl if user's lang direction is rtl and debug=assets") # there should be an css assets bundle created in /rtl if user's lang direction is rtl and debug=assets @@ -626,29 +608,20 @@ class TestJavascriptAssetsBundle(FileTouchable): attachments = self.env['ir.attachment'].search([('url', '=like', '/web/assets/%-%/test_assetsbundle.bundle4.%')]) self.assertEqual(len(attachments), 2) - asset_data_css = etree.HTML(html).xpath('//*[@data-asset-bundle]')[0] - asset_data_js = etree.HTML(html).xpath('//*[@data-asset-bundle]')[1] - format_data = { "js": attachments[0].url, "css": attachments[1].url, - "asset_bundle_css": asset_data_css.attrib.get('data-asset-bundle'), - "asset_version_css": asset_data_css.attrib.get('data-asset-version'), - "asset_bundle_js": asset_data_js.attrib.get('data-asset-bundle'), - "asset_version_js": asset_data_js.attrib.get('data-asset-version'), - } - - self.assertEqual(html.strip(), (""" + self.assertEqual(str(html.strip()), (""" - + - + @@ -659,27 +632,20 @@ class TestJavascriptAssetsBundle(FileTouchable): attachments = self.env['ir.attachment'].search([('url', '=like', '%/test_assetsbundle.bundle4.js')]) self.assertEqual(len(attachments), 1) - asset_data_css = etree.HTML(html).xpath('//*[@data-asset-bundle]')[0] - asset_data_js = etree.HTML(html).xpath('//*[@data-asset-bundle]')[1] - format_data = { - "asset_bundle_css": asset_data_css.attrib.get('data-asset-bundle'), - "asset_version_css": asset_data_css.attrib.get('data-asset-version'), - "asset_bundle_js": asset_data_js.attrib.get('data-asset-bundle'), - "asset_version_js": asset_data_js.attrib.get('data-asset-version'), "css": '/web/assets/debug/test_assetsbundle.bundle4.css', "js": '/web/assets/debug/test_assetsbundle.bundle4.js', } - self.assertEqual(html.strip(), (""" + self.assertEqual(str(html.strip()), (""" - + - + @@ -777,11 +743,11 @@ class TestAssetsBundleWithIRAMock(FileTouchable): def _get_asset(self): with patch.object(type(self.env['ir.asset']), '_get_installed_addons_list', Mock(return_value=self.installed_modules)): files, _ = self.env['ir.qweb']._get_asset_content(self.stylebundle_name) - return AssetsBundle(self.stylebundle_name, files, env=self.env, debug='assets') + return AssetsBundle(self.stylebundle_name, files, env=self.env, debug_assets=True) def _bundle(self, asset, should_create, should_unlink, reason=''): self.counter.clear() - asset.to_node() + asset.get_links() if should_create: self.assertEqual(self.counter['create'], 2, f'An attachment should have been created {reason}') else: @@ -1805,7 +1771,7 @@ class TestAssetsManifest(AddonManifestPatched): 'path': '/test_assetsbundle/%s' % path_to_dummy, }) - files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec') + files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec', {}) self.assertEqual(files, [('/test_assetsbundle/../../tests/dummy.xml', None, 'test_assetsbundle.irassetsec', None)]) # TODO, validate this behaviour # the idea is that if the second element is False (not None) it will be added to the assetbundle, but considered in any case as an attachment url) @@ -1822,7 +1788,7 @@ class TestAssetsManifest(AddonManifestPatched): 'bundle': 'test_assetsbundle.irassetsec', 'path': '%s' % path_to_dummy, }) - files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec') + files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec', {}) self.assertEqual(files, [('../../tests/dummy.xml', None, 'test_assetsbundle.irassetsec', None)]) def test_33(self): @@ -1871,7 +1837,7 @@ class TestAssetsManifest(AddonManifestPatched): 'bundle': 'test_assetsbundle.irassetsec', 'path': '/test_assetsbundle/data/ir_asset.xml', }) - files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec') + files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec', {}) self.assertEqual(files, [('/test_assetsbundle/data/ir_asset.xml', None, 'test_assetsbundle.irassetsec', None)]) def test_36(self): @@ -1880,7 +1846,7 @@ class TestAssetsManifest(AddonManifestPatched): 'bundle': 'test_assetsbundle.irassetsec', 'path': '/test_assetsbundle/static/accessible.xml', }) - files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec') + files = self.env['ir.asset']._get_asset_paths('test_assetsbundle.irassetsec', {}) modified = files[0][3] base_path = __file__.replace('/tests/test_assetsbundle.py', '') @@ -1925,3 +1891,136 @@ class TestAssetsManifest(AddonManifestPatched): .my_div.subdiv{color: blue;} """ ) + +@tagged('-at_install', 'post_install') +class AssetsNodeOrmCacheUsage(TransactionCase): + + def cache_keys(self): + keys = self.env.registry._Registry__cache.d + asset_keys = [key for key in keys if key[0] == 'ir.asset' and '_get_asset_paths' in str(key[1])] # ignore topological sort entry + qweb_keys = [key for key in keys if key[0] == 'ir.qweb'] + return asset_keys, qweb_keys + + def test_assets_node_orm_cache_usage_debug(self): + self.env.registry._clear_cache() + + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 0) + self.assertEqual(len(qweb_keys), 0) + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', debug='tests') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', debug='1') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + # in debug=assets, the ormcache is not used for _generate_asset_links_cache + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', debug='assets') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + def test_assets_node_orm_cache_usage_file_type(self): + self.env.registry._clear_cache() + + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 0) + self.assertEqual(len(qweb_keys), 0) + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', js=True, css=False) + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', js=False, css=True) + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 2) + + # NOTE: this result is not really desired but this is the current behaviour. In practice, we usually only generate one of them. + # This could be enforced or avoided + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', js=True, css=True) + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 3) + + + def test_assets_node_orm_cache_usage_lang(self): + self.env.registry._clear_cache() + self.env['res.lang']._activate_lang('ar_SY') + self.env['res.lang']._activate_lang('fr_FR') + self.env['res.lang']._activate_lang('en_US') + + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 0) + self.assertEqual(len(qweb_keys), 0) + + self.env['ir.qweb'].with_context(lang='fr_FR')._get_asset_nodes('web.assets_backend') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + self.env['ir.qweb'].with_context(lang='en_US')._get_asset_nodes('web.assets_backend') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + self.env['ir.qweb'].with_context(lang='ar_SY')._get_asset_nodes('web.assets_backend') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 2) # a second cache entry is created for rtl + + def test_assets_node_orm_cache_usage_website(self): + if self.env['ir.module.module'].search([('name', '=', 'website'), ('state', '=', 'uninstalled')]): + return # only makes sence if website is installed + self.env.registry._clear_cache() + + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 0) + self.assertEqual(len(qweb_keys), 0) + + self.env['ir.qweb'].with_context(website_id=None)._get_asset_nodes('web.assets_backend') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + self.env['ir.qweb'].with_context(website_id=1)._get_asset_nodes('web.assets_backend') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 2) # the content may be different for different websites, even if it is not always the case + self.assertEqual(len(qweb_keys), 2) + + def test_assets_node_orm_cache_usage_node_flags(self): + self.env.registry._clear_cache() + + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 0) + self.assertEqual(len(qweb_keys), 0) + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1) + self.assertEqual(len(qweb_keys), 1) + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', media='print') + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1, "media shouldn't create another entry") + self.assertEqual(len(qweb_keys), 1, "media shouldn't create another entry") + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', defer_load=True) + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1, "defer_load shouldn't create another entry") + self.assertEqual(len(qweb_keys), 1, "defer_load shouldn't create another entry") + + self.env['ir.qweb']._get_asset_nodes('web.assets_backend', lazy_load=True) + asset_keys, qweb_keys = self.cache_keys() + self.assertEqual(len(asset_keys), 1, "lazy_load shouldn't create another entry") + self.assertEqual(len(qweb_keys), 1, "lazy_load shouldn't create another entry") diff --git a/odoo/tests/common.py b/odoo/tests/common.py index 2614111e1c2..2fa3849de7f 100644 --- a/odoo/tests/common.py +++ b/odoo/tests/common.py @@ -653,6 +653,7 @@ class BaseCase(case.TestCase, metaclass=MetaCase): when normalizing both archs. Takes either "xml" or "html" :type parser: str """ + self.maxDiff = 10000 if original: original = _normalize_arch_for_assert(original, parser) if expected: