The 'like'/'ilike' operators automatically add the wildcard character
(`%`) at the beginning and at the end of the value. This commit fixes
the incorrect usage.
closesodoo/odoo#136007
Related: odoo/enterprise#47886
Signed-off-by: Raphael Collet <rco@odoo.com>
When loading a registry from the route with auth="none" and where module
'website' is set as 'to install', request.env will be None, and install will
crash as follows:
Error:-
AttributeError: 'NoneType' object has no attribute 'context'
reference of previous commit-
https://github.com/odoo/odoo/commit/0ebf849b8e0de33d3b6e7e06c66189591dc7a72c
sentry-42474563
closesodoo/odoo#136096
X-original-commit: ca43e03e6f510520e874b2a8bba092ac9a7f2453
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Issue:
Search on website has a big performance issue because the database has
too many product_template.
Analyze :
The website search mechanism has hardcoded sql query. And the generic
query don't have any filtering. Which lead to make a expensive search
(similarity) on the whole product_template even if the products are
not published and then displayed to the user.
Fix :
Filter the is_published field in the generic website method when the
model has a 'is_published' field and the request is not done by a user
(thus for customers or portal users).
Note:
The internal user is still able to search on unpublished product.Thus
have no performance improvement.
Benchmark:
| SQL Query | # Input data | Before PR | After PR |
|:---------:|:------------:|:---------:|:--------:|
|General best_similarity |700000 products (161 published) | 12.65 s | 0.12 s |
|FROM ir_translation|2,796,000 ir_translation | 6.338 s (586k hit) | 0.091 s (185 hit) |
Related task:
task-3473786
Related ticket:
opw-3418457
closesodoo/odoo#136080
X-original-commit: fcb968836a2d0e49516fe3ecb23dde61f4597749
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
When building a website with the website configurator, you get pages
with a nice layout. However, the text inside is not adapted to your
company, industry or even, most of the time, language.
Using a LLM, we could easily get text much more relevant to customer
needs. The scope of this work is static pages generated with the website
configurator (homepage, about us, pricing, ...).
Later, in other tasks, we also want users to be able to generate text
inside the website builder.
This commit implement the bare minimum to generate and replace a website
content by AI generated sentences (based on the information provided by
the user).
Related IAP PR at https://github.com/odoo/iap-apps/pull/656
task-3248852
closesodoo/odoo#121021
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Co-authored-by: qsm-odoo <qsm@odoo.com>
During the rendering of website configurator pages, we check that the
rendering of each snippet is done for some reason. That check is useless
because:
- Either the rendering fails and in that case, there is a raised
exception that we catch (note that catching it could also be useless
actually: we will check that post 17.0 release).
- Either the rendering is empty (weird case to cover ?) and the
remaining code should work regardless anyway.
Note that in the original code, that `if` covered the case of trying to
render a non existing view... but was changed in [1] such that in that
case, an exception (that we catch) will be raised.
[1]: https://github.com/odoo/odoo/commit/880954ebfc1106411b7f7a7d60aee05dfae60893
Related to task-3248852
Part-of: odoo/odoo#121021
The configurator code organization should be reviewed. This is a first
step that will be needed for the integration of a new feature: this
removes the one-call inner functions inside that `configurator_apply`.
Related to task-3248852
Part-of: odoo/odoo#121021
Users sometime want to rename existing controllers URL, e.g. /shop as
french /magasin. The feature is possible within Odoo thanks to 308 type
redirections that can be configured via a hidden ?debug=1-only website
menu.
Upon generating the routing-map (the structure that links URLs to
controllers), when a URL is found inside of the `website.rewrite` model,
two links (called Rules in werkzeug jargon) are registered: the new,
translated, URL is linked to the controller and the original URL is
linked to a redirection to the new URL.
For the context of this PR, it is important to note that the redirection
only applies to the very URL saved inside the `website.rewrite` model:
if a controller has multiple routes, e.g. `/shop` and `/shop/shop`, only
`/shop` is redirected to `/magasin`, `/shop/shop` is left as-is.
Without 308 redirection:
/shop -> def shop()
/shop/shop -> def shop()
With 308 redirection:
website.rewrite(from_url='/shop', to_url='/magasin')
/shop -> /magasin
/shop/shop -> def shop()
/magasin -> def shop()
The redirection is set on the routing dictionnary of the endpoint, this
is the dictionnary that collect the informations set via the `@route`
decorator (auth=, method=, type=, ...).
Prior to [HTTPocalypse], that dictionnary was duplicated so that the
redirection was applied on the single route endpoint that matched
the `website.rewrite` record. With [HTTPocalypse] that duplication has
been wrongly removed: all original routes redirected to the new
translated one.
Bug introduced in [HTTPocalypse]:
website.rewrite(from_url='/shop', to_url='/magasin')
/shop -> /magasin
/shop/shop -> /magasin <-- wrong
/magasin -> def shop()
This PR fixes the problem, it makes sure that the redirection is saved
*only* on the route that matched the website.rewrite record, not the
other routes.
[HTTPocalypse]: https://github.com/odoo/odoo/pull/78857closesodoo/odoo#133960closesodoo/odoo#134206
X-original-commit: 144a22c22c95004171860cdaecd1f7d7975dc468
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Since [1] if the partner related to a visitor belongs to a company that
cannot be accessed by the current user, an error is raised when trying
to display the list of visitors.
This commit makes the name obtained through an additional sudo on the
`partner_id` record which might not be accessible to the current user.
This is equivalent to using the `name` field of visitor, except that
accessing the `partner_id` field will fail if `website.visitor` itself
cannot be accessed.
Steps to reproduce:
- Go to Settings / Companies
- Create a second company
- Go to Website / Reporting / Visitors
- Select "Edwin Hansen"
- Navigate to its linked partner in the contact field
"Gemini Furniture, Edwin Hansen"
- Navigate to its company "Gemini Furniture"
- In the "Sales & Purchase" tab, assign the second company as the
Company
- Save (this creates an error - but it is saved)
- Go to Website / Reporting / Visitors
=> The page was not displayed and an access error message was
displayed.
[1]: https://github.com/odoo/odoo/commit/d348bed1ad9d3d16b295f013f015706be6c07820
opw-3462104
closesodoo/odoo#134190
X-original-commit: a75673217aef2ffaaa93585526dc1e647a270495
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Issue
=====
The previous commit forced consistency between `check_access_rule` and
`_apply_ir_rule`. Now the parent model `ir.rule` (via inherited) is also
checked before the unlinking (`check_access_rule("unlink")`). The
`website.page` unlink override sometimes calls `unlink` on its parent
view and because view_id has `ondelete="cascade"`, it will actually
deletes the `website.page` itself. Then calling to `super().unlink()`
with self will raise a MissingError.
Fix
===
Batch the old logic and remove already unlinked record from `self`
before calling `super`.
closesodoo/odoo#125916
Signed-off-by: Raphael Collet <rco@odoo.com>
Steps to reproduce (on Chrome with browser cache enabled):
- Go to website > Create a new blog post.
- Switch to edit mode > Add an image to the blog cover from unsplash.
- After save, switch to a secondary language > Unsplash images disappear
and you need to refresh the page to get them to appear.
This behaviour is a "very specific" Chrome related issue: when switching
language, Chrome cannot load inline background images correctly right
after the redirect... and unlike other image URLs, background images
don't have the language code prefix (The `url_for()` will add the
language code before the image "src" E.g. `<img src="/unsplash/...."/>`
=> `<img src="/fr_BE/unsplash/...."/>` when switching to `fr_BE`...).
The goal of this commit is to prevent this behaviour by adapting the
"background-image" URLs in the same way as image "src".
opw-3412961
closesodoo/odoo#131891
X-original-commit: 5f20f50f9a9fd4bc5145d380609cdca80a8d7511
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Since #121376 a clear_cache was removed when unlinking an attachment
This clear_cache was not useful when restarting a server with new
sources, an other operation changing the content of an assets should
invalidate the cache manually. This is the case of ir.asset CUD
operations.
Unfortunately, a manual update was left missing in website, when
archiving ir.assets used for snippets.
This was discovered on runbot, with two workers, when the first workers
generates assets before the cron, and the second one after the cron.
The second worker unlinks attachments creating an inconsistency in the
cache of the first one.
This problem can be solved quickly by invalidating the assets cache in
the cron manually but this will be done in all cases. The proposed
solution will check the ir.asset that should change and only clear the
cache if the state changed.
Regarding performances, this should actually be a slight improvement in
query count since at the cost of one more select to prefetch the record
we can avoid multiple update, one per snippet. In most case no update at
all should be done, at most 2 can be done (one for archive, one for
unarchive).
Example with some assets to unarchive:
Before:
TOTAL ENTRIES: 277
SELECT: 158 (~0.16591858863830566s)
UNKWOW: 54 (~0.0202481746673584s)
UPDATE: 65 (~0.03202557563781738s)
After:
TOTAL ENTRIES: 214
SELECT: 159 (~0.1663439826965332s)
UNKWOW: 54 (~0.029229164123535156s)
UPDATE: 1 (~0.0002865791320800781s)
If the number of query is lowered, the python processing is slightly
higher. Locally the test time is similar, slightly longer since an
additional call to _disable_unused_snippets_assets was added to check
the cache invalidation
closesodoo/odoo#130973
X-original-commit: 3aeab51ff4379e6d76172614fd39f09d6c449ac6
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Since commit [1] (introduced in Odoo 15), when a published website page
has no URL set, the search snippet would crash when typing something
inside it.
I am not sure why was the page url not required at the model level, this
will be considered in master.
Note that to create a page without a URL, the only flow is to create a
new page in the backend.
Indeed, the `write` is overridden to add a trailing `/`, and the
frontend flows to create/write a page are also shielded against empty
URL.
Note that it's easy to create a page without a URL through legit flows.
Depending on the Odoo version, there is always a way to reach a website
page form view: either simply Website > Configuration > Pages or
Site > Pages > Debug mode > Click on Bug icon in tree view > Click on
page m2o.
Step to reproduce:
- Create a page with no URL (see above explanation how to do it)
- Make sure it's publish (you can do it in the form view)
- Drag & drop the Search snippet on a page
- Type something in the search snippet
-> TB
[1]: https://github.com/odoo/odoo/commit/9f9c4bb7e40233e633f97c60fb00ae191e9077af#diff-77bd6b19c39e211959885024bcad914655ff84cfc10c16633687d014e50aa69aR69
Fixes https://github.com/odoo/odoo/issues/129728closesodoo/odoo#129915
X-original-commit: f75ec6131eff9165b06d42d27a49dd756387f4a0
Signed-off-by: Benoit Socias (bso) <bso@odoo.com>
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Since [this commit], when the user configures a new website, he can
set an unknown industry. Unfortunately, if he does so, by default, we
set the industry to abbey, which is not very good for the default images
the user will see and for internal IAP stats. This commit permits to use
the new industry called Unknown, which is used for unidentified
industries. When the industry is unknown, we leave the default theme
images.
[this commit]: https://github.com/odoo/odoo/commit/e82a1cb2ef1bcb413d99e7eb7521405d1d2e88d2
IAP PR: https://github.com/odoo/iap-apps/pull/635
task-3337894
closesodoo/odoo#129154
X-original-commit: 17dd3a84d9d9d42b04fde9aac708c032a472b13b
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Guillaume Dieleman (gdi) <gdi@odoo.com>
**Description of the issue/feature this PR addresses:**
- Increase user experience in creating website themes
- Eliminate redundant creation of 1 record leading to unnecessary model
`ir_model_data` contraints error
**Recreate the Situation**
- Install the website module
- Choose `a business website` in line 1, `fashion designer` in line 2,
`get lead` in line 3 on `configurator - step 2`
- Choose random paletee on `configurator - step 3`
- `Build my website` on `configurator - step 4`
- Suppose in this step that we have a bad connection to the api
`https://website.api.odoo.com`, the error `duplicate key value violates
unique constraint` will appear.
**Reason**
- When a user creates a website, the `configurator_apply` function is
called to the api `https://website.api.odoo.com`. If the connection time
exceeds 5s, the `configurator_apply` function will be called again.
Maximum of 3 times
- Each time `configurator_apply` function runs, will create records of
model `ir_model_data` at `set_images` function. This function will
create records with a name field like
`configurator_1_s_cover_default_image`.
- If this function runs again for the second time, it will also generate
a `configurator_1_s_cover_default_image` for the second time. This will
result in a `duplicate key value violates unique constraint` error.
**Before commit:**
- When creating a website theme by industry, if the user's connection to
odoo's api has a delay, it will cause an api contraints error of model
`ir_model_data`
**After commit:**
- Check if the record of model `ir_model_data` has been created before,
if created then skip
closesodoo/odoo#128959
X-original-commit: 56ccf4efeb946e2e0b6ef54b977b6d8b070306ee
Signed-off-by: Benoit Socias (bso) <bso@odoo.com>
One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.
Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.
The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.
Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.
The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.
We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.
Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.
Note that this poc is inspired from the long term cache but not all
use case where applie yet.
Part-of: odoo/odoo#119813
Since commit [1], as soon as you have website installed, the result of
google_recaptcha was ignored. It is because website depends of
google_recaptcha and not the opposite.
Now we don't override the result of google recaptcha in website. And if
you have recaptcha + turnstile, Turnstile will check the result of
google_recatpcha first, and if result is valid, add his own test.
[1]: https://github.com/odoo/odoo/commit/4aca39a533e9d41f5f452f36a1ffc001f586b4f4
opw-3380702
opw-3392206
closesodoo/odoo#128634
X-original-commit: c8cc447f70f2a132d49b57c3c61c181a63876d26
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
If a user is not present in the request, he is in no group at all and
can not access any model, including the one available for public
users.
Avoid ambiguity by using sudo or add a user specifically.
Part-of: odoo/odoo#125216
When deleting a page, `search_url_dependencies` tries to trawl through
all models to see if they might have a link to the page being deleted.
However if the user invoking that function does not have access to a
model with an HTML field, it raises an error. Given pages are managed
by website designers which are *not administrators* there is no reason
to believe the current user has access to every model in the
database (not that even admins do these days). Since
`search_url_dependencies` is a best-effort search anyway, just ignore
any model to which the current user doesn't have access.
An alternative would be to do the search in sudo mode, but that
doesn't seem necessary, and could even be problematic if a match is
found:
- it might leak information the user should not access (because the
record name is returned, as well as the model & field names)
- it will trigger further access errors (because links to problematic
records are provided, on which the user might want to click)
closesodoo/odoo#127973
X-original-commit: 47a69a186bc415bf61a523da9686ca96c0beb74e
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
The website frontend apps menu list is not working when the user has a
`Home Action` defined on his user.
The `Home Action` is meant to redirect to the defined action whenever
that user is login in.
But since the backend menu links on the website have most of the time
no `action` defined but just a `menu_id` defined, the `Home Action` will
kick in and take over the redirection, the same way as if the user just
type `/web` without any params.
To solve that, we simply force the `action` of those links (if they
don't already have one).
This will make sure that the redirect is working as it should for users
having a `Home Action` set.
Step to reproduce:
- Set a Home Action for any user, like "Contacts"
- Go to the website frontend, eg on `/`.
- Click on the top left button to show the backend app menus list
- Click on any menu (CRM, Invoicing, Calendar..)
-> Most of those menu will not redirect you were you are supposed to be
but on your Home Action instead.
You can figure which one will be buggy or not by just mouseovering
the link and see if the URL param `action` is set to something or
not.
--- Technical hints ---
There is multiple methods to get the list of menus in Odoo:
- `load_web_menus`: called by the web client rpc, calling `load_menus`.
If a top/app menu has no action defined on it, it sets the first found
action of their children menus to it.
It returns the full (flat) list of menus, not only the top/app ones.
This method is not ormcached but is calling an ormcached method and
just doing some tiny work on the data.
- `load_menus_root`: called only by website backend template to add the
app list on the website (in the frontend) to jump to the backend.
It does not force the action if a menu has no action set on it.
It returns only the top/app menus.
This method is ormcached.
Note that this method seems only used by the website module.
- `load_menus`: returns the full (flat) list of menus without a force
action
This method is ormcached.
Fixes https://github.com/odoo/odoo/issues/119971
task-3378963
closesodoo/odoo#127840
X-original-commit: f28a349aa40b2ea48ef8f2b71e806b965777014f
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Before the task
when the default language for a website is not en_US, and users try to
edit_translations from the website, all terms are marked "translated"
because odoo adds
`data-oe-translation-state="to translate"` if the term is not extracted from
the en_US value and is the same as its en_US term
After this commit:
if the record's bounded website's default language is lang_base
odoo adds
`data-oe-translation-state="to translate"` if the term is not extracted from
the lang_base value and is the same as its lang_base term
closesodoo/odoo#127468
Task-id: 3344973
X-original-commit: 7c20510bda2f1312a9392df445ee38aea7b2267b
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
Signed-off-by: Chong Wang (cwg) <cwg@odoo.com>
The previous commit introduced an optimization to reduce the number of
queries and fields fetched when we call `name_search`. But it works much
better when the dependencies of `display_name` contain field names used
in the calculation (on the same record/model).
Then, to improve the performance and the cache coherency, add `depends`
and `depends_context` depending on the custom `_compute_display_name`.
Add only the first level of dependencies (never traverse relational
field) because only these have a positive impact on the previous
optimization and the cost is very low (see `modified`).
About `depends_context`, we don't include `lang` because (when `_` is
used by example) it is unlikely to get the same display_name in the same
request with two different lang.
closesodoo/odoo#122085
Related: odoo/documentation#4639
Related: odoo/enterprise#42599
Related: odoo/upgrade#4780
Signed-off-by: Raphael Collet <rco@odoo.com>
Rationale
=========
Since v8, the `display_name` field is present on all models. By default,
`display_name` uses `name_get` which has pretty much the same purpose
(return record name used by the web client). Gradually, many (backend)
developers (and the ORM: https://github.com/odoo/odoo/commit/6da1c3ac4c036eac289597602976538e243cb939)
started using `display_name` (more convenient than
`record.name_get()[0][1]`) but it still had the `name_get` override.
It becomes more complex than necessary and poeple start to misunderstand
the two (and sometimes override both, leading to inconstiencies between
`display_name`/`name_get`).
To simplify the ORM and the API, we decided to keep only one of them,
the `display_name` field:
- It is much more convenient from a backend point of view
(`record.name_get()[0][1]` vs `record.display_name`)
- It is cached during the same transaction (and invalidated if
its dependencies change)
- It can be overridden like any other compute field (override
`_compute_display_name` with any extra dependencies)
- `name_get` is replaced by `read(['display_name'])`
(API perceptive), which can actually be more efficient
(if `display_name`'s depends are correct, the ORM will only fetch the
fields it needs instead of every prefetchable field)
Changes
=======
- Deprecates `name_get` for the v17 and based the method on
`display_name` (the opposite of before)
- Converts all usage of `name_get`
- Overrides of `name_get` are now overrides of `_compute_display_name`
- For `res.partner`, rename the field store `display_name` into
`complete_name` because `display_name` context-dependent and it makes
no sense to have a compute store that is context-dependent.
- Previously, it was possible to return multiple names for the same
record with `name_get`, but it was tricky and most of the usage of
this `name_get` didn't take this into account. The only example of
this is the `name_get` of `product.product`
(now use `", ".join(<names>)`).
Part-of: odoo/odoo#122085
TypeError: expected string or bytes-like object
This error occurs when the 'url_from' field is left empty during website rewrite
To reproduce this issue, follow these steps:
1. go to website -> configuration -> redirects
2. create a new record -> select 308 redirect / rewrite in action
3. keep 'url from' empty and put any value in 'url to' such as '/'
4. Click on the save button, and this error message will be displayed.
After applying this commit will fix this issue.
task-3346588
sentry-4229658026
closesodoo/odoo#124558
X-original-commit: 26fa923f6ab01298bf1a1e9f6a615c9aa9e2e5ed
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
This is a proposal to try to find a similar asset before generating one.
If get_attachments fails, the next step will be to generate the
attachments from scratch, a slow operations.
When creating a new website, all assetsbundle would actually be
similar to their version without website, but the url is different.
This can be visible because the first loading of /web is slow after
creating a new website: the website_id is forced in the session
and the assets_backend are regnerated, identical to the original ones.
This commit proposes to try to find an attachments with differents extra
but the same uniquifier when possible and copy it's content.
Note that just returning the other attachement url may work, but it
would be confusing to randomly have links to assets comming from another
website_id. This would also be a problem if the original attachment is
unlinked, forcing to recompute it for other websites.
Good to know, since the content is the same, no duplication of the
content should appear in the filestore, just an entry in the database.
Part-of: odoo/odoo#121376
1. move cache to _get_asset_paths
The `_get_asset_content` cache has many cache key that are related to a
posprocessing of the `_get_asset_paths` result, the heavy part of this
method. Moving the cache to _get_asset_content will have the benefit
to create less duplicates entries in the ormcache as well as less cache
miss.
To simplify even further, the css and js parameters are removed since
they only filter the output of get_paths, the heavy part of globing the
file will be done before that. Anyway, they are both true when called
from _get_asset_content, and the only other call, in
`_get_related_bundle` don't really need to filter them since it is not
a critical part regarding performance, and the funtional result will
stay the same.
The initial orm cache key was using `_get_template_cache_keys`, a little
overkill and possibly creating duplicates entries again. The only
context key needed is website_id for `_get_related_assets`.
Note that it is not really enough, the orm cache key should actually
contain `request.session.get('force_website_id')` as well has
`request.httprequest.host`. This will be addressed latter since a nicer
solution would be to have website_id as a unique parameter computed
earlier.
2. better _get_asset_paths cache key
The orm cache key was simplified in previous point but there is still
one concern, the website_id depends on more parameters than that:
- request.session.get('force_website_id')
- request.httprequest.host
- existing websites
The idea here is to call `get_current_website` instead of using all
parameters that could define the webiste.
In the same spirit of `_get_template_cache_keys` `_assets_path_params`
can be overriden to give extra params that are usefull to list assets
path. Those params are computed before entering the method
`_get_asset_paths`. This may latter put at a higher level latter, in
get_asset_node, to simplify the _generate_asset_nodes_cache key.
3. better assets_node caches key
The main purpose of this part is to improve ormcache containing assets
nodes. The ormcache key contains
- to much context key
- missing session/host/env info
- unwanted boolean options.
- keys leading to the same cache value
The main goal being to reduce the size of the cache keys, decrease the
number of cache entries and improve the cache hit.
This will also make the behaviour more coherent and hopefully less bug
prone because of mismatch in parameters.
The main reason of the orm cache is the slowness of the validation of
the assets. This includes:
- listing files (dedicated orm cache)
- computing version
The cache key was depending on
- `debug`
The only relevant value for debug is "contains assets"
We dont need to differ between debug='', debug='1', debug='test',
and 'debug=assets', 'debug=tests,assets', ...
- `defer_load`, `lazy_load`, `media`
Those values are only useful to generate html node, a leightweight
operations that does not really needs to be in cache. `media` was also
used in the generation but it looks useless if we have the media on the
node. THIS NEEDS TO BE VALIDATED but in any case, since media is not
used to generate the url, it doesn't make sence to use it in the
generation.
The main idea to remove them from the ormcache key is simply to generate
the nodes outide the ormcached values.
-`async_load`
This one is similar to `defer_load` and `lazy_load` but it looks like
it wasn't used anymore. This was simply removed
- context.get('lang')
The only information needed is the direction, rtl or ltr. This means
en and fr languages, despite sharing the same css assets, will duplicate
the ormcache entries.
-`_get_template_cache_keys`
Only the lang and webiste where really relevant in this flow. Other
keys are actually useless in this flow.
Some information used in the generation where not in the orm cache key
- `self.env.user.lang` if there is no lang in the context
- `request.session.get('force_website_id')`
- `request.httprequest.host`
- ...
The proposed solution is to:
- extract any informùation needed from thecontext, request, environment
before entering the ormcache, reduce it to the minimal possible set of
values needed
```
rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl'
assets_params = self.env['ir.asset']._get_assets_params() # website_id
debug_assets = debug and 'assets' in debug
```
and remove a leightweight part of the logic
```
def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None):
links = self._get_asset_links(bundle, css=css, js=js, debug=debug)
return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media)
```
Where _get_asset_links is the cached part, and _links_to_nodes is the
lightweight part generating the nodes based on the `defer_load`, ....
Additionnal notes:
- data-asset-version and data-asset-bundle are removed from the node
since they don't seem to be used anymore since 65d70acdbf
- async_load is removed since there is no occurence of this in the code.
- a small hack is still needed to pass javascript content instead of
links, this is only to manage css compile error and will hopefully be
removed in the future.
- a context key is still in use to generate the bundle, the
`commit_assetsbundle` but it has no impact on content and will hopefully
be removed in the future.
4. Add test for ormcache hit/miss
In this context, hit/miss is about having the same cache key for the
same result. This test demonstrates the current state, were entries are
create in the ormcache only if the key is really different and will lead
to a different result.
5. remove cache invalidation
This cache invalidation is quite agressive since everytime an
assetbundle is updated, all workers will clear their cache.
The concerned cache by this clear_cache is `_generate_asset_nodes_cache`
throug `_get_asset_nodes`.
The cache is ignored, both in dev=xml and debug=assets.
This clear cache was made conditionnal in 553ea82f81 but this does
not solve an issue we can have in production.
Lets imagine a clean solution
- all sources are updated
- all workers are restarted.
The orm caches are all empty, but since the sources
changed, all bundles will be recomputed. This means that every bundle
updated in database with save_attachement will invalidate the cache of
all workers. Rendering a pdf report of any kind using a specific bundle
will invalidate all cache. Starting a debug=assets for the first time
will invalidate all cache, even if the cache is not used in this case.
But for a regenerated bundle we would expect the ormcache to be:
- empty (did not generate the same bundle yet)
- have the same value (concurrent generation of the same bundle)
Having a different value would mean that the bundle was generated with
another version of the sources. In this case it is maybe even better not
to invalidate the cache since it could lead to an invalidation war
between two workers.
The only case where invalidating this cache is useful is when a bundle
changes, Usually if an ir_asset is created, modified, ...
There is still another rare but possible possibility to have a 404 if
the transaction is rollbacked after populating the assets node cache.
In this case, we only need to clear the cache locally in case of
rollback.
Part-of: odoo/odoo#121376
Fix an issue where products with internal references containing point
or slash were not recognized as direct matches in the website searchbar.
Instead, fuzzy search was used to find similar words.
This commit addresses the problem by fixing the regular expression
used for matching words in Odoo's search functionality.
X-original-commit: dee84ff3b0ec56149780e6eefb8724f954918fd1
Part-of: odoo/odoo#124211
When generating a new page key, it was only made sure to not match
existing page keys. This leads to COW happening on existing views if the
key already existed in a view.
This commit ensures that new page keys are not existing view keys
either.
Steps to reproduce:
- Create a page named "snippets".
=> Notification was shown indicating that `website.snippets` is private.
task-3328827
closesodoo/odoo#123693
X-original-commit: e7ef9f0bfc59a468c9f883561c371367cc06c1b7
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Benoit Socias (bso) <bso@odoo.com>
Postgfresql functions can only take a max of 100 arguments by default,
when using 'jsonb_build_object' to update translations in jsonb menus
each lang adds 2 args as key value pairs. The languages should be added
in batches of 50.
closesodoo/odoo#123562
X-original-commit: 7dfbbcf91baf796174b171e46c182b8eeb2423a9
Signed-off-by: Wang Chong (cwg) <cwg@odoo.com>
While creating redirects/rewrite if the user does not enter any url in the
'Url to' field of the website module under 'Configuartion/Redirects' then during
redirection, the error 'NotFound: 404 Not Found: The requested URL was not
found on the server' will be produced.
Applying this commit will solve the issue.
sentry-4206504892
closesodoo/odoo#123391
X-original-commit: 14a850976711431f36b7f889ea9cf31b1114513d
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Saurabh Mishra (sami) <sami@odoo.com>
The 'frontend_lang' cookie is used to 'cache' the user's preferred lang.
We want to make sure that this language preference is preserved for a
longer period of time than just the life of the browser. This means that
even if you quit your browser and come back into the year, your
preferred language will be used, until you choose to remove your cookies.
The 'utm_*' cookies are used to 'track' where you are coming from on the
instance. The purpose of these cookies is to know the tracking value
to improve the overall user experience or compute the profitability of
some campaigns. Now we keep these cookies for 1 month.
closesodoo/odoo#122573
X-original-commit: 058e0abcf621796bf23d8dcaaf3b2297f632b5fd
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Jérémy Kersten <jke@odoo.com>
When a new user is created from the website, the company id was always
set to the first company of the database even if the website was the one
of another company. This flow has been already fixed if there is the
"Specific User Account" setting activated (see [this other commit]).
This commit fixes the same issue but for every case.
Steps to reproduce the issue:
- Create 2 companies A & B
- For each company, create a website linked to a different URL
- Activate 'Free sign up' for company B
- As a public user, go to website of company B
- Go to 'Sign in > Don't have an account?' and create an account
=> If as an admin you check the company of the created user, it is
company A instead of company B.
[this other commit]: https://github.com/odoo/odoo/commit/77c708c516beb322df37220634e178ba82e894c9
task-3277317
closesodoo/odoo#121834
X-original-commit: 3fbfb5301c7583583e4f46c9b4ef16e048e5800c
Signed-off-by: Benoit Socias (bso) <bso@odoo.com>
Signed-off-by: Dieleman Guillaume (gdi) <gdi@odoo.com>
Before this commit, links scrolling to an anchor with a special
character did not work and displayed a traceback. The issue was that to
check that the anchor is valid, we don't need to check that the anchor
is a valid url as we have been doing since these commits [1], [2]. But
we only need to check if the jQuery selector is valid to correctly
target the element to which the page must scroll.
Indeed, the anchor widget returns stuff like 'ok%C3%A9%25' when typing
'oké%' wich is not valid jQuery selector. It has to be encoded to
'#ok\\%C3\\%A9\\%25' to be valid and that's what this commit does.
We also changed the way to display a new anchor to the user in this
commit. Before, we showed the anchor unencoded in a notification and now
we show it encoded. That way, if the user copies the anchor from the
notification, it's the real anchor.
Also, this commit detect if the success URL of the redirect of a from is
the current page to perform a scroll to the anchor instead of a
redirect. To make this comparison, we needed to add the url code of the
language of the current page to the session info.
Also, before this commit, the page froze when we clicked on the "submit"
button of a form that redirected to an anchor that did not exist.
[1]: https://github.com/odoo/odoo/commit/0abfaeda96c2eaa868cc7fc5fa1926dfa90fc420
[2]: https://github.com/odoo/odoo/commit/b492bde6a121be1c15ed90ce0827fcfd72a12f5c
task-2172312
closesodoo/odoo#121656
X-original-commit: f09a3fc47b5fd088c3aa5112a0fac5f442d9c915
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
One of the most costly part of a page loading when the ormcache is cold
is computing the assets node, the unique identifier of an attachment to
validate whether the existing attachment is still valid with the current
version of the static files.
This operation needs to glob assets path in the filesystem,
get the modification date, check attachments, ...
Right now this task is not really optimized and can take some time
because of an excessive number of glob on the filesystem, unnecessary
exists to define absolute path, double computation of file list and
modified times when getting js and css bundle separately, ...
A list of modifications mainly discussed in the pr message are made
with this commit to speedup things.
- split css and js unique
- prepare api for an in memory glob
- change api to propagate absolute path and meta information through
`ir.asset._get_paths`-> _get_asset_paths -> `_get_asset_content` ->
`AssetsBundle`
closesodoo/odoo#121159
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
*: website, website_blog, website_event
This commit adds the TikTok social network to the other social networks
stored in DB. Thus, users of the Website application will be able to
reference their TikTok page on their websites. This commit also updates
the different templates and the social media block to integrate this new
social network.
task-3235451
Part-of: odoo/odoo#116837
Since [1] when the fuzzy search was introduced on website pages, the
filtering on most specific pages only happens at the end of the search
operation. Because of this, the search happens on pages that would be
excluded anyway, the limit might be wrongly applied and the count might
be wrong.
This commit makes the page search begin by keeping only the most
specific pages, then performing the actual search within those pages.
Steps to reproduce:
- Install `website` only and drop a search snippet.
- Search "ax".
=> No results found, but the "All results" link is displayed.
[1]: https://github.com/odoo/odoo/commit/7559626c54e34b41e1549e28276a650accec6986
task-3203794
closesodoo/odoo#120950
X-original-commit: 9725a14be22ad0f20bb6591054b60d91663ca88f
Signed-off-by: Arthur Detroux (ard) <ard@odoo.com>
Before this commit an ir_assets generated automaticaly by the web editor
will generate an url ending with ...custom.addon.bundle_name.ext
After this commit the url will start with /_custom/addon.bundle_name/...
This will make it easier to spot at immediately if it is a custom asset
and thus it is useless to apply the glob. Actually, it will fail on the
/_custom when trying to glob, making it faster.
This is mainly useful to clarify and debug but in a case with mainly
customised ir_asset for one bundle, it may have an impact on speed.
An upgrade script was created for this change.
closesodoo/odoo#120699
Related: odoo/upgrade#4636
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This commit avoid to invalidate the cache on creation of 301/302 that
have no impact on the routing_map as only 308 and 404 alter the routing
map:
- 404: remove entry from routing map
- 301/302: served as fallback later if path not found in routing map
- 308: add "alias" (`redirect_to`) in routing map
closesodoo/odoo#120500
X-original-commit: 1430644908d7d9a3119a1315389f5bf11888356a
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
== Issue ==
A business code error was detected by the internal team on our
production. The cache and assets where invalidated !WAY! too often for
the past months.
It was hard to figure but finally the error was tracked down to be
located in the assets retrieval stack of our code when a database is
accessed through multiple different domains.
In our production use case, whenever one was accessing `odoo.com/web`
after someone accessed `accounts.odoo.com/web`, the assets would be
invalidated and recomputed, again and again, whenever someone accessed
the backend on a domain after someone else did with another domain.
Obviously, on our production, this could be occuring multiple time per
minute.
Technically, this is because the "assets retrieval stack" had a mismatch
in multiple endpoint when trying to find if a current website was
involved (serving for the frontend).
Some business method were using `env.context.get('website_id')` while
others were using `env['website'].get_current_website(fallback=False)`.
From there, when the code was called without a `website_id` in the
context, `get_current_website()` would still return a `website_id` when
called from `http://odoo.com` as there is a website having its domain
set to it. `get_current_website()` is then finding it and returning it.
But it would not when the user is on `http://accounts.odoo.com`.
Since we have a custom scss override (done through our website builder,
basically an ir.asset linked to a "url type" attachment:
`/website/static/src/scss/options/colors/user_color_palette.scss`) for
our website to define the website colors which is shadowing the scss
file from disk.
So, depending of the host/domain, either the real file disk for this URL
or the ir.asset linked to our website for this URL would be fetched to
generate the bundle hash (which is basically the last modification date
of the files/attachments).
Obviously, the file on disk and the ir.assets have a different last
modification date.
The system would then consider the assets as outdated and would
regenerate it.
You can see it in the logs where the attachment id of the assets URL
would get higher and higher everytime you access the DB through another
domain.
Using `get_current_website(fallback=False)`:
- `_get_related_assets()` https://github.com/odoo/odoo/blame/30d3b97b5ece379d9ddcbceda9d12c03dc7f4a48/addons/website/models/ir_asset.py#L14
- `filter_duplicate()` https://github.com/odoo/odoo/blame/30d3b97b5ece379d9ddcbceda9d12c03dc7f4a48/addons/website/models/ir_asset.py#L41
- ..
Using `get_current_website()`:
- `_get_custom_attachment()` https://github.com/odoo/odoo/blame/30d3b97b5ece379d9ddcbceda9d12c03dc7f4a48/addons/website/models/assets.py#L162
- ..
Using `context.get('website_id')`:
- `_get_asset_url_values()` https://github.com/odoo/odoo/blame/30d3b97b5ece379d9ddcbceda9d12c03dc7f4a48/addons/website/models/ir_qweb.py#L23
- ..
== Fix ==
A fix could have been to aligned those to use the same way of retrieving
the website but it would be too fragile (definitely some other places
where the same bug is involved but not yet found).
What is done in this commit is something we wanted to do for a long time
(see [1]) but was based purely on guess and feeling rather than concrete
bug / use case, but now that we found a real use case, we will do it:
- It doesn't seems to make sense to consider the request host/domain
when we are in the backend
- Same for the forced session, those should only impact the frontend
calls.
But this seems to have too much impact in stable to be changed, as it
would require to check every caller to also check for the session if
it makes sense. This will be done in master as not really needed to
prevent the critical bug fixed here.
- When something wants to alter the backend with a website, it should
explicitely be passed in the context, which is still considered
regardless if it's a backend/frontend call.
- If something needs to consider the forced website in session in the
backend, it should explicitely check it, not relying on
`get_current_website()`.
== Step to reproduce ==
- Start a db with website installed
- Enter the website builder in edit mode and change the "Theme Colors"'s
first "Color Presets"'s background color (it is white by default).
- Set the website domain to `http://127.0.0.1:8069/`
- Go to `http://127.0.0.1:8069/web` and login
- Go to `http://127.0.0.2:8069/web` and login
- Now start refreshing those 2 pages one after each other.
Everytime you will refresh the page, it will take a very long time
(~5-10 seconds) before loading the page, and monitoring the logs will
show something about invalidating the cache and huge query count.
== Benchmark ==
For the explained "multiple domain access" case, the backend /web will
now be loaded in less than 10ms and with ~10 SQL Queries when website is
installed, while it was taking ~4 seconds and ~200 Sql Queries before
the fix.
Before the fix:
```
odoo.modules.registry: At least one model cache has been invalidated, signaling through the database
GET host1.com/web HTTP/1.1 200 - 222 0.135 3.840 <-- 222 Queries, ~4s
odoo.modules.registry: At least one model cache has been invalidated, signaling through the database
GET host2.com/web HTTP/1.1 200 - 181 0.101 3.692 <-- 181 Queries, ~4s
odoo.modules.registry: At least one model cache has been invalidated, signaling through the database
GET host1.com/web HTTP/1.1 200 - 215 0.121 3.704 <-- 215 Queries, ~4s
odoo.modules.registry: At least one model cache has been invalidated, signaling through the database
GET host2.com/web HTTP/1.1 200 - 181 0.100 3.616 <-- 181 Queries, ~4s
```
After the fix:
```
odoo.modules.registry: At least one model cache has been invalidated, signaling through the database
GET host1.com/web HTTP/1.1 200 - 101 0.043 0.353 <-- 101 Queries, ~0.3s
GET host2.com/web HTTP/1.1 200 - 11 0.004 0.007 <-- 11 Queries, ~10ms
GET host1.com/web HTTP/1.1 200 - 11 0.003 0.005 <-- 11 Queries, ~10ms
GET host2.com/web HTTP/1.1 200 - 11 0.003 0.008 <-- 11 Queries, ~10ms
```
[1]: https://github.com/odoo/odoo/pull/94161#discussion_r904780031 (Also other PR/task but couldn't find those.)
closesodoo/odoo#120364
X-original-commit: 28dd35eb3c681b630f0b3c109a7d8209f9fa42d8
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Commit [1] made sure that to be considered active, the current page URL
should have the same query strings as the ones defined in the menu URL
(if any).
But it was not fully accurate as extra query strings on the page URL
would make the menu not active even if the menu URL query strings would
be found in the visited page URL.
With this commit, when trying to determine if a website.menu is active,
we only take into account the subset of menu's query arguments.
For ex, a menu with an url of `/my-page?country=BE` should be
considered active if the request url is:
`/my-page?country=BE&utm_source=marketing-campaign&utm_medium=email`
[1]: https://github.com/odoo/odoo/commit/065ca15closesodoo/odoo#120188
X-original-commit: bc1f2c092122e171950348c3e32438e8b862f231
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Xavier Alt (xal) <xal@odoo.com>
Before this commit:
The scheduled action will repeatedly time out if the number of visitors
to remove is too high. In the case of the client, he has ~ 4 millions
inactive visitors to remove. This might happen after a bot spam attack.
After this commit:
The visitors are removed by batch of 1000, reducing the number of old
visitors with time
opw-3256349
closesodoo/odoo#119757
X-original-commit: b69917ec0e508f8354d831525c5c48ee79b5967a
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
a new context `prefetch_langs=True` allows ORM to prefetch all translations of
translated fields while fetching.
For example
The activated languages are 'fr_FR' and 'nl_NL'
In the database the value is '{"en_US": "English", "fr_FR": "French"}'::jsonb
after fetch with `prefetch_langs=True` the raw cache value will become
{'en_US': 'English', 'fr_FR': 'French', 'nl_NL': 'English'}
closesodoo/odoo#116947
Signed-off-by: Raphael Collet <rco@odoo.com>
Rémy Voet (ryv)