Website: the context install_filename='dummy' is used to prevent
arch_updated from becoming True while updating translations of
ir_ui_view.arch_db (if arch_update becomes True, test_inherit_specific
fails)
Fuzzy search for jsonb translated fields has been adapted in the case of
website. It may require some refactoring later.
* im_livechat, test_event_full, website_blog, website_crm,
website_event, website_event_track, website_event_track_quiz,
webite_livechat, website_sale
There is 6 main changes in this commit:
1. Using raw SQL Upsert instead of the ORM methods. While raw SQL should
generally be avoided, it makes sense for such a low level behavior which
is impacting every flows.
Indeed, tracking visitors is a generic behavior done on all pages and
controllers. It is important to optimize it to reduce processing time
and SQL Queries.
Benchmark of that change alone:
> Rendering a tracked page improves from ~19.5ms to ~17ms (using `ab`
with 1000 loop) and the requests involved in the tracking process are
reduced from 8 SQL Queries to 3:
- 1 request to upsert the visitor
- 1 request to fetch the visitor data
- 1 request to add the tracking record
2. Adding in that upsert query the `visitor.track` insert, creating both
records in one go, bringing the query count from 3 to 2.
3. Refactoring of the `parent_id` behavior that was introduced in stable
with [1]. The purpose was to keep track of multiple visitor linked to a
same user to merge the tracking together. Especially useful for tracking
a same visitor on different devices (when logged in).
Only one visitor was kept as active, others would be archived and their
tracks would be set/moved to the main partner.
Removing those duplicate visitor was not possible because those archived
duplicated visitor were holding the devices notification push token.
Since [2], those token were moved to their own table, all related to the
main visitor.
We can then now safely remove those duplicate visitors after merging
their track to the main visitor. Thus, the `parent_id` field is no more
useful. Removing it removes a layer of complexity.
Note that thanks to this part, the `active` field can also be removed.
4. Deeper functionnal change, inspired from Plausible: The access_token
is no more stored in a cookie but is the result of a hashing method
based on <IP Adress, User Agent>.
The reason behind that change is that, in an upcoming refactoring,
sessions won't be stored anymore unless absolutely needed (login, add to
cart..). It will also ship a no cookies policy, trying to get rid of all
cookies.
This change is bringing some functional changes:
- Since the IP is included in the hash to generate the token, it means
that:
A. If an anonymous user switch IP (eg from 4G to wifi), it is
considered as a new visitor.
B. If 2 anonymous users with the exact same user agent (same browser,
same browser version, same exact os or phone) are on the same IP,
those will be considered as the same visitor.
- Since the request host is not included in the hash, it means that
visiting a DB from 2 differents URLs (domain and/or ip) on the same
device and same browser will result in a shared visitor.
It shouldn't imply any issue as this is A. not wrong and B. mostly
used for tests.
As all this is only related to non logged in user, it shouldn't be a
real issue as anonymous visitors are not supposed to be meant to be
business critical, even if we use them for "a bit more" than simple
analytics data.
5. The access_token is now replaced by the partner_id once the user logs
in, so:
- We don't need to either search on the partner_id field or the
access_token field (depending if the user is logged in or not), we can
only use the access_token row/field to do both.
- On logout, everything works out of the box as the access_token will be
regenerated since there is no partner_id anymore.
- On login, if an access_token matches the user's partner_id, that
visitor is returned.
If there is no such token, a new visitor is created for that partner_id.
In both 2 cases, tracks are moved to that visitor and the anonymous
visitor is removed.
- We can remove the code that was in charge of checking if the
access_token / visitor cookie was wrong (coming from another user eg,
different user login on same device). Indeed, such collision is not
possible anymore as the access_token automatically match the logged in
user.
- We can remove the code that was in charge of checking if the
access_token / visitor cookie was wrong (coming from a logged in user
while the current visitor is not loggedin). Such collision is not
possible anymore as the access_token is (re)generated as an anonymous
token (hash) when not logged in.
6. There is no more check to prevent a track to be created if there was
already a track for that URL in the last 30 minutes.
While this can easily be re-introduced (one CTE on the upsert), it was
adding ~100ms (from ~20 to ~110ms) to the request on a big database as
Odoo where there is ~100 millions tracks and ~100 millions visitors.
It has been validated that it was not a real issue as it is not
fundamentally wrong. If a visitor visited 20 times a product or a
specific page in that short amount of time, you might want to know that
because the user is most likely interested by it.
Changes (1+2), 3, (4+5) and 6 are all independant from each other and
could have existed on their own.
[1]: https://github.com/odoo/odoo/commit/c6b8a44b970a46dcd87a4e2cb1ad52fa340b209f
[2]: https://github.com/odoo/enterprise/pull/16781/commits/f75090fe8b42484e89e933976e8441d2f5eb9415
task-2867045
closesodoo/odoo#87857
Related: odoo/enterprise#28004
Related: odoo/upgrade#3566
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Using lazy values allows you to not do query when the content is not
displayed or if it is already cached. Adding `t-cache` only reduces
render time, but combining it with lazy values saves browses, computes
and query.
Thus for the `/shop` page the page is displayed in 60ms (before: 250ms).
Part-of: odoo/odoo#88276
*: test_website, website_blog (test perfs)
An error regarding tests was not catch during httpocalypse [1] review.
The test query counts were lowered in website by 2 everywhere, but it
shouldn't have as there were no SQL perf improvement from this
refactoring regarding website.
The query count actually decreased because the test-only SQL Savepoints
got divided by 2, probably thanks to a low level test improvement in
the PR.
But outside of tests, there were no SQL Query gained.
This should have been reflected in the `EXTRA_REQUEST` variable
instead.
Before:
Savepoint > Rollback > Savepoint > [Actual SQL Queries] > Rollback
After:
Savepoint > [Actual SQL Queries] > Rollback
It is important to fix it because:
1. (Minor) There is a de-sync between the query count in the test and
the actual query count (in the logs outside of tests), making it
hard to figure.
2. (Major) Some controller got then wrongly lowered to 0 query counts
instead of 2. An incoming PR in master would then make those tests
expects negative query counts, which doesn't make any sense.
[1]: https://github.com/odoo/odoo/pull/78857
Note that the exact commit can't be found by bisect as those don't work
on their own, DB can't be started due to circular import.
Part-of: odoo/odoo#87856
There were inconsistencies in the calls to `_render`.
* the view context could contain information that misled developers.
Indeed, the context and value of the view are not supposed to be found
in the rendering. Thus by calling `ir.qweb` with the name of the
template, we ensure that there is no unwanted information and in
addition the cache key is that of the name of the template which saves
a query.
* the context used for rendering was modified by a method on
`ir.ui.view`, except this is not information used by this model. There
is now a `_prepare_environment` method residing on `ir.qweb`. This
method allows to modify the value dictionary as well as the context in
which the rendering will be done. This preparation of the data as well
as my security check is done only once per rendering. This also saves
some queries
* Freeze options for rendering were inconsistent. It could be that
options on which rendering depends were not part of the cache key. Thus,
depending on the user who generated the generation of the rendering
function, there was or was not information in the template. For example
for automatic branding. This is no longer possible, because it is the
context that is used. The options serving as a cache key are only
recorded for information (for the profiling system for example). A
simplification of the `ir.qweb.field` models could be made.
The report rendering and call `ir.qweb` instead of `ir.ui.view`.
Part-of: odoo/odoo#85110
Issue
-----
When website is installed, the rendering of template uses a side effect
of the ORM cache (cache shared between sudoed env vs non-sudoed env) and
the fields prefetching feature to work correctly.
The `self.visibility` in (`_handle_visibility`, website/ir_ui_view.py)
is done in sudo mode, then it will fetch all prefetchable fields and put
them in the cache (that will be read in non-sudo mode in the render of
the template). Another example of issue related to this:
https://github.com/odoo/odoo/pull/83341.
Because of this, the fields of mixin `website.seo.metadata` were forced
to be prefetchable (the default for translate is to be not prefetchable
since https://github.com/odoo/odoo/pull/82896), which causes a useless
LEFT JOIN on "ir_translation" in most of business flow.
Fix
---
Remove the `prefetch=True` on mixin fields, and add a extra read to fill
the cache in case of website rendering. It also allows to read these
fields at the same time.
Part-of: odoo/odoo#85220
* website, website_blog (perf)
If the source values is only composed of empty elements (list of `None`
values), don't try to translate it, as there is nothing to translate
anyway.
While the translation method being called (xml_translate,
html_translate) won't be costly as they will return instantly if the
provided value is falsy, there is still an SQL Query made to get the
callable translation method, which we won't actually use.
This commit avoids that query if we already know we won't do anything
with the translation method.
task-2774979
X-original-commit: e1eb5a6e1610a9550a59ba229a83b9e70e10aa88
Part-of: odoo/odoo#85456
* test_website
The `nocache` URL param was introduced with [1] which allows to bypass
the cache when serving a page.
Bypassing that cache, despite not being the most common flow, was done
in the perf tests since that commit. Ultimately, it was not testing
real use cases anymore (despite still being useful as it would still
prevent perf killer feature to be merged).
This commit ensures the perf tests are also tested with cache.
[1]: https://github.com/odoo/odoo/commit/cbf8d3b3047c94a2ff7fb197bbae92a402e0b58b
task-2774979
closesodoo/odoo#85261
X-original-commit: 5d44d890021cfc7767daa1f366e636288e84f1f6
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
This commit is the 14th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
* `request.uid = x` => `request.update_env(user=x)`.
* `request.context = x` => `request.update_env(context=x)`.
* `request.context = dict(request.context, x=y)`
=> `request.update_context(x=y)`.
* `request.cr = None` => `request.cr.close()`.
* `http.mono_db()` => `request.db`.
* `http.dispatch_rpc()` => `service.dispatch_rpc()`.
* `@service.model.check` => `service.model.retrying()`.
* `request.endpoint`
=> `env['ir.http']._match(request.httprequest.path)[0].endpoint`.
* `request.routing_iteration `=> `removed`.
* `request.jsonrequest` => `request.dispatcher.jsonrequest`.
Note that `request.params` is now set much later in the process. If you
are in a situation where you values from the query string or the
http body you can use `request.get_http_params()`.
Note that using the new `request.future_response`, it is possible to
add headers and cookies on the response object before the response
object is initialized. Please note that headers/cookies saved on
the future response will NOT be injected in case of error.
PR: odoo#78857
Task: 2571224
The field prefetching mechanism was poorly customizable. Before this,
we could only tell if a field was prefetched with other fields or not at
all. We have no way to inform the framework, like: "When I need data of
that field, prefetch these other fields, which are likely be used in the
same transaction".
From now on, the `prefetch` attribute is used as a grouping key for
prefetching fields. When a field is fetched, all the fields with the
same value for `prefetch` are taken for prefetching.
For example, consider a small set of fields that are rarely used, except
for one flow A using them. You want to prefetch those fields only in
the flow A, and you want to fetch them in a single query. With the new
feature, simply set `prefetch=A` for some string `A` on those fields,
and they will be grouped for prefetching.
Part-of: odoo/odoo#83818
Issue
-----
When website is installed, the rendering of template uses a side effect
of the ORM cache (cache shared between sudoed env vs non-sudoed env) and
the fields prefetching feature to work correctly.
The `self.visibility` in (`_handle_visibility`, website/ir_ui_view.py)
is done in sudo mode, then it will fetch all prefetchable fields and put
them in the cache (that will be read in non-sudo mode in the render of
the template). Another example of issue related to this:
https://github.com/odoo/odoo/pull/83341.
Because of this, the fields of mixin `website.seo.metadata` were forced
to be prefetchable (the default for translate is to be not prefetchable
since https://github.com/odoo/odoo/pull/82896), which causes a useless
LEFT JOIN on "ir_translation" in most of business flow.
Fix
---
Remove the `prefetch=True` on mixin fields, and add a extra read to fill
the cache in case of website rendering. It also allows to read these
fields at the same time.
Part-of: odoo/odoo#83818
Issue
-----
Via the field prefetch mechanism, when we need a value of one field
(not in cache of course), the ORM will prefetch all fields
(which has the attribute to `prefetch=True`, the default value of this
attribute is `True`) for all record ids in `_prefetch_ids`.
Then, for each translate fields (where translate is not a callable)
the ORM need to make a `LEFT JOIN` on the `ir_translation` to fetch the
translated value. For big model, it leads to a simple `SELECT` with
several `LEFT JOIN` on ir_translation but each LEFT JOIN have a cost
in the planner time (a small cost in the execution time) of PostgreSQL.
By example, for `product.template` (stock/sale/purchase installed),
there are 6 LEFT JOIN to get all translated fields (5 of this
fields are rarely used).
Proposed solution
-----------------
Deactivate the prefetch by default for all translate fields expect if
this field is the `_rec_name` of the model (which is more likely to
be used).
In the example on the `product.template`:
Without prefetching the translated fields, there is only one LEFT JOIN
(the name, which is translated but is the `_rec_name` of the model).
With the 6 translated fields to fetch, the
query takes 5 ms to plan and 2 ms to execute VS with 1 translate field,
it 1 ms to plan and 1.5 ms to execute.
Side change note
----------------
- All translate of fields of `website.seo.metadata` should be prefetch
to avoid lot of website errors (it is because, website put in cache data
in sudo before reading it without sudo)
- `description` (`mail.message.subtype`), `subject` (`mail.template`),
`body_html` (`mail.template`) should be prefetch to avoid lot of extra
query from mail module.
- `vat_label` (`res.country`) should be prefetch to avoid a extra query
for each website page.
- Increase some queryCount (when it is legit, due to `subtitle` of
`blog_post` or `description` of `event.type.ticket`, etc)
task-2738029
closesodoo/odoo#82896
Signed-off-by: Raphael Collet <rco@odoo.com>
Disabled livechat in blog query count
if website_livechat is installed before website_blog, the get_livechat_channel_info add
unrelated query for the current test. So we disabled it.
Courstesy of @Xavier-Do for analyse
closesodoo/odoo#58044
X-original-commit: 86f065e3571a1339798501156804fda6c57c6989
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Related to digest update in 13.4 (a595ebfca5ab7041224c661f404138612257bdda).
QUERY COUNT IN BLOG
Local installation of blog still makes those tests use 27 or 28 queries. On
runbot it grows until 32.
Due to dependency change, installation order changes. It may have an effect on
query count in website loading some pages in blog. This is worth checking why
queries are added.
Digest module itself has nothing to do with website. It could either be a side
effect of some computed field, either an installation order change adding more
modules before installing blog, with side-effect of adding queries due to
another module.
As without this fix installing website is not possible, let us merge this one
and investigate further soon.
closesodoo/odoo#55835
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Seems like randomly one request disappear. So 28 vs 29
Closes the eyes, while it is not 30 vs 29 ;)
closesodoo/odoo#50667
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Http routing and websits module will already do it later and handle the case.
The only ModelConverter used that will be impacted is:
/web_editor/attachment/<model("ir.attachment"):attachment>
that is a json controller called with eisting data
task-2211013
Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
We use `author_id` (res.partner record) to instanciate our frontend widget
address so it can be changed through left panel in a searchable list.
This require a read on res.partner to get his name, even for visitors.
Using a related will have the same result for visitors while it won't need to
query res.partner table.
task-2211013
Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
Without this commit, if `lang_code` or `env.context.get('lang')` was set, we
would still access `env.user.company_id.partner_id.lang` to fill the loop array
even if we break and don't need that value.
Switching from loop to if conditions prevents that.
task-2211013
Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
Without this commit, a simple search like:
`env['blog.tag'].search([('website_id', 'in', (1, False))])`
would generate 4 SQL queries instead of 1.
3 requests would be done on `blog.blog`, see `right_ids` at ORM level.
task-2211013
Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
`tags_list()` is called in loops to get nicely formatted URL in blog tag button
but doing an exists here will do everytime an SQL query regardless of the
cache.
We can safely remove the call to `exists()` as the `tag_ids` parameter is
coming from the controller `active_tag_ids` key which is already ensuring the
exists condition.
It make also the number of requests determinists for test with a random number
of tags. From between 200 ~ 300 queries (depending number of tags)-> 116
task-2211013
Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
Most of the website rendering process depends of customize_show views which are
part of a configuration setup.
Once it has been setup, it will most likely never change afterward.
It makes sense to orm cache those infos to avoid having to query ir.ui.view
table everytime we render something.
Note that just store basic and small infos used for customize_show case.
task-2211013
Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
Chong Wang (cwg)