Before this commit if a computed field is already in cache, but not its
dependencies, `fetch` would fetch those dependencies.
This commit ensures that fetch checks first if a computed is in cache
before fetching its dependencies.
This commit also follows dependencies of computed fields, if they depend
on other computed fields.
Finally, this commit consolidates `fetch` and `search_fetch`:
they should use the same heuristics to know which fields to fetch.
closesodoo/odoo#120001
X-original-commit: 6b680c463956f929db10d4c3058c36112a67e674
Signed-off-by: Vincent Schippefilt (vsc) <vsc@odoo.com>
Signed-off-by: Raphael Collet <rco@odoo.com>
Co-authored-by: rco-odoo <rco@odoo.com>
There was two issues regarding the slides public views counter
1. If the `public_views` is set to `NULL` in database,
`increment_fields_skiplock` wasn't properly incrementing the count.
Indeed, in SQL, doing NULL + 1 returns NULL
```sql
16.0=# SELECT NULL + 1;
?column?
----------
(1 row)
```
To have the result we expect, COALESCE must be used
```sql
16.0=# SELECT COALESCE(NULL, 0) + 1;
?column?
----------
1
(1 row)
```
2. There is a mechanism, using the session,
supposed to prevent incrementing the public views
counter when a same user visits multiple times the same slide.
However, since 84d17e57e8
the visited slide was never actually added in the session,
because it was adding the slide id in a copy of the set
in session rather than adding in the set from the session.
Or, as this commit does, to re-assign the new set in the session.
closesodoo/odoo#119370
X-original-commit: fa5962d6f08979017842985004b3ec43416ee181
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Calling `fetch` with computed fields will also check that the dependencies
of those fields are fetched, which is good.
This commit adds a check that verifies that all the dependencies are already
fetched before re-fetching them.
closesodoo/odoo#119247
X-original-commit: 7ab724450c9aa29ad6c1739cff1e928d79666dac
Signed-off-by: Rémy Voet <ryv@odoo.com>
Signed-off-by: Vincent Schippefilt (vsc) <vsc@odoo.com>
Calling `fetch` with 'id' in the `fields_name`, will always generate
SQL query even if all requested field values are in the cache.
This is because we also look for values in the 'id' field cache,
but we don't ever fill the cache for `Id` fields.
closesodoo/odoo#119107
X-original-commit: 3ba8d0e5e57e1358cc8958abf511d514515eccb2
Signed-off-by: Vincent Schippefilt (vsc) <vsc@odoo.com>
Make the public `read_group` depends of its private method `_read_group`
refactored to match the backend usage.
We try to keep the public API similar for this first part of the
rafactor, but there are still some API change:
- We cannot order by `id` anymore.
- The display_name of many2x group values are not lazy anymore.
Part-of: odoo/odoo#110737
This fulfills the goal of searching and fetching fields in a single SQL
query. We introduce the new method search_fetch() for that purpose.
Also introduce method fetch() to fetch some fields for a recordset if
they are not in cache yet.
The call graph is as follows:
search() calls search_fetch()
search_read() calls search_fetch() and _read_format()
read() calls fetch() and _read_format()
search_count() calls _search()
search_fetch() calls _search() and _fetch_query()
fetch() calls _search() and _fetch_query()
The methods _search() and _fetch_query() are usually the ones to
override to implement business-specific logic. The method _search()
returns a Query object to retrieve the records that satisfy the given
domain and are accessible for reading. The method _fetch_query() uses a
Query object to retrieve fields from the database and store them in
cache.
Also use search_fetch() to save one query in search_read() and the
reading of one2many fields.
Part-of: odoo/odoo#112126
* = blog, forum, slides
For performance reasons, allow to increment multiple fields of the same record
within the same query.
With python tests.
Task-2663320
Part of odoo/odoo#79615
The idea is to avoid flushing the fields to fetch in method _read().
This delays UPDATE queries, and makes the prefetching mechanism simpler
and more effective.
We do this by not overwriting the cache values by the values fetched
from database. This simple idea allows to fetch more fields and more
records without having to care about pending computations and updates.
But it requires the cache consistency to be much more strict, because
nothing will "fix" the cache inconsistencies "by chance". And it also
requires pending updates to be present in cache.
Part-of: odoo/odoo#66938
Before when we do `reserved(records)`, it iterates in a reverse
order of `records`. Unfortunately, the `__reserved__` method don't
exist explicitly in BaseModel then Python fallback on
its own implementation using `__getitem__` and `__len__` (coming from Sequence): https://github.com/python/cpython/blob/3.10/Lib/_collections_abc.py#L1047-L1049
Because it uses __getitem__, it breaks the prefetch of the recordset.
Example:
-------------
```
partners = self.env['res.partner'].browse(1, 2, 3, 4, 5)
for partner in reversed(partners):
partner.name
```
will generate 5 SQL requests to fetch data (one by record)
Then create our own `__reversed__` and handle the prefetch correctly
(like `__iter__`). Now in the example it will correctly generate only
1 SQL request because of the prefetch.
task-2687953
closesodoo/odoo#79622
Signed-off-by: Raphael Collet <rco@odoo.com>
Co-authored-by: rco-odoo <rco@odoo.com>
The processing of the LINK command in a one2many should fail when the
line being linked does not exist. However, there is one case where it
should not fail: when that line existed and was linked before applying
the commands. That use-case may seem strange, but it actually exists:
deleting a line in a sales order automatically deletes the corresponding
reward lines.
closesodoo/odoo#78318
X-original-commit: 4bfe1b5cab10d3171d386d76799a7d7729f49154
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The license is missing in most enterprise manifest so
the decision was taken to make it explicit in all cases.
When not defined, a warning will be triggered starting from
14.0 when falling back on the default LGPL-3.
closesodoo/odoo#74245
Related: odoo/design-themes#48
Related: odoo/enterprise#19862
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Writing on x2many field should be done last, because deleting x2many
lines causes pending computations and updates to be flushed. Writing on
column fields after that inevitably adds extra update queries.
We introduce an attribute `write_sequence` on fields to order fields for
write. The prescribed order is: all fields except monetary and x2many,
monetary fields, x2many fields.
closesodoo/odoo#65959
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The goal of this change is to simplify the code managing `create_date`
and `write_date` in methods `create()` and `write()`, and also to remove
weird behaviors caused by the way those fields were updated.
Assume we update a simple field on a record. This adds pending updates
for the field and `write_date`. However, the value of `write_date` is
not known yet: it will be updated as `NOW() AT TIME ZONE 'UTC'` in SQL.
So `write_date` is actually given a dummy value in pending updates, and
it is invalidated from cache, until its value is flushed to the database
and fetched again.
Now assume we access another field on the record, and that field is not
in cache. The prefetching mechanism will read all column fields,
including `write_date`, and flush them first.
# this adds pending updates foo: 42, write_uid: 1, write_date: False
record.foo = 42
# assume 'bar' is not in cache; this prefetches all column fields,
# which flushes the pending updates above before reading them back
result = record.bar
We can avoid flushing pending updates if the values read from database
do not overwrite existing values in cache. If you assume that the value
of a pending update is in cache (in the example, `foo: 42`), you don't
need to flush the corresponding field. Indeed, the value of `foo` will
remain 42 in cache, whatever its value in the database. This assumption
(pending updates are in cache) is true for all fields *except* for
`write_date`: it is invalidated from cache, and given a dummy value in
pending updates. This branch actually makes this assumption true for
all fields. The avoidance of flushing pending updates will be done in
another commit.
In order to directly assign `write_date` its value, we use a cache for
the value `NOW() AT TIME ZONE 'UTC'` from the database. This costs at
most one query per transaction, and potentially saves a few queries.
Co-authored-by: Victor Feyens <vfe@odoo.com>
Adapt some query counts to their optimal value, in order to measure the
effect of the following commits on queries.
In module test_performance, some query counts were actually not correct:
the initial flush() done by the context manager assertQueryCount() may
prefetch some data to the cache, and that prefetching is not accounted
for in the query count. This is very true when assertQueryCount() is
preceded by a cache invalidation. We have to move the invalidation
inside the context manager, so that the prefetching is now counted.
Make the method `_search` return a `Query` object, and make that object
generate a subquery when used on the right-hand side of a condition.
closesodoo/odoo#52403
Related: odoo/enterprise#10945
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Before this commit, calling `mapped()` on a relational field of a single
record inside a for loop does not correctly propagate the prefetch ids
from the bigger recordset down to the record inside the loop:
[rec.line_ids.mapped('name') for rec in recs]
To be more precise, `recs` correctly propagates prefetching information
down to `rec.line_ids`, but method `mapped()` does not pass it along to
the record that triggers the prefetching.
This resulted in one query per record in `recs`, so if `recs` were a
1000 records recordset, at least 1000 queries would be necessary.
With this commit, the `mapped()` function correctly propagates the
`_prefetch_ids` of the larger recordset (`rec.line_ids`) so that the
prefetching works properly and the 1000 queries are brought down to 1.
This commit is a followup on #42611.
closesodoo/odoo#54565
X-original-commit: 0e97053fee36c0c76b70cb94be5340c3144b178b
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
Consider a context-dependent field, and successively access it on a
recordset with different contexts. On the first context, the field is
correctly computed in batch. After that, the field is always computed
one by one.
The bug is in the method that determines which records in a given set
have no value in cache. On the first context, the cache is empty for
the field, so all records are returned. After that, the method
considers that all records have a value in cache: they do, but for
another context key! Simply using the context key when looking up the
cache fixes the issue.
closesodoo/odoo#52360
X-original-commit: 35d69589d9b43afe6c6fc9779458323f7180153e
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Co-authored-by: Xavier-Do <xdo@odoo.com>
Also non-browser jsonrpc (as it goes through a similar process): for
internal performance reasons, name_search and read_group have been
converted to a *lazy* name_get, so the "display name" is not
unnecessarily computed.
However this is an issue for the RPC endpoints (/xmlrpc and /jsonrpc)
as they have no support for `lazy` and thus tend to blow up and / or
do the wrong thing when trying to output a lazy:
* xmlrpc has no way to handle lazy at all and straight blows up
* jsonrpc falls back to `json_default` so they try to stringify the
lazy, which might have worked except
*Problematically* both endpoints delegate the actual work to
`dispatch_rpc` which handles dispatching between various services and
ultimately creates a *new* cursor before calling model
methods (`object` service and `execute`/`execute_kw`).
This means by the time the result is serialized to be output, the
lazy's cursor has long been closed, and thus any access to an
unevaluated `lazy` errors out when trying to fetch the underlying
item.
This also means we can't just add a hook to serialize the lazy
in the xmlrpc marshaller, though we do have to do that. We *also* (for
both xmlrpc and jsonrpc) have to force evluation of lazy values before
our cursor is closed, meaning it has to be done right after the method
is invoked, iterating the entire response.
Related to task 2170343
closesodoo/odoo#49286
X-original-commit: e2b5a359c1d5eccbe725c1c3169b4130d7bca49b
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When the `select` returns nothing more than ids that are already known, there is
no need to make it at all.
This removes one query from `_read` every time it has to fetch only fields that
are stored in a different table (o2m, ...), which happens all the time when
reading a stored field first (triggering prefetch) and then reading a o2m.
The query that is now removed was used to check access rules, but the trick is
to use `check_access_rule` to verify the rules in python instead.
Part of task-2061122
closesodoo/odoo#36263
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Query counts weren't adapted to recent performance changes.
This commit updates the different query counts to make sure
any commit changing the query counts knows it and does it on purpose.
Some query counts may vary between community and enterprise
and therefore have a higher value than needed in community version.
closesodoo/odoo#43202
Related: odoo/enterprise#7682
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Before this commit, creating a record of a model with at least one
property/company_dependent field would trigger the field's inverse
method regardless of whether an actual value was being passed in for the
field or not.
This means that if no value was given or if the value was the same
as the default value, we would waste precious time in the property
field's inverse method.
With this commit, we do not call the inverse method if:
1) there is no value for it in the vals dict (use default)
2) the value in the vals dict is the same as the default value
As an example, when creating a res.partner record while having account
installed (which introduces a property field in res.partner), the
creation time goes from 24ms to 14ms.
closesodoo/odoo#36267
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
This branch is the combination of several optimizations in the ORM:
* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;
* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;
* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);
* make method `modified` take advantage of inverse fields to inverse
dependencies;
* filter records by evaluating a domain on records in Python;
* a computed field with `readonly=False` behaves like a normal field
with an onchange method;
* computed fields are computed in superuser mode by default.
Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.
closesodoo/odoo#35659
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
After 12744bc81ebdaa1671a953c14ba54c637d6a9255, x2m operations are
fully batched, and deletions are processed last when the operations are
flushed, regardless of the order in which they were specified by the
write() or create() call.
This carries a risk of violating (non-deferred) unique SQL constraints,
when the operations for deleting previous lines and re-creating new
ones are processed in the same batch.
This patch executes the deletions before other operations during a
flush, which should be safer with regard to SQL constraints.
An extra constraint is added in test_performance.line to simulate this
corner case, then covered by an extra unit test.
Another unrelated test had to be altered to avoid violating the
new constraint.
This can be reproduced easily by upgrading the `project` module in master,
due to the unique constraint[1] on `ir.actions.act_window.view`, that
gets violated when processing the batch write on this o2m[2].
[1] https://github.com/odoo/odoo/blob/ccc42f16/odoo/addons/base/models/ir_actions.py#L288-L289
[2] https://github.com/odoo/odoo/blob/ccc42f16/addons/project/views/project_views.xml#L394-L396closesodoo/odoo#28314
Purpose of this commit is to give description more "business oriented"
because those descriptions appears in Odoo Studio which is supposed to be used by end users, not only by developers.
Related Task ID : 37311
When a record fields are prefetched, the currently accessed record is
prefetched as well as PREFETCH_MAX (=1000) minus currenly accessed
records.
Since 439fa826 the id field was thought as "prefetched" but was not,
which caused that in an onchange, we would have a prefetching as follow
for 1200 records:
- get records 1-1000
- get record 1001 (1001-1200 INTERSECTION 1-999 (id) + 1001 (current))
- get record 1002 (1002-1200 INTERSECTION 1-999 (id) + 1002 (current))
- get record 1003 (1003-1200 INTERSECTION 1-999 (id) + 1003 (current))
- ...
- get record 1200 (1200 INTERSECTION 1-999 (id) + 1200 (current))
So we would do 201 queries instead of 2 when prefetching the records.
The added test without this change failed the query count with:
"AssertionError: 284 not less than or equal to 5 : admin"
opw-1837548
opw-1837552
closes#24326
