For historical reasons, a field in the read_group grouby parameter was required to be in the list of fields parameter without any necessity. This is not the case anymore. A byproduct of this commit is that it fixes a bug in pivot view introduced earlier in [47395c6].
d28f8f704f carefully clears only the
relevant bit of the cache (only the top-level object which we're
exporting), however the commit did not considert that _export_rows
calls itself recursively for relational fields, and thus when
exporting relational fields the relation would become the new
top-level and get cache-cleared on every iteration, significantly
slowing down cases where such records are shared and would need to be
re-fetched every iteration, even more so if they're somewhat/somehow
expensive.
OPW-1833545
OPW-1836361
While c059e7b93c64b9dd9bc96eeb6ee5cadf significantly lowers the cost
of generating xids, a safety invalidate cache can regress cases
significantly e.g. 18s -> 430s (7mn) for some recursive exports of
xids as the cache would be completely cleared *after each record*
requiring complete fetching & recomputation of the following records.
Fix by only clearing the cache if we've actually had to create
xids *and* only for ir.model.data records.
It's possible that clearing the cache isn't even necessary at all (?),
check with @rco-odoo
OPW-1835226
* add UUIDs to XID sections (4 bytes / 8 hex digits) so it's not
necessary to handle collisions & add a fallback generator
* use COPY for performances over executemany: executemany just
performs an implicit loop on all statements (in psycopg2)
* ~pure SQL was possible:
INSERT INTO ir_model_data (module, model, name, res_id)
SELECT
'__export__',
'{model}',
'{table}_' || A.id || '_' || uuid_generate_v4(),
A.id
FROM {table} A
LEFT JOIN ir_model_data B
ON A.id = B.res_id AND B.model = %s
WHERE B.res_id IS NULL;
but would have required installing pg extensions (problematic
especially in stable) and performances are about the same as
the COPY version
task 36343
Fixes#22493
The caller of `read_group` can now provide the aggregating function to use for
a given field:
# set aggregating operator for fields 'foo' and 'bar'
model.read_group(domain, ['foo:sum', 'bar:avg'], ...)
One can also aggregate the same field several times, by giving a specific
output name for each:
# aggregate 'foo' with both 'min' and 'max'
model.read_group(domain, ['foomin:min(foo)', 'foomax:max(foo)'], ...)
This fix avoids wild prefetching when serializing
onchange results.
We fell on a strange situation in method `onchange`, where
some data was missing from cache when serializing the
result. In that case, the prefetching overwrites so
many stuff in cache that the results of the onchanges
are lost.
Don't let that happen.
This is a backport of 5f660cd, with rco-odoo's blessing.
A use case this commit solve is
- Install the sale app.
- Create new quotation and assign new partner
- From inline editable listview add a new order line
(created the product inline) then directly save.
- Click on the order line (NOT IN EDIT MODE). This opens
the order line in a popup.
- Close the popup.
- Click edit again.
- Now change the order line description.
When unfocusing order line field, the description is reset
to previous one.
Closes#23276Closes#23277
Issue: when the inverse method writes on a dependency of the first record, it
invalidates the cache of the field for the other records... Solution: simply
evaluate the inverse method one record at a time.
Issue: when the inverse method writes on a dependency of the first record, it
invalidates the cache of the field for the other records... Solution: simply
evaluate the inverse method one record at a time.
Configurations uses an onchange to populate the fields visible to the
user. However, the related fields are by default called sudo, when it is
an onchange, there may be a missmatch in the cache, the cache used being
empty, there is no value to return (cache fix is not currently possible).
Before this fix we must use 'related_sudo=False' to use the good cache but
it's an inconsistent fix because in some case we must use sudo to avoid
access error.
opw-1823363
This replaces the former modified preorder tree traversal (MPTT) with the
fields `parent_left`/`parent_right`. Each record is associated to a string
`parent_path`, that represents the path from its root node to itself. The path
is made of the node ids suffixed with a slash:
a node | id | parent_path
/ \ a | 42 | 42/
... b b | 63 | 42/63/
/ \ c | 84 | 42/63/84/
c d d | 85 | 42/63/85/
This field provides an efficient implementation for parent_of/child_of queries:
the nodes in the subtree of record are the ones where `parent_path` starts with
the `parent_path` of record. It is also more efficient to maintain than the
MPTT fields, and less sensitive to concurrent updates, because the value of
`parent_path` does not depend on sibling nodes.
Overly attached cache's overhead turns out problematic. Clearing the
cache after batches of records keeps the cache overhead low and does
not change performances.
Explanations:
env.cache is 3 levels of maps {field: record_id: {env: value}}, where
env can be either an Environment or a pair (cr, uid) depending on the
field's dependency on the context or not.
This can be an issue when the current request loads many fields in an
enormous number of records in a single environment. The (PaaS) case
here was the export of a res.partner field from 36358 records in a single
environment[0]:
* prefetch expanded the single field to 68, leading the base `cache`
to have 68 entries. getsizeof(d<len=68>) == 3360 (3kB, which we will
soon see we can ignore entirely).
* *each* of these entries would hold a map of 36358
records. getsizeof(d<len=36358>) = 3146016 (3MB), 68 times = 213MB.
* finally each record entry is also a {(cr, uid): value}, here the
dicts have a single entry which makes them 280B, and their key is a
2-tuple "worth" 72B, or 352B/record/field, or 352 * 36358 * 68 ~
870MB[1].
For a total of ~1GB, which is roughly the issue we can observe.
Future possibilities: extract the cache-clearing iterator to be more generic
and available on BaseModel directly? Or even make the default iterator
batched & cache-clearing?
[0] note that sys.getsizeof only provides the size of the object it's
called on, it is not recursive
[1] slightly more in actuality as there's some variation between the
leaves depending on the field type e.g. M2O values are a 1-tuple
adding 60B, ...
Fixes#22475
If a record is deleted *and* has property fields, the deletion of the
property fields will trigger the various recomputes before the record
itself has actually been deleted, and thus stuff which depend(ed) on
that record will be recomputed under the assumption that the record
still exists, which probably will not work.
Under the assumption that the same issue would occur when deleting
more than one batch of records (unlinking the data, values or
attachment would also trigger a recompute before the proper end of the
function and could lead to incoherence, e.g. some of the records being
deleted being visible to recomputes), lift the entire effective body
of the unlink into a norecompute context, to ensure that only the very
final recompute (from the top-level unlink) is actually triggered.
opw-807036
Before this commit, if the first exported field of an M2M is `id` (the
xid) the entire M2M is folded into a single cell with comma-separated
xids and any following field is ignored. If `id` is any but the first
field, the export behaves normally (with the m2m exported as a "table"
inside the parent record).
This behaviour makes sense for import-compatible exports where the id
is the only thing which can be exported anyway, but it is troublesome
outside of that mode as the behaviour of m2m under export becomes
incoherent/unpredictable (ish) as it depends on the position of the
m2m's `id` in the exports list.
Change it so we only perform folding in import-compatible mode (which
is the default for backwards compatibility with e.g. API calling
export_data directly & the like).
opw-813361
Fixes#22600
We fell on a strange situation in method `onchange`, where some data was
missing from cache when serializing the result. In that case, the
prefetching
overwrites so many stuff in cache that the results of the onchanges are
lost.
Don't let that happen.
The function `stripped_sys_argv` was broken because of `itertools.groupby`
being hidden by the local `groupby`. Aligning APIs makes the local one a valid
substitution for `itertools.groupby`.
Put the code to update the MPTT in specific methods, and reduce the number of
queries being made (from 5-6 queries to 2-3 queries). Add test on MPTT to
validate the refactoring.
Current behaviour when duplicating a record:
apply copy override in the user language (e.g. "%s (copy)")
copy the previous term in all the other languages (including)
So if product was named:
cheeses (en)
kaas (nl)
fromage (fr)
Duplicating the product, while in French, will result into:
cheeses (en)
kaas (nl)
fromage (copie) (fr)
This is a problem if there is a unicity constrain on the field, it will be
raised (there is already a product named 'cheese')
After this PR:
Duplicating the product, while in French, will result into:
fromage (copie) (en)
kaas (nl)
fromage (copie) (fr)
This applies **only** if there is an override of copy
This is detected by comparing the old and new value for the user language
Fixes#7010Closes#22052
backport of afef71d6b9
Original commit message:
Let `create` and `write` round monetary field values before sending them to the
database. Pass the values to be written to `field.convert_to_column`, so that
the currency can be retrieved from the values, and the value be rounded.
When doing a group_by in a tree view, and then ordering by an
aggregated field which contains at least one capital letter in its name,
you will get an error and won't be able to ordery by this field.
Why? read_group is called when sorting a group by, which calls
_read_group_raw which then goes to call _read_group_prepare, which will
return order by terms in this manner ['id asc', 'x_Test desc']
the problem with this is that postgres automatically converts
non-quoted/non-qualified column name to lowercase, of course this means
that if we have a field "x_Test" but no field "x_test", it will try to
look up "x_test" instead of "x_Test" and since the column doesn't exist,
it will throw a ProgrammingError.
This commit changes this behavior by wrapping the order_by terms generated in
_read_group_prepare with double quotes, which postgres interprets
correctly.
Fixes#21348
Cherry-Pick of c338e24ee7 as previous
forward-port has been badly done...
Mathieu Duckerts-Antoine