Prefetching is underused when all fields are traversed one record at a time.
So instead, traverse all records one field at a time. This guarantees batch
prefetching/computation on every field being accessed.
That function is called when computing related fields in onchange mode.
The new implementation makes a direct access to the cache's implementation.
opw-772303
When reading a one2many field, the inverse field of the retrieved lines may
already be assigned in cache to a new record. This can happen during an
onchange. In that case, the algorithm crashes. Let's clean it up!
The command does not need to remove existing relations, as there is no relation
yet on new records. This saves one query per many2many field with that command
in a `create`.
An update by a list of commands is now guaranteed to have at most three SQL
queries to update the relation, whatever the number of commands and records.
The record cache is indexed by `(field, record_id, key)`, where `key` depends
on the environment. The key is either `(cr, uid, context)` or `(cr, uid)`,
depending on whether the field's value is context-dependent. As most fields
are not context-dependent, this should avoid some cache prefetching when
swiching context.
Make `RecordCache` work on a single record only, and implement a mapping from
field names (only) to values.
Also add explicit methods on `RecordCache` to wrap special behavior in cache,
and simplify all special values as a wrapper for a getter function.
The record cache may contain regular and special values. Modify `RecordCache`
so that only methods ending with `_value` check for regular values. The
dictionary methods considers both regular and special values as equivalent.
name in record._cache # test if cache has a value
iter(record._cache) # iterate on values
record._cache[name] # get value
record._cache.get(name) # get value or default
record._cache.has_value(name) # test if cache has a regular value
record._cache.get_value(name) # get regular value or default
The x2many "create" command is now like: `(0, ref, vals)`, where `ref` is an
arbitrary reference that may be used to identify a new line in the relation.
The reference is returned in a similar command by the method `onchange`.
`onchange` now supports self-modifying x2many fields.
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
* unichr -> pycompat (builtin removed from Python 3, ``chr`` has
become unicode-aware)
* Psycopg2 bytea values (binary fields) are returned as memoryviews in
P3 but buffers in P2
* Both bytes and (text) strings are conventionally iterable in
P3 (they have a __iter__ method), so fix up the exclusion pattern
for flatten
* remove references to basestring & unicode (use relevant pycompat
helpers)
* remove some str calls (either entirely or replaced by relevant
helper, either text or native)
* use better API to avoid unnecessary conversions
* remove some XML declarations in views
* StringIO removed from stdlib, replace with io
* try to correctly handle BytesIO/StringIO (one is for bytes the other
is for text)
* fix base64: Python 3 removed bytes-encoding and bytes-bytes
codecs (via #encode) so replace all calls to str.encode('base64'),
also b64encode is a bytes->bytes conversion so attempt to properly
handle that
issue #8530
When deciding to prefetch records (getting records from the cache with
no value for the field being fetched), if the field was computed
`determine_value` would just get all records, not limited by the normal
prefetch limit; for large recordsets this would generate gigantic
prefetch lists for records we may not need at all.
Fix by applying the `PREFETCH_MAX` limit to records from the cache as is
done in `_prefetch_field`.
Complementarily, when traversing related fields the prefetch
environment would be lost and every record would get an empty prefetch
environment, so the values would ultimately be read one by one.
Example: select (search) 1000 product.product records, access a
related field (e.g. categ_id) in a loop, on the first iteration the
system would first read 1000 templates, then it would read each
categ_id individually, resulting in >1000 SQL queries rather than the
~2 we would expect.
Fixes#18511
When several fields are modified and trigger the same recomputations, the same
SQL queries were made to determine the records to recompute, because each field
was processed independently. In order to avoid duplicated SQL queries, process
the triggers of the modified fields altogether.
This reduces the number of queries by 15% to 25%, depending on the number of
computed fields.
Python 3's hash randomisation means columns can be iterated — and thus
initialised — in any order (within a given model/override anyway),
which means there can be an implicit dependency on column
initialisation order e.g. (res.company).currency_id has a default
which fetches the currency_id of the company_id of the administrator,
and res.company has a _order=sequence,name, if the currency_id field
is created before the sequence the SELECT query will blow up as the
sequence column will not exist yet.
Attempt to always manipulate (iterate & initialize) fields based on
their order of definition using a global sequence number.
Let `create` and `write` round monetary field values before sending them to the
database. Pass the values to be written to `field.convert_to_column`, so that
the currency can be retrieved from the values, and the value be rounded.
* cross-version metaclass spec
* more formally deprecate browse_record and browse_null since they
were using metaclasses anyway
* update docstrings referencing the latter
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
In a form view, when a field onchange lead to a change on a x2many,
there was two different behavior:
- if the x2many had an embedded view (eg. a tree view inside a form
view) the onchange would notify that it expected the x2many field in
this embedded view to be changed and handled the changes correctly.
- if the x2many had a default view, the onchange ORM would not be
aware the x2many could be modified and would not sent the changes
back causing blank or not updated x2m lines and error on save.
---
Two solutions were birthed to solve the second point:
=> PR #10557 = solving everything
With this PR the onchange in the ORM is aware of every fields in the
current view (even field in a x2m in a x2m in a x2m in a form view) and
if any of these are change the javascript gets back the value of the
fields present in the view.
This PR has currently not been merged by fear of changing too much and
anyway could only be done in master.
=> PR #12249 = if no field for x2many, send its form view fields
With this change, if the ORM onchange is not aware of the fields in the
x2many widget to returns, all the field in the x2m default form view are
returned.
This was merged in bbdf960 but introduced a number of other issue:
- in most situation the x2many is represented by a list view, which may
have fields missing of the form view, so the original is still present.
- the view used may differ from the default form view in other way
(depending on value in context or other possibilities).
- the form view could have fields not present in the form view which
could end up in `write` on fields which should not be written to.
---
This commit reverts bbdf960 and adapts a small part of #10557 so the
x2many with default view works as an embedded x2many. For more than one
level (eg. a x2many in a x2many) this would still not work but it is
only solvable by a PR such as #10557 which could only be targetted for
master.
With this commit:
- the list of fields sent to ORM onchange is computed at the first onchange
- the fields from a x2many field default view is sent for onchange
- the initial onchange on record creation is delayed to when x2many are loaded
closes#12249, closes#15336, closes#15890fixes#11236, fixes#12249, fixes#15129, #15419
opw-705965 opw-716095 opw-715619 opw-710440
Christophe Simonis