Commit Graph
636 Commits
Author SHA1 Message Date
Adrian TorresandRaphael Collet 143cffe001 [FIX] base: reset _rec_name if x_name field is deleted
Commit 6a0028f91944b1d9e4eac026e86ca249ef5bc7ee introduced a change that
allowed field_triggers to be computed lazily per registry, this change
introduced a behavioral change that is not easy to notice, in order to
explain the behavioral change I will use the following example which was
the original bug reported:

    - Install studio
    - Create an app (create a new custom model M)
    - Uninstall studio
    -> Uninstall fails because field display_name of the custom model M
    depends on field M.x_name which has been removed due to the
    uninstall process.

To understand why it didn't happen before the aforementioned commit, we
must first understand what happens with said commit applied:

    - We trigger the uninstall of studio, this triggers the uninstall of
    any modules that depend on it, namely studio_customizations which is
    the module in which all customizations done with studio live in.

    - We gather all data belonging to the studio_customization and we
    start deleting in the following order: ...,
    ir.model.fields.selection, ir.model.fields, ..., ir.model

    - During the unlink process, we first remove the actual fields from
    the model instances before deleting their database reflections, this
    is done in the ir.model.fields._drop_column() method, it is this
    method that will delete the x_name field but **not** the
    display_name field, since it is a base field and not a custom one.

    - After the deletion of the fields in memory, we call modified() to
    mark fields that might've depended on the fields we just modified so
    that they can be recomputed later.

    - The call to modified will in turn access field_triggers, but since
    we're in a new registry and field_triggers is a lazy property, it
    will be computed right at this moment, this means that it will call
    resolve_depends on the display_name field which still exists, and
    this field has a dependency on the x_name field that we just
    deleted! This is what will trigger the crash.

With that context, we can now understand how it didn't crash before
commit 6a0028f91944b1d9e4eac026e86ca249ef5bc7ee:

    - Before the aforementioned commit, the field_triggers attribute was
    computed during the registry's setup_models(), in the case of an
    uninstall this call to setup_models was done way before the
    uninstall step of the registry (Step 3 is the last to call
    setup_models before Step 5).

    - This means that the old behavior was technically a bug, because
    right after removing x_name from the model, the field_triggers still
    contained a dependency from display_name to x_name, the former being
    no-longer present in memory.

With this commit, we simply reset the _rec_name and the dependencies of
the display_name if the x_name field is being removed, this ensures that
the computation of field_triggers won't crash and burn.

opw-2452498
opw-2478589

closes odoo/odoo#67823

X-original-commit: e6d22a43ff9f40e5fc7b8c84cd4dfe43f926d872
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2021-03-15 09:30:21 +00:00
Julien Castiaux dddf1bc34b [IMP] models: Better error than apple and orange
The "TypeError: Mixing apples and oranges" error message is raised when
users are using recordsets from different models together. The error
message was confusing some users thus have been reworded to be more
explicit.

The initial proposal was to replace "apples" and "oranges" by "torchons"
and "serviettes" but it was rejected because the French are not capable
not to mixup the two. They suggested to instead replace "apples" and
"oranges" by "pain au chocolat" and "chocolatine", this was rejected
because none of us understood the difference between the two, they both
are couques.

closes odoo/odoo#62266

Task: 2366612
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
2020-12-11 14:15:29 +00:00
Christophe MonniezandRaphael Collet c23b13eafe [FIX] models: call inverse method right after cache invalidation
When a record is written on a computed field with an inverse method and
an x2many field wit a delete command, the computed field is not properly
handled by the inverse method, because of a whole cache invalidation
that happens during the unlink.

With this commit, we write in cache on the fields that have been
invalidated before invoking the inverse method.

opw #2439861

closes odoo/odoo#67525

X-original-commit: d39855bad5153654d443d1b7ede3907f8bae3e66
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2021-03-09 11:24:09 +00:00
Raphael Collet e6369c2535 [IMP] core: write x2many fields last
Writing on x2many field should be done last, because deleting x2many
lines causes pending computations and updates to be flushed.  Writing on
column fields after that inevitably adds extra update queries.

We introduce an attribute `write_sequence` on fields to order fields for
write.  The prescribed order is: all fields except monetary and x2many,
monetary fields, x2many fields.

closes odoo/odoo#65959

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-02-22 16:20:55 +00:00
Raphael ColletandVictor Feyens 840609975a [IMP] core: better way to set/update magic fields
The goal of this change is to simplify the code managing `create_date`
and `write_date` in methods `create()` and `write()`, and also to remove
weird behaviors caused by the way those fields were updated.

Assume we update a simple field on a record.  This adds pending updates
for the field and `write_date`.  However, the value of `write_date` is
not known yet: it will be updated as `NOW() AT TIME ZONE 'UTC'` in SQL.
So `write_date` is actually given a dummy value in pending updates, and
it is invalidated from cache, until its value is flushed to the database
and fetched again.

Now assume we access another field on the record, and that field is not
in cache.  The prefetching mechanism will read all column fields,
including `write_date`, and flush them first.

    # this adds pending updates foo: 42, write_uid: 1, write_date: False
    record.foo = 42

    # assume 'bar' is not in cache; this prefetches all column fields,
    # which flushes the pending updates above before reading them back
    result = record.bar

We can avoid flushing pending updates if the values read from database
do not overwrite existing values in cache.  If you assume that the value
of a pending update is in cache (in the example, `foo: 42`), you don't
need to flush the corresponding field.  Indeed, the value of `foo` will
remain 42 in cache, whatever its value in the database.  This assumption
(pending updates are in cache) is true for all fields *except* for
`write_date`: it is invalidated from cache, and given a dummy value in
pending updates.  This branch actually makes this assumption true for
all fields.  The avoidance of flushing pending updates will be done in
another commit.

In order to directly assign `write_date` its value, we use a cache for
the value `NOW() AT TIME ZONE 'UTC'` from the database.  This costs at
most one query per transaction, and potentially saves a few queries.

Co-authored-by: Victor Feyens <vfe@odoo.com>
2021-02-22 16:20:55 +00:00
luz paz c9e29e5917 [FIX] *: correct typos
Various user facing an non-user-facing typos
Found via `codespell`

Closes odoo/odoo#65648

Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2021-02-19 13:20:48 +00:00
Raphael Collet 34c3498981 [FIX] core: exists() with string ids
The call model.browse('xxx').exists() should not return the record.
Instead, it should fail as the string 'xxx' is not a valid ID.

closes odoo/odoo#64783

X-original-commit: 824a6515a528cd2b682c3d3504c0d27ed9427cdb
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-01-20 10:00:42 +00:00
Raphael Collet 4de2ec280c [FIX] core: compute editable field with a default should not recompute
Consider a model with a field F and a computed editable field G that
depends on F.  Assume we open a form view, and both F and G have a
default value.  The first onchange should return F and G with their
respective default values.  In other words, the field G should not be
recomputed even if its dependency F was assigned a default value.

The use-case that showed the issue is batch payments.  When adding a new
payment in the one2many field of a batch, the new payment should use the
batch's payment method by default.  The default value for the payment's
method is assigned by context on the one2many field in the view.  Before
this patch, the new payment's method was given by the computation of the
field, and its default value was ignored.

Side note: another test was fixed, but that's because the test was not
consistent with the way Odoo 14.0 manages defaults and onchanges.

X-original-commit: 2ccc735253405b03927e2b08eab57427e65e4050
2021-01-13 16:19:13 +00:00
Raphael ColletandThibault Delavallée 65fd4373c5 [FIX] core: invoke constraint methods once per write
Consider a constraint method with two fields: a normal field, and a
field with an inverse method.  When calling write() with both fields,
the constraint method should be invoked once.

closes odoo/odoo#64433

X-original-commit: 25760348859f594abb796b5bc7b28f63e298cd53
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Thibault Delavallée <tde@odoo.com>
2021-01-12 15:59:10 +00:00
Raphael Collet 1e7659ad48 [FIX] core: avoid access error when flushing
This takes a former patch one step further.

Assume you recompute a field on two records, and the computation assigns
the first record but not the second one.  Before this patch, the
recomputation process was accessing the field on both records.  When
accessing the first record, both records are recomputed, and the value
of the first one is found in cache.  When accessing the second record,
no recomputation is made (it has been done already), and the cache is
empty. The field is thus fetched from database, and this crashes because
of insufficient access rights.

This scenario was causing some obscure nondeterministic crashes in
tests. The nondeterminism comes from the choice of the environment for
flushing (it should not be in superuser mode); the order of records in
the set to compute; the order in which records are assigned in the
compute method; the fact that the compute method should assign some
records and some not.

The fix consists in adding a method that processes pending computations,
without doing anything special for unassigned records.  The method is
used in field accessors (__get__ and mapped) and in method recompute().
The access error is gone because the recomputation no longer tries to
get the value of a field to compute.

closes odoo/odoo#64264

X-original-commit: b7676ec33172e6196b5fd1eda4df730ebf90eb6b
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-01-08 13:06:25 +00:00
Guewen BaconnierandRaphael Collet e14b52a56d [FIX] core: marking of stored computed recursive fields during creation
Fixes 3a6ac95 that ignores computation of recursive stored fields during
creation of records.

The problem happens when we have:

* a recursive stored computed field
* an `api.depends` of this field which is a relation to another record

When this "another record" is created and should recompute the recursive
field, it is ignored.

closes odoo/odoo#64117

X-original-commit: 9fe4fe404b73e9ab18ab6b3b913f075c64bb6954
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet (rco) <rco@openerp.com>
2021-01-05 19:09:16 +00:00
Raphael Collet 0a098aceaf [FIX] core: marking and traversing computed fields before modification
This fixes an issue that occurs when marking fields to compute in the
following situation:
 - it occurs before the actual modification,
 - a relational field is marked to compute,
 - the same field is traversed to inverse some dependency.

When the marking occurs after the modification, the traversal of the
field should actually recompute the field.  However, if the marking
occurs before the modification, the inversion of dependencies must be
based on the current value of the field.  This is the case with method
unlink(): we call method modified() to mark the fields that currently
depend on the records to be deleted, and the fields must be computed
only after the deletion!

X-original-commit: cd19c2d93db8911fe42802640ea32e5d87aadeec
2021-01-04 09:16:20 +00:00
svs-odoo 603d907450 [FIX] core: avoid duplicate onchanges
When performing the onchange for the first time on a new record, if a
field has a default value, it could be two times in the `todo` field
list to compute.

closes odoo/odoo#64005

X-original-commit: cf2b0a505c117a50a91fa022bc1d6bf1eb6d9f77
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-01-04 12:16:46 +00:00
Raphael Collet dcc2a7e39d [FIX] core: first onchange should always compute fields
When a form view is opened, the first call to `onchange` should always
compute fields, even if their dependencies have no default value.  Make
sure it is the case for main records, and for records inside one2many
fields.  The latter case was actually not working as expected.

closes odoo/odoo#63646

X-original-commit: 45422d56bce413b8577f1784e10dd22ede93c751
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-12-21 13:55:20 +00:00
Raphael Collet f2d4ba6890 [FIX] core: imports with one2many lines with dangling external ids
Consider a model M with a one2many field with comodel L, such that
deleting records in M cascade-deletes records in L.  Import a bunch of
records in M with one2many lines having specific external ids.  Delete
the imported records: the database cascade-deletes the corresponding
lines, but the lines' external ids are not deleted.  Now re-import the
same records: this crashes, because the import process finds the lines'
external ids and tries to update the deleted lines!

The fix consists in checking that an external id's record actually
exists.  The existence check was removed for performance reasons by
https://github.com/odoo/odoo/pull/26496.  We reintroduce it by resolving
the external id with a join on the model's name, which checks the
record's existence faster than an extra query.  This patch allows to
re-create the imported records' lines.

Another patch is necessary to actually update the external ids that
refer to deleted records, otherwise they remain dangling, and importing
the same records again duplicates the lines, because their external ids
do not match any existing record.

OPW 2409649

closes odoo/odoo#63318

X-original-commit: 0322febbea2cfd2d2fd7aedc42ef03632b17c967
Related: odoo/enterprise#15285
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-12-14 11:29:21 +00:00
Raphael Collet 18435a6b49 [FIX] core: filtered_domain() should return its result in order
This is a minor issue, but determinism is better than non-determinism.

closes odoo/odoo#62904

X-original-commit: 1b0869554d8755f8fdeb73dd2aa2ae7426522e84
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
2020-12-04 14:47:18 +00:00
Raphael Collet 19bd80758b [FIX] core: avoid multiple calls to default_get() in onchange
Upon the first call to `onchange`, the `values` dict is filled with
default values.  But the fields without a default are left unassigned.
This allows to compute fields when accessing them, but non-computed
fields will ask again for their default value, which is a waste of time
because we know they don't have one.  This may even be a performance
issue on forms with many fields.

Fix this issue by completing `values` with falsy values, and consider
the fields with some initial value (given or default) as the assigned
ones for triggering onchanges.  This explicitly triggers the computation
of fields that depend on fields with some initial value, without forcing
them to `False`.

Also avoid accessing x2many fields on the record in `snapshot.diff()`.
Enable the prefetching by simply putting the right value in cache.

closes odoo/odoo#62844

X-original-commit: 80c1dad6d9124da9459ce38d9e8c03e5c461c180
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-12-03 17:50:14 +00:00
Adrian Torres 1c8a958098 [IMP] core: introduce api.ondelete decorator
With this commit, a new ORM api decorator is introduced:
`api.ondelete(*, at_uninstall)`.

This decorator is to be applied to Model methods that check for specific
business conditions when attempting to unlink a record via the
interface.

E.g. trying to unlink a validated journal entry

This decorator allows this logic to exist outside the `BaseModel.unlink`
method and is automatically bypassed when in uninstall mode, this means
that during an uninstall any and all data related to a module can and
will be removed easily and cleanly while still being able to apply
business logic to manual deletion of records.

This feature opens the gates to solving a very big problem with
uninstalls: records and tables that remain in a database despite the
relevant module being uninstalled, because if an override to unlink
raises an error, the data will never be deleted from the database.

Henceforth, overrides of unlink shall solely be used for data-cleaning
purposes, i.e. deletion or modification of data that is related to the
one currently being deleted but cannot be automatically deleted because
there are no proper SQL relations.

In certain very specific, low-level scenarios an unlink may be
overridden to raise an error, but this should only be done if you know
what the fuck you're doing, most of the time you'll want to resort to
`@api.ondelete`.

Note that this new decorator includes a keyword-only, required argument
called `at_uninstall`, in most business cases this argument shall be
False as this argument dictates whether or not this method should be
executed in the `unlink` call during uninstall. It should only be set to
True if you are certain of all the implications which most likely means
that the records of the model in which this ondelete function is defined
will NOT be removed during uninstall, they will forever linger in the DB
until manual intervention, this in turn can mean a wide range of
undefined problems due to crap left on the database.

Following commits will replace any `unlink` overrides that raise
business errors by methods decorated with `api.ondelete`, another commit
will introduce a pylint checker that will raise a warning any time that
an unlink override raises an error.
2020-12-04 09:15:51 +00:00
Laurent Smet 6f2a6f7f2e [FIX] core: invalidation of computed editable field in one2many
This fixes the issue introduced by revision fd50ba9fb8

The way computed fields are invalidated depends on the order of the dict
passed as a parameter to method onchange().  This causes some unexpected
behavior when dealing with computed editable fields: the user loses the
value in that field because of the onchange.

Explanation: during the onchange, the modified field is cached by

    record._update_cache(changed_values, validate=True)

If the field is a one2many, 'value' contains an update command for each
modified line.  Because of 'validate=True', the assignment triggers
field computations on the lines, which are already handled by another
call to onchange().  In case anything on the parent model modifies some
line, the whole one2many values are returned to the user, and
accidentally recomputed fields show up in the interface.

closes odoo/odoo#62748

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-12-02 15:36:51 +00:00
Julien Castiaux eded14b4c4 [ADD] fields.py: New x2many command helper
We provide a new helper class to help redact x2many commands for create
and write methods. To ensure best compatibility with the xmlrpc layer we
do not change the protocole, the commands are still 3-elements tuples
where the first element is still an integer in between 0 and 6. The
helper class provide the cannonical constants and static methods to ease
working with the commands. The new helper class is also available in QWeb.

Developers are encouraged to transition their code so it uses this new
helper class.

Task: 2366606
2020-11-30 10:16:09 +00:00
Martin Trigaux 1658473bf2 [FIX] *: rephrase, correct typos
Courtesy of Transifex's translators for reporting bad/unclear sentences.

closes odoo/odoo#62564

X-original-commit: 9b3b2e8d711e3be75b1faffaf6f4f8f4ec90186e
Related: odoo/enterprise#15038
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2020-11-30 07:07:58 +00:00
Raphael Collet c2150d9117 [FIX] core: use 'x_name' as _rec_name only on custom models
This fixes a very obscure bug that happens on workers serving multiple
databases.  Consider two databases A and B with the same model M, that
does not have a field 'name', but has a custom field 'x_name' only on
database A.  The bug can be reproduced with the module 'account' and its
model 'account.register.payments'.

Assume the server loads a registry for database A.  In that registry,
the model M uses 'x_name' as its _rec_name, and the field 'display_name'
on model M determines its dependencies to be the field 'x_name'.

Now assume the server load a registry for database B.  In that registry,
the model M has no _rec_name.  However, an optimization reuses the field
'display_name' for the model M on the registry of A.  The field's
attribute 'depends' is equal to the tuple ('x_name',).  When the ORM
tries to resolve the field's dependencies, it does not find the field
'x_name' on M and crashes.

In order to avoid this situation, we forbid the usage of 'x_name' as
_rec_name on non-custom models.

OPW 2349238

closes odoo/odoo#61534

X-original-commit: 731e676fa8a9f7bdc6a66056c124c3b39a0800ad
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-11-09 12:39:19 +00:00
Daniel DuqueandRaphael Collet 55bd019a62 [FIX] core: missing default values with inherits
Consider some model A that "_inherits" from model B, which itself
"_inherits" from model C.  Also consider a field F on model C, which is
inherited by both models B and A.  Now create a record from model A,
with a given record for model C, no record for model B, and a default
value for field F.  This should create a record in A, connected to a new
record in B, connected to the given record in C, and the default value
should be ignored, as a record from C is given.

Before this commit, the given record in C is modified with the default
value for F.  The default value is considered because no record is given
for B, and that value is used to modify the given record in C.  The fix
consists in discarding default values by considering potential ancestor
records when parent records are not given.

closes odoo/odoo#61165

X-original-commit: b83dfaa2c9bcc6bc7f8df19ed0e442554816709b
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2020-11-02 13:53:36 +00:00
Rémy Voet (ryv) c7192d98ae [FIX] base: fix populate.randint tool
The randint generator generated always `None`
because the `return` was forget in method of `populate.compute`.
- Add a test for the randint tool, and fix the issue.
- Change the documentation of `_populate_factories` to be more
explicit about the custom generator.

closes odoo/odoo#61076

X-original-commit: 09d2ddecaf4fc6474c96acf1f4a12de4789caef8
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2020-10-30 16:38:57 +00:00
Raphael Collet 9e6df0fb71 [FIX] core: reinstall hooks after setting up models in registry
Before this patch, adding a field on a custom model discards all
automated actions on that model.  The explanation is relatively simple.
When models are set up in the registry, the classes of custom models are
dropped then recreated.  Given that automated actions are implemented as
monkey-patches on model classes, the setup of models simply loses those
monkey-patches, which explains why they stop working on custom models.

The fix introduces an `_unregister_hook()` method, that is expected to
clean up what has been done in `_register_hook()`.  When the registry is
ready (i.e., not being loaded), the setup of models first invokes
`_unregister_hook()` on models, proceeds with the setup, and finally
invokes `_register_hook()` to reinstall the hooks.

OPW 2362308

closes odoo/odoo#60833

X-original-commit: 67152bf82da2674179297d32e4cec9dd534fa0c9
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-10-27 13:59:20 +00:00
Loan (lse) e45767db3e [FIX] models.py: performance on filtered_domain on big recordset
Performance issue arise when the recordset is very large due to the union operator which apply at worst `len(self)` times.
By storing the ids then browse on them we avoid the union operation execution.

PySpy: https://drive.google.com/file/d/1jnxuSWGf2WmqeRz7zQOpBpmeRE5YnTwS/view?usp=sharing
filtered_domain (odoo/models.py:5398) take 65.6% of the graph (55.7% of the 85% of button_confirm)

To confirm a RFQ with 10 000 of the same product:
Before this commit: ~42s
After this commit:  ~10s

OPW-2347525

closes odoo/odoo#60131

X-original-commit: 8b0eeb5076ed5387ca7132d79182800e10c6895e
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-10-15 17:28:05 +00:00
Raphael Collet f243b71dcb [FIX] core: optimize first snaphot in onchange
This patch speeds up the performance of method `onchange`.  Our use-case
is an invoice with 300 lines, where we modify the unit price or quantity
on several lines.  Each modification triggers a call to `onchange` on
the invoice itself.  The latter call went from 1.3 to 0.8 seconds, which
represents a speedup of 30% to 40%.

In the implementation of `onchange`, the first snapshot is preceded by a
"prefetching" phase, where the lines of x2many fields are read, so that
the fields of unmodified lines are in cache.  This is useful because the
fields of those lines are not sent by the client, which only sends ids.
This prefetching represents more than 40% of the duration of `onchange`,
in our use-case.

The prefetching is inefficient for several reasons.  First, it uses
`mapped`, which formats data that is actually never used.  Second, it
accesses new records that have the actual lines as origin.  And on those
records, computed stored fields are not taken from the origin record,
but are (uselessly) computed instead.

We have optimized the prefetching in the following way.  It now reads
stored fields on the actual lines (using `_read` to avoid formatting),
then copies the cache of those fields on the corresponding new lines (to
avoid useless computations).  This makes the prefetching about 10 times
faster!

closes odoo/odoo#59781

X-original-commit: ec50c426d7c2a17c7b72e64adab65eda6328e163
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-10-12 15:20:47 +00:00
Raphael Collet 5d6b697ada [FIX] core: optimize prefetching on large recordsets
This patch optimizes the performance of prefetching when iterating on
large recordsets.  The optimization is *transparent*, i.e., it requires
no code change for it to apply.

The worst-case scenario for prefetching is the following: the ORM has to
fetch a field for a given `record`, `record._prefetch_ids` (its prefetch
set) is *very* large (many thousands), and most records in the prefetch
set are already in cache.  This requires the ORM to iterate a lot on the
prefetch set in order to make a batch of records not having the field in
cache.

    records = model.browse(ids)     # large recordset
    for record in records:
        record.foo                  # fetch 'foo' every 1k records

When running such a loop on an empty cache, the overhead of prefetching
(determine a batch) grows as the loop progresses.  The time complexity
of this loop is actually O(N²)...

The overhead of prefetching is minimal when `record._prefetch_ids` is
about the size of a prefetching unit, i.e., 1k records.  This commit
modifies the iterator method such that every record returned by the
iterator has a prefetch set of maximum 1k records.

We measured the time taken by the loop above on an empty cache, before
and after this commit, on a simple model (res.partner.category) with
100k records.  The third measure is a reference one: a `_read` on all
prefetched fields (to fill in the cache) followed by the loop.

                            Total time      Time per 1k records
    Before this commit      3.690s          30ms - 45ms
    After this commit       1.176s          12ms
    Read then loop          1.161s          -

The measures are enlightening: the overhead of prefetching was more than
200% of the reference time, and the time to prefetch records grows as
the iteration goes on!  This commit reduces the overhead of prefetching
to less than 2% of the reference time, and make it scale gracefully with
data size.

closes odoo/odoo#59239

X-original-commit: d0d54d63dbf49a97e7fea36fe7fb4b860a0b6606
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-10-06 10:29:38 +00:00
wanandRaphael Collet f89d7bbf8c [FIX] core: prefetching of x2many lines in snapshot.diff()
The prefetching optimization was simply broken for x2many lines in the
snapshot method `diff()`.  Indeed, iterating on `line._prefetch_ids`
actually returns nothing, because this iterable object relies on the
value in cache of the x2many field, and the cache has been invalidated
before calling `diff()`.  In order to re-enable prefetching, we simply
populate the cache of x2many fields before looping on the lines.

Some testing with `account.move` and the relation `invoice_line_ids`:
For 335 invoice lines, changing the value of a field in the invoice
lines took about 1.9s with 1415 queries.  It now takes about 1.1s and 82
queries.

closes odoo/odoo#58687

X-original-commit: 5bd7ddd0850e56fbbae52b4478970331f9db1589
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2020-09-28 14:07:47 +00:00
Raphael Collet bc0dd33e5b [FIX] core: compute on batches of maximum PREFETCH_MAX records
Calling a compute method on 1M records inevitably brings a worst case in
cache prefetching: the code that determines which records to fetch has
time complexity O(N²).  We avoid this situation by calling the compute
method on maximum 1K records.  Each computed batch has a "prefetch set"
of maximum 1K records too, which avoids the worst-case scenario above.

closes odoo/odoo#58664

X-original-commit: 0b1aa660b1d94fe46dea09670b875f2d81ca36a3
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-09-28 11:59:23 +00:00
Raphael Collet e05361fb72 [FIX] core: small optimization in first computation of records
X-original-commit: 99f765139076671ecba3b49359652e7333a90e17
2020-09-28 11:59:22 +00:00
Raphael Collet a216844e6b [FIX] core: reintroduce model dependencies
Make sure that model dependencies are flushed before searching with a
given domain.  This fixes inconsistencies in `search` and `read_group`.

closes odoo/odoo#57037

X-original-commit: 6b80632e98a932d2b552d6dec0fc7208ca8e04c1
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-09-03 13:42:15 +00:00
Raphael Collet 773b98cfd7 [FIX] core: documentation of flush()
X-original-commit: 60c01cac4f272392cce534302f2ecba4918008d2
2020-09-03 13:42:15 +00:00
Raphael Collet 0e6a8b5faf [FIX] core: filtered_domain with operators 'like' and 'ilike'
Make the method work when the value to test is `False`.

closes odoo/odoo#58064

X-original-commit: 6fb19219d333dc324603bcb2a937e977eecb037e
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-09-18 16:20:05 +00:00
Simon Genin (ges) 618067aa3f [IMP] base: Document layout preview improvements
The document layout preview is a complete defferent simplified template
with its own css that replicates at best the different styles.
It does not have the external layout features and lack of fidelity.

The new preview actually use the real documents templates and put the
result in an iframe. It now has a high fidelity, though not perfect.

The goal is for a better onboarding, where clients see easely how
documents will look if they had an app to generate them. Of course, the
data on the document is a false invoice.

Refactor all this from base to web.

Task ID 2304177

closes odoo/odoo#56995

X-original-commit: c121a246f16899735306266a3a12b526e08e7620
Signed-off-by: Lucas Perais (lpe) <lpe@odoo.com>
2020-09-03 08:44:35 +00:00
Victor Feyens c482fdbf05 [IMP] various: improve code style and performance by improve `all()` usage
PURPOSE

Clean code. Be more performance oriented.

SPECIFICATIONS

Improvements applied in this commit

  * not all() --> any(not) for earlier returns;
  * all([generator]) --> all(generator) to avoid unnecessary list casting.
    This code construct is better managed by all;

This commit will probably not have a big performance effect on standard
production databases. However each performance and cleaning improvement
is welcomed.

LINKS

Task ID-2328619

closes odoo/odoo#56810

X-original-commit: 1cc6bb1231401ea7f501d2f5b5e9641ec8734850
Related: odoo/enterprise#12802
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
2020-08-31 13:59:27 +00:00
Moisés López 732fd98bbc [IMP] models.py: Method search_read propagate kwargs to read method
Currently the method `read` uses by default the parameter `load='classic_read')`
  - `self.read(fields, load='classic_read')`

So, it computes `name_get` for all m2o fields for all records in `self`.

If you want to avoid computing the `name_get` to save time and process
You can use an empty string in load parameter:
 - e.g. `self.read(..., load='')`

The method `self.search_read` call to `search` and `read` methods
but `search_read` method is not possible to assign `load=''` argument
(or other arguments of the method `read`)
 - e.g. `self.search_read(..., load='')`

So, you need to use 2 lines of code:
records = self.search(...)
records.read(..., load='')

This commit changes `search_read` method to receive all keyword arguments of the method `read`
So, you can use `self.search_read(..., load='')` and the `read` parameter will be propagated

closes odoo/odoo#46391

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-08-19 17:08:52 +00:00
Yannick Tivisse 3eec4d6a69 [FW][MERGE] resource,calendar,hr_holidays: Improve the performances
TL;DR
=====

Improve performances with nearly a factor of 2. Use case, call action_validate on hr.leave for 100 employees
- 1987 requests -> 844 requests
- 1300 ms -> 700 ms

Purpose
=======

The first purpose of this commit is to add a test ensuring the number of request while creating
a company leave for 100 employees, if 15 of them already have a leave during that period.

It includes, the mass leave generation, and the conflicts resolutions. (Cancelling/Splitting the
already existing one and adapting the dates accordingly).

The second one is to reduce the number of request for this test.

In term of requests, currently we have:
- 5154 requests without bypassing the mail tracking + the activities management
- 1987 requests when bypassing the mail post-process (this is the current value, the bypassing was
  already done several month ago)
- 844 requests with all the optimization done in resource/calendar/hr_holidays

In terms of execution time, we have a reduction from +- 1300 ms to call the method action_validate
to +- 700 ms

As the performances issues severity increases with the number of leaves to create and the real time access
to the database, on the production base, we reduced the execution time to generate more than 500 hr.leaves
from several minutes to 21 seconds. A fix to avoid deadlock was already made at
https://github.com/odoo/enterprise/pull/10740/files

Some contortions were made to avoid changing a signature method in a stable release and thus
introducing for each method a second one, with the "batched" implementation, to keep a retro-compatibility
for the existing custom code.

But surely this could be cleaned in the master version. The old one will be deprecated while waiting to be
removed in a few versions.

closes odoo/odoo#56534

Taskid: 2256705
X-original-commit: 8c96a887d3f05680c923dcd44f9c70c0e5e0bd32
Related: odoo/enterprise#12670
Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
2020-08-25 15:44:07 +00:00
Raphael Collet 3e91fa1dbf [FIX] core: first onchange on new one2many line
Consider models `M` and `L`, with a one2many field `M.l_ids` with
inverse `L.m_id`.  Consider also a computed field `L.foo` that depends
on `L.m_id`.

The field `L.foo` is not computed correctly by `onchange` on a new line
in the form view of `M`: its value is forced to `False` because it has
no default, and is not recomputed by `onchange` since the field `L.m_id`
is never triggered an `onchange`.

Modify the method `onchange` to not force fields without a default to
`False`, and return all fields.
2020-08-20 13:39:04 +00:00
Raphael Collet 488e334fc8 [REF] core: combine default_get() with first onchange()
When creating a new record, the client calls `default_get()`, completes
the returned values with `False`, and calls `onchange()` to apply the
onchange to the defaults.

Optimize the double round-trip by integrating `default_get()` inside
`onchange()` for the first call.  The method is called with an empty
list of fields, and usually no field values, except for records in a
one2many field.  In this case, `onchange()` does both steps above.

Task 2261084
2020-08-20 13:39:04 +00:00
wanandrco-odoo f2ceef0e2f [IMP] core: add models defined by a query instead of a table/view
By adding a parameter `_table_query` on the Model, we can now have views
that depend on the context.  The query is used instead of the table name
in ORM operations.  This allows to pre-compute some values to improve
performance, instead of storing context values in the database.

Co-authored-by: william-andre <wan@odoo.com>
Co-authored-by: rco-odoo <rco@odoo.com>
2020-08-20 07:45:18 +00:00
Raphael Collet 69869ab681 [IMP] core: introduce subqueries in search
Make the method `_search` return a `Query` object, and make that object
generate a subquery when used on the right-hand side of a condition.

closes odoo/odoo#52403

Related: odoo/enterprise#10945
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-08-18 13:02:41 +00:00
Raphael Collet 0f7f163d2e [REF] core: improve the API and implementation of Query
Introduce better methods for introducing tables, join clauses and where
clauses to the Query object, and other methods to generate a complete
SELECT query.  The Query object can also behave as a tuple of ids, where
the query is made on demand, and the result is memoized.
2020-08-18 13:02:41 +00:00
Raphael Collet 1f48130d2b [REF] core: _name_search() now returns a list of ids
This provides a better API to search for records by name without the
formatting part (`display_name`).

This also simplifies all the overridings of `_name_search` that no
longer need to call `name_get()`.  The call to `name_get()` is done in
method `name_search` in a generic way.
2020-08-18 13:02:41 +00:00
Xavier Morel bfa00919cd [IMP] core: clarify making fields inaccessible
* add a constant to `fields` for that purpose
* add a test to ensure that it works as expected
* fix the formatter so it handles the pattern correctly
2020-08-17 09:30:53 +00:00
Xavier Morel 48335c04bf [IMP] core: orm doc
* fix link from backend tutorial to log_access section
* flesh out log access documentation
* add documentation for log_access itself & improve doc for a few
  other model-level attributes
2020-07-28 13:03:12 +00:00
Xavier Morel ef709fd92b [IMP] core, base: avoid multiple xids when importing records w/ inherits
Before this change, we create an xid for every parent of a record
being imported, regardless of whether it already has an xid, or if
it's being created implicitly through the child.

This generates unnecessary extra xids on pre-existing objects
e.g. update 5 product variants -> the product gets 5 new xids despite
already having one.

We should *only* set a xid on parent records which are being
implicitly created by the creation of a child with a specified
xid. That is, we should never set a xid on the parent if it exists
before the child is created.

Update _process_end to try and see if "non-loaded" xids correspond to
an automatically generated "parent" xid: we're still setting a xid on
implicitly created records (if the child is created with a xid) so
they're properly removed if e.g. the module is uninstalled, but
because we're only doing so at creation these xids will not be visited
during update and _process_end will try to delete them.

A special case can be added to check that "unknown" parent xids don't
have children which _inherit them, in which case we want to protect them.

Task 2251039

closes odoo/odoo#53283

Related: odoo/enterprise#12023
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-07-27 06:32:59 +00:00
Raphael Collet 4147f1d313 [FIX] core: make check_company's error message clearer
Make the error message more comprehensible for users, using the record's
display name, the company's name, the field's label and value.

    Incompatible companies on records:
     - 'Foo' belongs to company 'X' and 'Contact' (partner_id: 'Bar')
       belongs to another company.

closes odoo/odoo#54883

X-original-commit: e410c90bbddb581100d170bf8180e790bb7af5fd
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-07-24 08:37:45 +00:00
Victor Feyens 71ab12840b [IMP] *: do not specify unwanted defaults in default_get
When default_get is called, the wanted fields are specified through the
fields_list arg.  It is useless to fill the values for unwanted fields.

As default_get is called for nearly all records creation, simplifying
the default_get overrides:
* remove potential wrong side-effects of the values
* remove some useless or wrong  defaults computations (searches, refs,
...)
2020-07-23 16:38:17 +00:00
root e3f68b9ba8 [IMP] base: specify which order is invalid
Before this commit knowing which _order option was having issues may
be non-obvious.
This changes makes it a bit more obvious.

closes odoo/odoo#54703

Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2020-07-23 08:58:48 +00:00