Magic and inherited fields are not really useful on abstract models.
The _inherits specification is used anyway by models that inherit from
those abstract models.
The main goal of this change is to prepare a refactoring of models where
fields are no longer duplicated on the registry classes, but fields
defined on classes are used directly. But this new design cannot be
applied to all fields: a field being overridden simply cannot be used
directly. This branch improves the situation by avoiding unnecessary
field overridings.
closesodoo/odoo#69372
Related: odoo/upgrade#2409
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
With this commit, it is now possible to group the records of a model by
a Many2many field of said model.
The result of a read_group grouped by a m2m will return as many entries
or "groups" as there are different records in the comodel that are
linked to the model through the m2m, plus a null/false group, for records
of the model that have no linked records of the comodel or for records
of the comodel for which the current user has no read access to due to
ir.rules.
For a more illustrated explanation, the tests should cover all cases in
detail.
Note that this commit only introduces this change at the ORM level, the
frontend does not yet handle grouping by m2m fields but it is planned
in the near future.
Task-2428971
closesodoo/odoo#68958
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
CRM install query count can vary from one execution to another, leading
to difficulties when analysing performances evolution.
The main reason for this is that some compute methods were called in
different order. Even if compute order shouldn't have any effect on the
final result, making it well defined will help finding other causes of
non-determinism.
The initial observation was that sorting Environment.fields_to_compute
leads to a fixed number of query when installing crm.
The main cause of non-determinisim is the usage of `set` impacting
Field.compute_value and BaseModel._modified_triggers.
Transforming all these `set` to `OrderedSet` solves the problem.
The query count is now deterministic when installing a database from
scratch, but not when updating a database with -i crm.
OrderedSet is also slightly optimised by using a dict instead of an
closesodoo/odoo#68692
Ordereddict: dict order is deterministic since python3.6
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Purpose
=======
Propose an import tool that is more intuitive and allows more import options to
ease the import.
Specifications
==============
Update upload import file screen
--------------------------------
- Relabel import welcome screen
- relabel the 'load file' button to 'upload file'
- relabel the first (bold) line of the helper to 'Upload an Excel or CSV
file to import'
- Move all import options in a left panel
- Invert rows and columns in the mapping view
- One row for each column (header) to map in the imported file
- Columns: File Column / Odoo Field / If a match cannot be found / Comments
Left panel details
------------------
The sidepanel to the left of the mapping view is composed as followed:
- Imported File
- Import file name
- Sheet: Dropdown to select the sheet in the file if there are more than 1.
- 'use first row as header' checkbox: like the existing option, defines
whether the first row of the file should be considered as the header.
True by default. If False, display the first value only under 'File Column'
(see above)
- Formatting: only available if the file is a .csv
- list every option that already existed in previous import implementation.
- Batch Import
- is only available in debug moide and if the file exceeds the batch limit
- Batch limit : 2000
- Allow to define the threshold and the batch size.
- Help
- Download Template link
- Go to FAQ link
- Advanced
- Track history during import. (same as existing functionallity)
- Show Fields of relation fields. (same as existing functionallity)
Columns details
---------------
- File Column:
- Contains the column headers of the import file and the first non-empty
value of the column as a 'subtitle' (in grey italic)
- If 'use first row as header' is False, only display the first column value
(without the grey italic)
- Odoo Field:
- Contains dropdowns to the fields of the current model
- Required fields are diplayed in bold in the dropdown
- Icon: In the front of the field label, the field type (char, many2one,..)
is indicated by an icon.
- Tooltip: when hovering a field, display a tooltip with the following
information: field label, technical name, type, related model (if any)
- Placeholder: If no field is selected, display the placeholder
'To import, select a field...' in text-warning bold (i.e. orange)
- Allow clear: there is a close (fa-times) icon at the end of the dropdown
body to remove the Odoo field (and prevent import of this column)
- Multi Mapping: when setting a field already matched to another column,
unset it from the 'old' column (NB: exception made for char/text fields)
- Comments:
- Contains feedback from the system to the user, either import
errors/warnings or any additional details (multi mapping comments,...)
- For many2many field, display in related comment cell the following:
"To import multiple values, separate them by a comma".
- If there is an import error for a specific field, the error will be
displayed in the comment cell of the related field.
- If there is a mapping error, mapping options are displayed under the
error div to let the user choose the best option to fix that error.
Import errors management
------------------------
- Errors/warnings that can be matched to a specific field are displayed as
alert-danger/warning in the corresponding 'Comments' cell of the field.
- For 'no match found' errors, if X values couldn't be found,
display an unique error box will all the errors.
- Values beyond the first three are folded under a 'More' button.
- No changes on the 'See possible values' button.
- If an unmatched value is present in one row only, display :
'<value> at row X (<name of the row if the name field is matched>)'
- If an unmatched value is multiple rows, display:
'<value> at multiple rows'
- Errors/warnings/infos that cannot be matched to specific field are displayed
above the mapping listview. Global errors are not regrouped by error types to
avoid too much code complexy just to handle the rare times where multiple errors
of same types cannot be linked to a specific mapped field.
BaseImportError class have been introduced to ease the formatting of the various
exceptions that can occur during the import.
After testing the import, display the following above the mapping table:
- No warning/error:
'Everything seems valid' (alert-info)
- At least one error:
'The file contains blocking errors (see below)' (alert-error)
- At least one warning but no error:
'The file contains non-blocking warning (see below)' (alert-warning)
Mapping options
---------------
Mapping options are only displayed after testing or importing the file,
if there are import errors. The goal is to have a clean interface and to guide
the user step by step. The import process can therefore be a bit longer as it
needs to import -> choose solution for errors -> re-import but that is easier
for users to learn and understand this reworked import tool.
Possible values when a value cannot be matched:
- For many2one / many2many fields:
- Prevent import: (selected by default) not finding a match is blocking the
import
- Skip unknown values: values that cannot be matched will be skipped.
(hidden if the field is required)
- Create new values: Create records for values that cannot be matched
- For selection fields:
- Prevent import (selected by default)
- Skip unknown values: (hidden if the field is required)
- Set to <first value>: if cannot be matched, set it to <first value>
- Set to <second value>
- Set to <third value>
- etc.
- For boolean fields:
- Prevent Import (selected by default)
- Set to True
- Set to False
Note: With this rework, boolean warnings where the system assumes the
replacement value in case of matching error is removed and is replaced by a
blocking error. The user now has to choose the value to set.
"Prevent import" is the default behaviour when testing or importing.
Automatic mapping proposal
---------------------------
When loading a file, an automated mapping is directly proposed to the user,
based on word distance (see below), and on mapping created on previous
imports.
- Priority is given for mapping created on previous imports, skip fuzzy mapping.
- a distance of -1 is used to ensure priority during duplicates removal .
- In case multiple headers are mapped on the same field, if the mapped field
is already taken by another header, use fuzzy mapping instead.
- If no previous mapping for that header on that model, try an exact match on
every field id and field name of the model. (distance = 0)
- If no match is found, fuzzy mapping is applied (word distance).
The fuzzy mapping is executed only on the most likely fields (see below).
- Remove duplicates: keep the header-field couple that has the smallest
distance. In case of equality, keep the first.
Automatic mapping is therefore optimised for previous mapping or exact match,
as fuzzy mapping requires heaviest treatment. This is intended to prioritise the
import of files based on import templates.
Most likely fields
------------------
When parsing the import file, each header is analysed to guess what type of data
the column contains. For example, ff the column contains float, we suppose that
that header will most likely be matched on float or monetary fields.
The most likely fields are a subset of the model's fields that match the
header types.
Most likely fields are used for the fuzzy mapping, to propose the user a field
mapping based on the header types, if an exact match could not be found.
Most likely fields are also listed under "Suggested fields" in the mapping
dropdown in "Odoo Field" column of the import tool.
For now on, every fields (no mather their type) can be matched to any header,
but we prioritise the most likely fields. This way, we don't constrain the
mapping possibility based on what we suppose the user would do, but we instead
guide suggest the user the most likely mapping solutions.
Word distance mapping
----------------------------
In order to improve the mapping configuration, if an exact match cannot be found
between the file column and one of the odoo field:
- Use Word distance:
Word distance return a indice between 0 and 1.
- 0: exact match
- 1: completely different
- A: First try on field['name']
- B: Then on field['string']
- Keep the minimal distance between A and B for each odoo_field
- C: Keep the field that has the minimal distance.
- Match the column to the field by default if C['distance'] < 0.3.
Note: 0.3 has been chosen to ensure proximity but still having a little
error margin.
Multi mapping
-------------
When multiple file columns are matched to the same char/text/many2many field,
display the following alert-info box in the "Comments" column of the related
fields:
"Those columns will be concatenated in field <field label>"
Multi mapping rule :
- If it is a char field, separate the concatenated values by a space
- If it is a text field, separate the concatenated values by a line break
- If it is a many2many field, separate the concatenated values by a comma
Various improvements
--------------------
- During Import (or Test), the first waiting message displayed has been modified
to 'Importing...' or 'Testing...'. The progress (x record imported/tested) is
only displayed after the first batch of record have been imported/tested.
- Ease matching for Selection field: use case insensitive comparaison instead
of exact match.
- Add filename to import wizard.
- Add placeholder to search input of field mapping dropdown.
Tests have been adapted accordingly.
Links
=====
Task ID: 2352241
closesodoo/odoo#61948
Related: odoo/enterprise#17246
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Commit 6a0028f91944b1d9e4eac026e86ca249ef5bc7ee introduced a change that
allowed field_triggers to be computed lazily per registry, this change
introduced a behavioral change that is not easy to notice, in order to
explain the behavioral change I will use the following example which was
the original bug reported:
- Install studio
- Create an app (create a new custom model M)
- Uninstall studio
-> Uninstall fails because field display_name of the custom model M
depends on field M.x_name which has been removed due to the
uninstall process.
To understand why it didn't happen before the aforementioned commit, we
must first understand what happens with said commit applied:
- We trigger the uninstall of studio, this triggers the uninstall of
any modules that depend on it, namely studio_customizations which is
the module in which all customizations done with studio live in.
- We gather all data belonging to the studio_customization and we
start deleting in the following order: ...,
ir.model.fields.selection, ir.model.fields, ..., ir.model
- During the unlink process, we first remove the actual fields from
the model instances before deleting their database reflections, this
is done in the ir.model.fields._drop_column() method, it is this
method that will delete the x_name field but **not** the
display_name field, since it is a base field and not a custom one.
- After the deletion of the fields in memory, we call modified() to
mark fields that might've depended on the fields we just modified so
that they can be recomputed later.
- The call to modified will in turn access field_triggers, but since
we're in a new registry and field_triggers is a lazy property, it
will be computed right at this moment, this means that it will call
resolve_depends on the display_name field which still exists, and
this field has a dependency on the x_name field that we just
deleted! This is what will trigger the crash.
With that context, we can now understand how it didn't crash before
commit 6a0028f91944b1d9e4eac026e86ca249ef5bc7ee:
- Before the aforementioned commit, the field_triggers attribute was
computed during the registry's setup_models(), in the case of an
uninstall this call to setup_models was done way before the
uninstall step of the registry (Step 3 is the last to call
setup_models before Step 5).
- This means that the old behavior was technically a bug, because
right after removing x_name from the model, the field_triggers still
contained a dependency from display_name to x_name, the former being
no-longer present in memory.
With this commit, we simply reset the _rec_name and the dependencies of
the display_name if the x_name field is being removed, this ensures that
the computation of field_triggers won't crash and burn.
opw-2452498
opw-2478589
closesodoo/odoo#67823
X-original-commit: e6d22a43ff9f40e5fc7b8c84cd4dfe43f926d872
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
The "TypeError: Mixing apples and oranges" error message is raised when
users are using recordsets from different models together. The error
message was confusing some users thus have been reworded to be more
explicit.
The initial proposal was to replace "apples" and "oranges" by "torchons"
and "serviettes" but it was rejected because the French are not capable
not to mixup the two. They suggested to instead replace "apples" and
"oranges" by "pain au chocolat" and "chocolatine", this was rejected
because none of us understood the difference between the two, they both
are couques.
closesodoo/odoo#62266
Task: 2366612
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
When a record is written on a computed field with an inverse method and
an x2many field wit a delete command, the computed field is not properly
handled by the inverse method, because of a whole cache invalidation
that happens during the unlink.
With this commit, we write in cache on the fields that have been
invalidated before invoking the inverse method.
opw #2439861closesodoo/odoo#67525
X-original-commit: d39855bad5153654d443d1b7ede3907f8bae3e66
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Writing on x2many field should be done last, because deleting x2many
lines causes pending computations and updates to be flushed. Writing on
column fields after that inevitably adds extra update queries.
We introduce an attribute `write_sequence` on fields to order fields for
write. The prescribed order is: all fields except monetary and x2many,
monetary fields, x2many fields.
closesodoo/odoo#65959
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The goal of this change is to simplify the code managing `create_date`
and `write_date` in methods `create()` and `write()`, and also to remove
weird behaviors caused by the way those fields were updated.
Assume we update a simple field on a record. This adds pending updates
for the field and `write_date`. However, the value of `write_date` is
not known yet: it will be updated as `NOW() AT TIME ZONE 'UTC'` in SQL.
So `write_date` is actually given a dummy value in pending updates, and
it is invalidated from cache, until its value is flushed to the database
and fetched again.
Now assume we access another field on the record, and that field is not
in cache. The prefetching mechanism will read all column fields,
including `write_date`, and flush them first.
# this adds pending updates foo: 42, write_uid: 1, write_date: False
record.foo = 42
# assume 'bar' is not in cache; this prefetches all column fields,
# which flushes the pending updates above before reading them back
result = record.bar
We can avoid flushing pending updates if the values read from database
do not overwrite existing values in cache. If you assume that the value
of a pending update is in cache (in the example, `foo: 42`), you don't
need to flush the corresponding field. Indeed, the value of `foo` will
remain 42 in cache, whatever its value in the database. This assumption
(pending updates are in cache) is true for all fields *except* for
`write_date`: it is invalidated from cache, and given a dummy value in
pending updates. This branch actually makes this assumption true for
all fields. The avoidance of flushing pending updates will be done in
another commit.
In order to directly assign `write_date` its value, we use a cache for
the value `NOW() AT TIME ZONE 'UTC'` from the database. This costs at
most one query per transaction, and potentially saves a few queries.
Co-authored-by: Victor Feyens <vfe@odoo.com>
The call model.browse('xxx').exists() should not return the record.
Instead, it should fail as the string 'xxx' is not a valid ID.
closesodoo/odoo#64783
X-original-commit: 824a6515a528cd2b682c3d3504c0d27ed9427cdb
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Consider a model with a field F and a computed editable field G that
depends on F. Assume we open a form view, and both F and G have a
default value. The first onchange should return F and G with their
respective default values. In other words, the field G should not be
recomputed even if its dependency F was assigned a default value.
The use-case that showed the issue is batch payments. When adding a new
payment in the one2many field of a batch, the new payment should use the
batch's payment method by default. The default value for the payment's
method is assigned by context on the one2many field in the view. Before
this patch, the new payment's method was given by the computation of the
field, and its default value was ignored.
Side note: another test was fixed, but that's because the test was not
consistent with the way Odoo 14.0 manages defaults and onchanges.
X-original-commit: 2ccc735253405b03927e2b08eab57427e65e4050
Consider a constraint method with two fields: a normal field, and a
field with an inverse method. When calling write() with both fields,
the constraint method should be invoked once.
closesodoo/odoo#64433
X-original-commit: 25760348859f594abb796b5bc7b28f63e298cd53
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Thibault Delavallée <tde@odoo.com>
This takes a former patch one step further.
Assume you recompute a field on two records, and the computation assigns
the first record but not the second one. Before this patch, the
recomputation process was accessing the field on both records. When
accessing the first record, both records are recomputed, and the value
of the first one is found in cache. When accessing the second record,
no recomputation is made (it has been done already), and the cache is
empty. The field is thus fetched from database, and this crashes because
of insufficient access rights.
This scenario was causing some obscure nondeterministic crashes in
tests. The nondeterminism comes from the choice of the environment for
flushing (it should not be in superuser mode); the order of records in
the set to compute; the order in which records are assigned in the
compute method; the fact that the compute method should assign some
records and some not.
The fix consists in adding a method that processes pending computations,
without doing anything special for unassigned records. The method is
used in field accessors (__get__ and mapped) and in method recompute().
The access error is gone because the recomputation no longer tries to
get the value of a field to compute.
closesodoo/odoo#64264
X-original-commit: b7676ec33172e6196b5fd1eda4df730ebf90eb6b
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Fixes 3a6ac95 that ignores computation of recursive stored fields during
creation of records.
The problem happens when we have:
* a recursive stored computed field
* an `api.depends` of this field which is a relation to another record
When this "another record" is created and should recompute the recursive
field, it is ignored.
closesodoo/odoo#64117
X-original-commit: 9fe4fe404b73e9ab18ab6b3b913f075c64bb6954
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet (rco) <rco@openerp.com>
This fixes an issue that occurs when marking fields to compute in the
following situation:
- it occurs before the actual modification,
- a relational field is marked to compute,
- the same field is traversed to inverse some dependency.
When the marking occurs after the modification, the traversal of the
field should actually recompute the field. However, if the marking
occurs before the modification, the inversion of dependencies must be
based on the current value of the field. This is the case with method
unlink(): we call method modified() to mark the fields that currently
depend on the records to be deleted, and the fields must be computed
only after the deletion!
X-original-commit: cd19c2d93db8911fe42802640ea32e5d87aadeec
When performing the onchange for the first time on a new record, if a
field has a default value, it could be two times in the `todo` field
list to compute.
closesodoo/odoo#64005
X-original-commit: cf2b0a505c117a50a91fa022bc1d6bf1eb6d9f77
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
When a form view is opened, the first call to `onchange` should always
compute fields, even if their dependencies have no default value. Make
sure it is the case for main records, and for records inside one2many
fields. The latter case was actually not working as expected.
closesodoo/odoo#63646
X-original-commit: 45422d56bce413b8577f1784e10dd22ede93c751
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Consider a model M with a one2many field with comodel L, such that
deleting records in M cascade-deletes records in L. Import a bunch of
records in M with one2many lines having specific external ids. Delete
the imported records: the database cascade-deletes the corresponding
lines, but the lines' external ids are not deleted. Now re-import the
same records: this crashes, because the import process finds the lines'
external ids and tries to update the deleted lines!
The fix consists in checking that an external id's record actually
exists. The existence check was removed for performance reasons by
https://github.com/odoo/odoo/pull/26496. We reintroduce it by resolving
the external id with a join on the model's name, which checks the
record's existence faster than an extra query. This patch allows to
re-create the imported records' lines.
Another patch is necessary to actually update the external ids that
refer to deleted records, otherwise they remain dangling, and importing
the same records again duplicates the lines, because their external ids
do not match any existing record.
OPW 2409649
closesodoo/odoo#63318
X-original-commit: 0322febbea2cfd2d2fd7aedc42ef03632b17c967
Related: odoo/enterprise#15285
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
This is a minor issue, but determinism is better than non-determinism.
closesodoo/odoo#62904
X-original-commit: 1b0869554d8755f8fdeb73dd2aa2ae7426522e84
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
Upon the first call to `onchange`, the `values` dict is filled with
default values. But the fields without a default are left unassigned.
This allows to compute fields when accessing them, but non-computed
fields will ask again for their default value, which is a waste of time
because we know they don't have one. This may even be a performance
issue on forms with many fields.
Fix this issue by completing `values` with falsy values, and consider
the fields with some initial value (given or default) as the assigned
ones for triggering onchanges. This explicitly triggers the computation
of fields that depend on fields with some initial value, without forcing
them to `False`.
Also avoid accessing x2many fields on the record in `snapshot.diff()`.
Enable the prefetching by simply putting the right value in cache.
closesodoo/odoo#62844
X-original-commit: 80c1dad6d9124da9459ce38d9e8c03e5c461c180
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
With this commit, a new ORM api decorator is introduced:
`api.ondelete(*, at_uninstall)`.
This decorator is to be applied to Model methods that check for specific
business conditions when attempting to unlink a record via the
interface.
E.g. trying to unlink a validated journal entry
This decorator allows this logic to exist outside the `BaseModel.unlink`
method and is automatically bypassed when in uninstall mode, this means
that during an uninstall any and all data related to a module can and
will be removed easily and cleanly while still being able to apply
business logic to manual deletion of records.
This feature opens the gates to solving a very big problem with
uninstalls: records and tables that remain in a database despite the
relevant module being uninstalled, because if an override to unlink
raises an error, the data will never be deleted from the database.
Henceforth, overrides of unlink shall solely be used for data-cleaning
purposes, i.e. deletion or modification of data that is related to the
one currently being deleted but cannot be automatically deleted because
there are no proper SQL relations.
In certain very specific, low-level scenarios an unlink may be
overridden to raise an error, but this should only be done if you know
what the fuck you're doing, most of the time you'll want to resort to
`@api.ondelete`.
Note that this new decorator includes a keyword-only, required argument
called `at_uninstall`, in most business cases this argument shall be
False as this argument dictates whether or not this method should be
executed in the `unlink` call during uninstall. It should only be set to
True if you are certain of all the implications which most likely means
that the records of the model in which this ondelete function is defined
will NOT be removed during uninstall, they will forever linger in the DB
until manual intervention, this in turn can mean a wide range of
undefined problems due to crap left on the database.
Following commits will replace any `unlink` overrides that raise
business errors by methods decorated with `api.ondelete`, another commit
will introduce a pylint checker that will raise a warning any time that
an unlink override raises an error.
This fixes the issue introduced by revision fd50ba9fb8
The way computed fields are invalidated depends on the order of the dict
passed as a parameter to method onchange(). This causes some unexpected
behavior when dealing with computed editable fields: the user loses the
value in that field because of the onchange.
Explanation: during the onchange, the modified field is cached by
record._update_cache(changed_values, validate=True)
If the field is a one2many, 'value' contains an update command for each
modified line. Because of 'validate=True', the assignment triggers
field computations on the lines, which are already handled by another
call to onchange(). In case anything on the parent model modifies some
line, the whole one2many values are returned to the user, and
accidentally recomputed fields show up in the interface.
closesodoo/odoo#62748
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
We provide a new helper class to help redact x2many commands for create
and write methods. To ensure best compatibility with the xmlrpc layer we
do not change the protocole, the commands are still 3-elements tuples
where the first element is still an integer in between 0 and 6. The
helper class provide the cannonical constants and static methods to ease
working with the commands. The new helper class is also available in QWeb.
Developers are encouraged to transition their code so it uses this new
helper class.
Task: 2366606
This fixes a very obscure bug that happens on workers serving multiple
databases. Consider two databases A and B with the same model M, that
does not have a field 'name', but has a custom field 'x_name' only on
database A. The bug can be reproduced with the module 'account' and its
model 'account.register.payments'.
Assume the server loads a registry for database A. In that registry,
the model M uses 'x_name' as its _rec_name, and the field 'display_name'
on model M determines its dependencies to be the field 'x_name'.
Now assume the server load a registry for database B. In that registry,
the model M has no _rec_name. However, an optimization reuses the field
'display_name' for the model M on the registry of A. The field's
attribute 'depends' is equal to the tuple ('x_name',). When the ORM
tries to resolve the field's dependencies, it does not find the field
'x_name' on M and crashes.
In order to avoid this situation, we forbid the usage of 'x_name' as
_rec_name on non-custom models.
OPW 2349238
closesodoo/odoo#61534
X-original-commit: 731e676fa8a9f7bdc6a66056c124c3b39a0800ad
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Consider some model A that "_inherits" from model B, which itself
"_inherits" from model C. Also consider a field F on model C, which is
inherited by both models B and A. Now create a record from model A,
with a given record for model C, no record for model B, and a default
value for field F. This should create a record in A, connected to a new
record in B, connected to the given record in C, and the default value
should be ignored, as a record from C is given.
Before this commit, the given record in C is modified with the default
value for F. The default value is considered because no record is given
for B, and that value is used to modify the given record in C. The fix
consists in discarding default values by considering potential ancestor
records when parent records are not given.
closesodoo/odoo#61165
X-original-commit: b83dfaa2c9bcc6bc7f8df19ed0e442554816709b
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
The randint generator generated always `None`
because the `return` was forget in method of `populate.compute`.
- Add a test for the randint tool, and fix the issue.
- Change the documentation of `_populate_factories` to be more
explicit about the custom generator.
closesodoo/odoo#61076
X-original-commit: 09d2ddecaf4fc6474c96acf1f4a12de4789caef8
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Before this patch, adding a field on a custom model discards all
automated actions on that model. The explanation is relatively simple.
When models are set up in the registry, the classes of custom models are
dropped then recreated. Given that automated actions are implemented as
monkey-patches on model classes, the setup of models simply loses those
monkey-patches, which explains why they stop working on custom models.
The fix introduces an `_unregister_hook()` method, that is expected to
clean up what has been done in `_register_hook()`. When the registry is
ready (i.e., not being loaded), the setup of models first invokes
`_unregister_hook()` on models, proceeds with the setup, and finally
invokes `_register_hook()` to reinstall the hooks.
OPW 2362308
closesodoo/odoo#60833
X-original-commit: 67152bf82da2674179297d32e4cec9dd534fa0c9
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Performance issue arise when the recordset is very large due to the union operator which apply at worst `len(self)` times.
By storing the ids then browse on them we avoid the union operation execution.
PySpy: https://drive.google.com/file/d/1jnxuSWGf2WmqeRz7zQOpBpmeRE5YnTwS/view?usp=sharing
filtered_domain (odoo/models.py:5398) take 65.6% of the graph (55.7% of the 85% of button_confirm)
To confirm a RFQ with 10 000 of the same product:
Before this commit: ~42s
After this commit: ~10s
OPW-2347525
closesodoo/odoo#60131
X-original-commit: 8b0eeb5076ed5387ca7132d79182800e10c6895e
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
This patch speeds up the performance of method `onchange`. Our use-case
is an invoice with 300 lines, where we modify the unit price or quantity
on several lines. Each modification triggers a call to `onchange` on
the invoice itself. The latter call went from 1.3 to 0.8 seconds, which
represents a speedup of 30% to 40%.
In the implementation of `onchange`, the first snapshot is preceded by a
"prefetching" phase, where the lines of x2many fields are read, so that
the fields of unmodified lines are in cache. This is useful because the
fields of those lines are not sent by the client, which only sends ids.
This prefetching represents more than 40% of the duration of `onchange`,
in our use-case.
The prefetching is inefficient for several reasons. First, it uses
`mapped`, which formats data that is actually never used. Second, it
accesses new records that have the actual lines as origin. And on those
records, computed stored fields are not taken from the origin record,
but are (uselessly) computed instead.
We have optimized the prefetching in the following way. It now reads
stored fields on the actual lines (using `_read` to avoid formatting),
then copies the cache of those fields on the corresponding new lines (to
avoid useless computations). This makes the prefetching about 10 times
faster!
closesodoo/odoo#59781
X-original-commit: ec50c426d7c2a17c7b72e64adab65eda6328e163
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
This patch optimizes the performance of prefetching when iterating on
large recordsets. The optimization is *transparent*, i.e., it requires
no code change for it to apply.
The worst-case scenario for prefetching is the following: the ORM has to
fetch a field for a given `record`, `record._prefetch_ids` (its prefetch
set) is *very* large (many thousands), and most records in the prefetch
set are already in cache. This requires the ORM to iterate a lot on the
prefetch set in order to make a batch of records not having the field in
cache.
records = model.browse(ids) # large recordset
for record in records:
record.foo # fetch 'foo' every 1k records
When running such a loop on an empty cache, the overhead of prefetching
(determine a batch) grows as the loop progresses. The time complexity
of this loop is actually O(N²)...
The overhead of prefetching is minimal when `record._prefetch_ids` is
about the size of a prefetching unit, i.e., 1k records. This commit
modifies the iterator method such that every record returned by the
iterator has a prefetch set of maximum 1k records.
We measured the time taken by the loop above on an empty cache, before
and after this commit, on a simple model (res.partner.category) with
100k records. The third measure is a reference one: a `_read` on all
prefetched fields (to fill in the cache) followed by the loop.
Total time Time per 1k records
Before this commit 3.690s 30ms - 45ms
After this commit 1.176s 12ms
Read then loop 1.161s -
The measures are enlightening: the overhead of prefetching was more than
200% of the reference time, and the time to prefetch records grows as
the iteration goes on! This commit reduces the overhead of prefetching
to less than 2% of the reference time, and make it scale gracefully with
data size.
closesodoo/odoo#59239
X-original-commit: d0d54d63dbf49a97e7fea36fe7fb4b860a0b6606
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The prefetching optimization was simply broken for x2many lines in the
snapshot method `diff()`. Indeed, iterating on `line._prefetch_ids`
actually returns nothing, because this iterable object relies on the
value in cache of the x2many field, and the cache has been invalidated
before calling `diff()`. In order to re-enable prefetching, we simply
populate the cache of x2many fields before looping on the lines.
Some testing with `account.move` and the relation `invoice_line_ids`:
For 335 invoice lines, changing the value of a field in the invoice
lines took about 1.9s with 1415 queries. It now takes about 1.1s and 82
queries.
closesodoo/odoo#58687
X-original-commit: 5bd7ddd0850e56fbbae52b4478970331f9db1589
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Calling a compute method on 1M records inevitably brings a worst case in
cache prefetching: the code that determines which records to fetch has
time complexity O(N²). We avoid this situation by calling the compute
method on maximum 1K records. Each computed batch has a "prefetch set"
of maximum 1K records too, which avoids the worst-case scenario above.
closesodoo/odoo#58664
X-original-commit: 0b1aa660b1d94fe46dea09670b875f2d81ca36a3
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Make sure that model dependencies are flushed before searching with a
given domain. This fixes inconsistencies in `search` and `read_group`.
closesodoo/odoo#57037
X-original-commit: 6b80632e98a932d2b552d6dec0fc7208ca8e04c1
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Make the method work when the value to test is `False`.
closesodoo/odoo#58064
X-original-commit: 6fb19219d333dc324603bcb2a937e977eecb037e
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The document layout preview is a complete defferent simplified template
with its own css that replicates at best the different styles.
It does not have the external layout features and lack of fidelity.
The new preview actually use the real documents templates and put the
result in an iframe. It now has a high fidelity, though not perfect.
The goal is for a better onboarding, where clients see easely how
documents will look if they had an app to generate them. Of course, the
data on the document is a false invoice.
Refactor all this from base to web.
Task ID 2304177
closesodoo/odoo#56995
X-original-commit: c121a246f16899735306266a3a12b526e08e7620
Signed-off-by: Lucas Perais (lpe) <lpe@odoo.com>
PURPOSE
Clean code. Be more performance oriented.
SPECIFICATIONS
Improvements applied in this commit
* not all() --> any(not) for earlier returns;
* all([generator]) --> all(generator) to avoid unnecessary list casting.
This code construct is better managed by all;
This commit will probably not have a big performance effect on standard
production databases. However each performance and cleaning improvement
is welcomed.
LINKS
Task ID-2328619
closesodoo/odoo#56810
X-original-commit: 1cc6bb1231401ea7f501d2f5b5e9641ec8734850
Related: odoo/enterprise#12802
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Currently the method `read` uses by default the parameter `load='classic_read')`
- `self.read(fields, load='classic_read')`
So, it computes `name_get` for all m2o fields for all records in `self`.
If you want to avoid computing the `name_get` to save time and process
You can use an empty string in load parameter:
- e.g. `self.read(..., load='')`
The method `self.search_read` call to `search` and `read` methods
but `search_read` method is not possible to assign `load=''` argument
(or other arguments of the method `read`)
- e.g. `self.search_read(..., load='')`
So, you need to use 2 lines of code:
records = self.search(...)
records.read(..., load='')
This commit changes `search_read` method to receive all keyword arguments of the method `read`
So, you can use `self.search_read(..., load='')` and the `read` parameter will be propagated
closesodoo/odoo#46391
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
TL;DR
=====
Improve performances with nearly a factor of 2. Use case, call action_validate on hr.leave for 100 employees
- 1987 requests -> 844 requests
- 1300 ms -> 700 ms
Purpose
=======
The first purpose of this commit is to add a test ensuring the number of request while creating
a company leave for 100 employees, if 15 of them already have a leave during that period.
It includes, the mass leave generation, and the conflicts resolutions. (Cancelling/Splitting the
already existing one and adapting the dates accordingly).
The second one is to reduce the number of request for this test.
In term of requests, currently we have:
- 5154 requests without bypassing the mail tracking + the activities management
- 1987 requests when bypassing the mail post-process (this is the current value, the bypassing was
already done several month ago)
- 844 requests with all the optimization done in resource/calendar/hr_holidays
In terms of execution time, we have a reduction from +- 1300 ms to call the method action_validate
to +- 700 ms
As the performances issues severity increases with the number of leaves to create and the real time access
to the database, on the production base, we reduced the execution time to generate more than 500 hr.leaves
from several minutes to 21 seconds. A fix to avoid deadlock was already made at
https://github.com/odoo/enterprise/pull/10740/files
Some contortions were made to avoid changing a signature method in a stable release and thus
introducing for each method a second one, with the "batched" implementation, to keep a retro-compatibility
for the existing custom code.
But surely this could be cleaned in the master version. The old one will be deprecated while waiting to be
removed in a few versions.
closesodoo/odoo#56534
Taskid: 2256705
X-original-commit: 8c96a887d3f05680c923dcd44f9c70c0e5e0bd32
Related: odoo/enterprise#12670
Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
Consider models `M` and `L`, with a one2many field `M.l_ids` with
inverse `L.m_id`. Consider also a computed field `L.foo` that depends
on `L.m_id`.
The field `L.foo` is not computed correctly by `onchange` on a new line
in the form view of `M`: its value is forced to `False` because it has
no default, and is not recomputed by `onchange` since the field `L.m_id`
is never triggered an `onchange`.
Modify the method `onchange` to not force fields without a default to
`False`, and return all fields.
When creating a new record, the client calls `default_get()`, completes
the returned values with `False`, and calls `onchange()` to apply the
onchange to the defaults.
Optimize the double round-trip by integrating `default_get()` inside
`onchange()` for the first call. The method is called with an empty
list of fields, and usually no field values, except for records in a
one2many field. In this case, `onchange()` does both steps above.
Task 2261084
By adding a parameter `_table_query` on the Model, we can now have views
that depend on the context. The query is used instead of the table name
in ORM operations. This allows to pre-compute some values to improve
performance, instead of storing context values in the database.
Co-authored-by: william-andre <wan@odoo.com>
Co-authored-by: rco-odoo <rco@odoo.com>
Make the method `_search` return a `Query` object, and make that object
generate a subquery when used on the right-hand side of a condition.
closesodoo/odoo#52403
Related: odoo/enterprise#10945
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Introduce better methods for introducing tables, join clauses and where
clauses to the Query object, and other methods to generate a complete
SELECT query. The Query object can also behave as a tuple of ids, where
the query is made on demand, and the result is memoized.
Raphael Collet