Commit Graph
430 Commits
Author SHA1 Message Date
Raphael Collet b08676389c [FIX] fields: ondelete being None on many2many field
The issue is simply creating a related stored many2many field, which is
pretty easy to do with Studio.  When creating the field, Odoo crashes
with a traceback caused by ondelete being None on the field.

The source of the bug is the fact that the attribute field.ondelete is
set to a sensible default ('cascade') on non-related fields only.  The
fix consists in setting the default value on the attribute itself, so
that the setup of the field never falls on a case where field.ondelete
is unset.

closes odoo/odoo#86303

X-original-commit: d0e0d80806a1b63ac986e9bcbaf88f9fc5243319
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-03-11 20:36:12 +00:00
Rémy Voet (ryv) e1785e820a [IMP] base: add prefetching group feature
The field prefetching mechanism was poorly customizable.  Before this,
we could only tell if a field was prefetched with other fields or not at
all.  We have no way to inform the framework, like: "When I need data of
that field, prefetch these other fields, which are likely be used in the
same transaction".

From now on, the `prefetch` attribute is used as a grouping key for
prefetching fields.  When a field is fetched, all the fields with the
same value for `prefetch` are taken for prefetching.

For example, consider a small set of fields that are rarely used, except
for one flow A using them.  You want to prefetch those fields only in
the flow A, and you want to fetch them in a single query.  With the new
feature, simply set `prefetch=A` for some string `A` on those fields,
and they will be grouped for prefetching.

closes odoo/odoo#85220

Signed-off-by: Rémy Voet <ryv@odoo.com>
2022-03-03 11:03:55 +00:00
Pierre-Yves Dufays 354b9e8ed2 [IMP] web, base: export field res_partner.name by default in import-compatible
mode

Purpose:

Include the name of the partners so that:
- it can easily be updated
- it is easier for users to recognize who is who

To generalize this modification, a new attribute has been added to the model
field descriptor (odoo/fields.py): default_export_compatible.
Setting this value to True on a field of a model will force that field to be
included by default in the exportation when "import-compatible export" is
selected.

Specification:

If the user goes:
    Contacts > List view > Select Records > Action Export
	     >  I want to update data
The field name is selected by default

The fields selected by default for export are the columns of the list so that
what is exported by default is what the user sees on the screen.
The display name is one of the column but is not importable.
When the option "I want to update data" is selected, only field that are
compatible for importation are selected and then display name is no longer
selected.
With this modification, the name is selected instead.

Technical:

- a new attribute "default_export_compatible" has been added in odoo/fields
- the controller /web/export/get_fields that lists the fields available
for exportation has been modified to add a default_export attribute on each
returned field. This new attribute is set to True when import_compat is True
and default_export_compatible is True on a given field.
The client uses that information to force by default the field for exportation.

Task 2734222

closes odoo/odoo#83697

Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2022-02-28 16:33:50 +00:00
Raf Geens 731d0d85c2 [FIX] orm: string pattern for Reference wasn't correct
In reality the separator is a comma instead of a dot as can be seen
in the code a bit further below.

closes odoo/odoo#85396

X-original-commit: 2a6e9f166d493aaefc006e949ab45e42311ac886
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2022-02-25 12:35:51 +00:00
Julien Castiaux e655e8da8a Revert "[IMP] base: add prefetching group feature"
This reverts commit 041fe5e21e.

Part-of: odoo/odoo#85199
2022-02-23 12:59:52 +00:00
Rémy Voet (ryv) 041fe5e21e [IMP] base: add prefetching group feature
The field prefetching mechanism was poorly customizable.  Before this,
we could only tell if a field was prefetched with other fields or not at
all.  We have no way to inform the framework, like: "When I need data of
that field, prefetch these other fields, which are likely be used in the
same transaction".

From now on, the `prefetch` attribute is used as a grouping key for
prefetching fields.  When a field is fetched, all the fields with the
same value for `prefetch` are taken for prefetching.

For example, consider a small set of fields that are rarely used, except
for one flow A using them.  You want to prefetch those fields only in
the flow A, and you want to fetch them in a single query.  With the new
feature, simply set `prefetch=A` for some string `A` on those fields,
and they will be grouped for prefetching.

Part-of: odoo/odoo#83818
2022-02-23 10:01:59 +00:00
Raphael Collet 39c1ab249b [FIX] core: discard redundant code
closes odoo/odoo#85028

X-original-commit: b026f17988dfaecac41f0a350cf8cfb52aa3827c
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-02-21 16:07:45 +00:00
Rémy Voet (ryv) 534087fa8d [REF] core: simplify tooling of browse()
Remove str from IdType, because those kinds of ids are no longer
necessary, and lead to unusable recordsets.

Because IdType now only contains int and NewId, simplify (and speed up)
the conversion of the parameter of browse().

Preferably wrap single NewId into a tuple instead of a list, as tuples
are faster.

closes odoo/odoo#83687

Signed-off-by: Raphael Collet <rco@odoo.com>
2022-02-21 10:28:15 +00:00
Rémy Voet (ryv) 6580fe0610 [REM] core: remove attribute 'limit' from X2many fields
Part-of: odoo/odoo#83687
2022-02-21 10:28:14 +00:00
Victor Feyens 65baadeb75 [IMP] core: do not try to translate selection values for nothing
When we are only trying to know accepted raw values, there is no
need to fetch the translations for the description part of the
selection tuples.

This performance problem was first observed on res.config.settings
record creation, where the values are pruned to avoid writes and
invalidation for unmodified related values (cf create override
in base/models/res_config.py & 5a4e7b711e).

After more analysis, the 'problem' also impacts record updates on
any related field (readonly or not).

* Res.config.settings performance gains:

In a database with test_main_flows (& its diverse dependencies),
this reduces by ~50% the number of queries done when triggering
the creation of res.config.settings record through the interface
(aka with all the related updatable fields receiving their current
value as create value).

For the creation of 150 res.config.settings records, with full cache
invalidation between each record creation, we notice a gain of:

* 11853 to 6303 total queries
* 11.2s to 7.5s total creation time

Part-of: odoo/odoo#83104
2022-02-08 11:07:46 +00:00
Victor Feyens 4d4108e3d1 [IMP] base: correctly prefetch currency fields on monetary updates
closes odoo/odoo#81423

Signed-off-by: Raphael Collet <rco@odoo.com>
2022-02-07 14:17:50 +00:00
a6f310fca3 [FIX] base: ignore tags in translations matching
When comparing existing translations to new source value
html tag would invalidate existing translations.
The html tags used for styling (coloring, italic, ...) should not
invalidate the translation as the meaning of the text as not changed.

Changed the logic to compare translations based on textual content only.

+ add some tests.

task-2667950

closes odoo/odoo#83895

X-original-commit: 1fe4b0cc1981382cc3d944ec44276f0aee90945d
Signed-off-by: Antoine Guenet <age@odoo.com>
Signed-off-by: Geelen Sébastien (sge) <sge@odoo.com>
Co-authored-by: rco-odoo <rco@odoo.com>
Co-authored-by: xmo-odoo <xmo@odoo.com>
2022-02-03 11:45:55 +00:00
Rémy Voet (ryv) 5fb6200f46 [REM] base: remove field attribute 'deprecated'
It was only used one time on a unused field.
We don't want to keep useless fields anymore,
the migration is done for that.

task-2735546
odoo/upgrade#3194

closes odoo/odoo#82727

Signed-off-by: Raphael Collet <rco@odoo.com>
2022-02-01 10:54:19 +00:00
Rémy Voet (ryv) ae41be0c5c [REM] base: remove column_format of Field class
The `column_format` of the `Field` class was unused and create useless
noise in the ORM. Then remove it and simplify some flows.

task-2735546

Part-of: odoo/odoo#82727
2022-02-01 10:54:18 +00:00
Raphael Collet a1904aa6f6 [IMP] core: field index names
The possible index names have been renamed "btree", "btree_not_null"
(instead of "not null") and "trigram" (instead of "gin").

Task 2742526

Part-of: odoo/odoo#83274
2022-01-28 14:10:01 +00:00
Rémy Voet (ryv) 5a573c6f18 [IMP] base: don't prefetch translate field by default
Issue
-----
Via the field prefetch mechanism, when we need a value of one field
(not in cache of course), the ORM will prefetch all fields
(which has the attribute to `prefetch=True`, the default value of this
attribute is `True`) for all record ids in `_prefetch_ids`.
Then, for each translate fields (where translate is not a callable)
the ORM need to make a `LEFT JOIN` on the `ir_translation` to fetch the
translated value. For big model, it leads to a simple `SELECT` with
several `LEFT JOIN` on ir_translation but each LEFT JOIN have a cost
in the planner time (a small cost in the execution time) of PostgreSQL.

By example, for `product.template` (stock/sale/purchase installed),
there are 6 LEFT JOIN to get all translated fields (5 of this
fields are rarely used).

Proposed solution
-----------------
Deactivate the prefetch by default for all translate fields expect if
this field is the `_rec_name` of the model (which is more likely to
be used).

In the example on the `product.template`:
Without prefetching the translated fields, there is only one LEFT JOIN
(the name, which is translated but is the `_rec_name` of the model).
With the 6 translated fields to fetch, the
query takes 5 ms to plan and 2 ms to execute VS with 1 translate field,
it 1 ms to plan and 1.5 ms to execute.

Side change note
----------------
- All translate of fields of `website.seo.metadata` should be prefetch
to avoid lot of website errors (it is because, website put in cache data
in sudo before reading it without sudo)
- `description` (`mail.message.subtype`), `subject` (`mail.template`),
`body_html` (`mail.template`) should be prefetch to avoid lot of extra
query from mail module.
- `vat_label` (`res.country`) should be prefetch to avoid a extra query
for each website page.
- Increase some queryCount (when it is legit, due to `subtitle` of
`blog_post` or `description` of `event.type.ticket`, etc)

task-2738029

closes odoo/odoo#82896

Signed-off-by: Raphael Collet <rco@odoo.com>
2022-01-28 14:09:56 +00:00
Fabien Pinckaers 6b87526048 [IMP] Speed Imp: remove unnecessary base64 encode & decode
Avoid to base64 encode, then decode to process assets and images for a ~25% speed improvement.
Change image processing tool to work on images, rather than base64 encoded strings.

Performance is ~25% faster on assets & images:

  /web/assets/...frontend.min.css:    13ms to 7ms,  base64 enc/dec: 2 -> 0
  /web/image/XML_ID:                  10ms to 8ms,  base64 enc/dec: 3 -> 0
  /web/image/res.users/2/avatar_128:  40ms to 20ms, base64 enc/dec: 6 -> 2

closes odoo/odoo#82851

Related: odoo/enterprise#23537
Signed-off-by: Fabien Pinckaers <fp@odoo.com>
2022-01-22 11:51:42 +00:00
Fabien Pinckaers eedf37d6e2 [IMP] Better handling of indexes
Three supported types:
- btree (default for index=True)
- btree not null (when >90% of the data are null)
- gin trigram search (for char fields)

Review of indexes on all objects.

closes odoo/odoo#83015

Signed-off-by: Fabien Pinckaers <fp@odoo.com>
2022-01-19 16:52:23 +00:00
Rémy Voet (ryv)andrco-odoo 504589efb2 [FIX] base: add __reversed__ in BaseModel for efficiency
Before when we do `reserved(records)`, it iterates in a reverse
order of `records`. Unfortunately, the `__reserved__` method don't
exist explicitly in BaseModel then Python fallback on
its own implementation using `__getitem__` and `__len__` (coming from Sequence): https://github.com/python/cpython/blob/3.10/Lib/_collections_abc.py#L1047-L1049
Because it uses __getitem__, it breaks the prefetch of the recordset.

Example:
-------------
```
partners = self.env['res.partner'].browse(1, 2, 3, 4, 5)
for partner in reversed(partners):
    partner.name
```
will generate 5 SQL requests to fetch data (one by record)

Then create our own `__reversed__` and handle the prefetch correctly
(like `__iter__`). Now in the example it will correctly generate only
1 SQL request because of the prefetch.

task-2687953

closes odoo/odoo#79622

Signed-off-by: Raphael Collet <rco@odoo.com>
Co-authored-by: rco-odoo <rco@odoo.com>
2022-01-12 10:17:36 +00:00
Rémy Voet (ryv) 8b7c8d0b71 [REF] core: replace _browse() by __init__()
The creation of recordset was done by class method _browse() instead of
a regular call to the model's class.  As we removed the old usage of
method __init__(), we can now reuse it with a normal usage. After this
commit, we can create a recordset by simply calling its class:

    registry['model_name'](env, ids, prefetch_ids)

closes odoo/odoo#79563

Signed-off-by: Raphael Collet <rco@odoo.com>
2021-12-24 14:56:58 +00:00
Raphael Collet 4347491670 [FIX] core: unexpected display_name "False"
If a form view contains the field 'display_name', when creating a new
record, the initial value of 'display_name' is the string "False", and
that weird value also appears in the breadcrumb instead of "New".  In
order to avoid this unexpected behavior, the conversion of a value to a
display name should be False, like any other field would.

closes odoo/odoo#81788

Signed-off-by: Raphael Collet <rco@odoo.com>
2021-12-22 09:39:08 +00:00
Xavier Morel bdc9d9d369 [FIX] core; base: lots of docstrings
* add configuration for `flake8[flake8-rst-docstring]`
* enable docstring-related checks
* fix invalid docstrings in odoo's core & `base`
* fix a few more bits (mostly missing or incorrect `:param:` info
  fields) are out of scope for the lint but my editor catches

closes odoo/odoo#74604

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2021-12-09 14:36:58 +00:00
Yannick Tivisse 87224e158c [IMP] core: precompute docstring
Part-of: odoo/odoo#80687
2021-12-01 17:09:00 +00:00
d04a5b5c8c [REF] core: compute fields before database insertion.
Until now, stored compute fields were computed after database insertion.
This meant that required fields should not be computed, for instance,
unless some hackish code was added to make it work.  Another trick was
to provide some default value, but this actually prevents the field for
being computed after insertion.

This commit provides a field parameter to specify that the field should
be precomputed: adding precompute=True on the field definition force the
method create() to compute its value before inserting the new record in
the database.  For the reason explained below, precomputing fields is
not always correct, and therefore the default remains to not precompute
a field.

Some stored fields must be computed after insertion, for instance:
* statistics fields computed with search/read_group/...
* fields referencing the current record (res.partner.commercial_partner_id)
* fields referencing another record that does not exist yet (think about
  records created by one2many fields)
* fields depending on the create_date/write_date/create_uid/write_uid

Those fields shouldn't be defined with precompute=True, which triggers
their computation post record creation.  This is why, by safety, we
consider the default behavior to be precompute=False.

closes odoo/odoo#80449

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Yannick Tivisse <yti@odoo.com>
2021-11-30 14:30:48 +00:00
Raphael Collet aa0cdacced [FIX] core: related field attributes
A related field copies the attributes from its target field, except for
the attributes defined on the related field itself.  The implementation
of this feature was not working properly for attributes with a truthy
default value.

closes odoo/odoo#79025

X-original-commit: dec4a7ec478fa02f19dee8c8426c88d17dc3c7f3
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-10-26 16:06:48 +00:00
Xavier-DoandRaphael Collet d4539c188a [IMP] core: use cache for mro in setup_models
With a lot of module installed setup_models can be quite slow.
One of the reason for that are the call to resolve_mro.

This operation is fast, but called for a lot of models/fields this
become an important part of get_depends and setup_base.

- This is call once for each models in setup_base
- This is call more than once for each computed_field in get_depends.

The use of a cache shows significative performance improvements.

During the install of a database with all modules, the install times
is reduced by ~3 minutes over ~14

Part-of: odoo/odoo#78898
Co-authored-by: Raphael Collet <rco@odoo.com>
2021-10-26 13:22:40 +00:00
Raphael Collet 56660e39e6 [FIX] core: one2many linking non-existing lines
The processing of the LINK command in a one2many should fail when the
line being linked does not exist.  However, there is one case where it
should not fail: when that line existed and was linked before applying
the commands.  That use-case may seem strange, but it actually exists:
deleting a line in a sales order automatically deletes the corresponding
reward lines.

closes odoo/odoo#78318

X-original-commit: 4bfe1b5cab10d3171d386d76799a7d7729f49154
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-10-13 18:24:42 +00:00
william-andre d3f4ce1152 [IMP] core: allow set [value] in selection ondelete
In some cases we migh want to change to a value that isn't the default
one when uninstalling.
For instance: when we uninstall the module `event_sale`, a product has
the field `detailed_type` set to `event`, which is a subtype of `service`.
So we want to update the value to `service` when uninstalling instead of
the default value, which would be `consu` and wouldn't make any sense.

Part-of: odoo/odoo#77876
2021-10-11 10:15:48 +00:00
Xavier Morel b8da632e6e [IMP] core, base: allow disabling unaccent on a per-field basis
For searches, Odoo uses `unaccent` if it's available. On some
technical fields this is completely unnecessary and precludes the use
of indexes.

This PR provides:

* an opt-out (`unaccent = False`) on `String` and `Text` fields
* a warning if `unaccent` is enabled on *parent_path* fields as their
  performance can be rather critical and not using the index is quite
  an issue (note: the check that `parent_path` fields have been moved
  outside of the check for their existence as we want to check that
  the field is declared and correctly configured in all cases,
  probably)

Task 2627454

Part-of: odoo/odoo#76436
2021-09-20 12:53:01 +00:00
Julien Castiaux b8b89bf47d [FIX] core: prevent redundant default on related
Setting a default value on a readonly related field without an inverse
method is nonsense.  We log some warning when it happens.

We also fixed other cases where a related field has a default value
that overrides the target field's value.

closes odoo/odoo#67762

Related: odoo/enterprise#17313
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
2021-09-07 15:49:48 +00:00
Raphael ColletandWilliam André 71fc2e55f9 [FIX] core: inherited computed fields in onchange
Assume that model A has a computed field G that depends on both fields
F1 and F2.  Assume that model B inherits from A (_inherits).  When G is
computed during an onchange, it must use the form values for all its
dependencies F1 and F2.  Before this commit, only the field triggering
the onchange was assigned on the parent record (see (*) below.)

                      record        parent
    --- ------------+-------------+------------
     initial state  | F1=0, F2=0  | F1=0, F2=0
     assign F1=1    | F1=1, F2=0  | F1=1, F2=0
     assign F2=2    | F1=1, F2=1  | F1=0, F2=1 (*)

The commit also includes another patch: when assigning the parent
record, the ORM determines the dependent records (in this case, the main
record in the onchange).  If the delegate field (many2one from B to A)
has an inverse one2many field, the ORM uses that field to determine
dependent records.  However, in the case of a new record, that field is
not in cache, and its value is determined to be... empty!  The patch
consists in "fixing" the value of that field when we determine the
parent record (when the delegate field is accessed.)

Part-of: odoo/odoo#76042
Co-authored-by: William André <wan@odoo.com>
2021-09-06 18:29:03 +00:00
Adrian Torres 9f11a84d71 [IMP] Order groups by selection declaration order in read_group
Before this commit, calling read_group on a model and grouping by a
selection field would return the groups in alphabetical order, meaning
that if your selection field's options were declared as:

('z', 'Z'), ('a', 'A')

read_group would return first group 'a' and then group 'z', instead of
groups 'z' and then 'a' which some might expect.

With this commit, it is now possible to set a fields.Selection
group_expand attribute to `True` when declaring it, which will use a
default group_expand implementation specific to Selection fields, this
means that when grouping by a Selection field with `group_expand=True`
it will always return the groups in the definition order of the
selection options.

We achieve this by leveraging the group_expand field attribute which was
designed for changing the groups returned by read_group.

Since this attribute was thought only to be implemented on a
Model-by-Model basis and here we need to use it as a generic function
for all Selection fields (explicit group_expand declarations have higher
precedence), the generic method has been implemented inside
fields.Selection and takes an extra records parameter which holds a
reference to the recordset/model on which read_group was called. This
extra parameter only applies to field implementations of group_expand
and in this case is what allows this specific feature to work with
dynamic Selection fields (function as options).

A side-effect of this implementation is that a read_group call that
groups by a Selection field with `group_expand=True` that uses the
default group_expand will always return all possible groups, even
empty ones.

Task-ID 2635052

closes odoo/odoo#75856

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-09-06 08:21:34 +00:00
Ivan Yelizariev f9f5f889e4 [FIX] fields: allow to define several m2m with same relation but domain
Partial of #67648

Part-of: odoo/odoo#72559
2021-09-02 12:57:08 +00:00
Florent de Labarre 9e1e0ebb02 [FIX] core: better error message on wrong related path definitions
Before this commit, if a related field's path definition contained a
non-existing field (either because of a typo or field removal) the ORM
would simply send a generic Python KeyError that didn't really help in
debugging.

With this commit, we raise our own KeyError stating which related field
has the wrong path definition and exactly which field within the path is
incorrect.

See test for an example.

closes odoo/odoo#31889

Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
2021-07-16 10:17:30 +00:00
Raphael Collet ddcabf3acb [FIX] core: origin of main record in onchange
When invoking onchange() on a line in a one2many field, the inverse
many2one field is given as a dict of values.  Those values are the ones
of the "container" record in the form view, and they include the "id" of
the container record.  The method onchange() instantiates the container
record as a new record with the given values.  Fix the code to set the
expected "origin" of that record to the record with the given "id".

closes odoo/odoo#71491

X-original-commit: ebfb5d90fd4ff65ee1d2fc73faf8f9eaa18f4521
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-05-31 12:01:55 +00:00
Raphael Collet 8ff079366d [FIX] core: computed editable one2many field with a domain
Manually adding a line should not trigger the computation of the field,
just like modifying the line fields that occur in the field's domain.
In other words, the dependencies of the field should be limited to the
ones declared on field or its compute method.

closes odoo/odoo#71049

X-original-commit: 1c39814bd729cc08882f1545c7d88b0592e63f9a
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-05-19 16:29:53 +00:00
Raphael Collet 4fb958d086 [IMP] core: group all field_inverses dicts on registry
This makes the union of many dicts into a single dict.

On a registry with 296 modules, this saves 300 kilobytes of memory,
which is about 3% of the registry's memory footprint.
2021-05-10 14:29:43 +00:00
Raphael Collet e2c4167826 [IMP] core: do not put '_sequence' in field.args
This does not save memory in registries.
2021-05-10 14:29:43 +00:00
Raphael Collet 5ae684f79d [IMP] core: do not assign default values on fields
This keeps field.__dict__ as small as possible.

On a registry with 296 modules, this saves 40 kilobytes of memory,
which is about 0.35% of the registry's memory footprint.
2021-05-10 14:29:43 +00:00
Raphael Collet 1abe965b59 [IMP] core: keep field.related as a string
This mainly keeps field.related's type consistent.

On a registry with 296 modules, this saves 325 kilobytes of memory,
which is about 3% of the registry's memory footprint.
2021-05-10 14:29:42 +00:00
Raphael Collet dab2a7cd52 [IMP] core: make field._modules a tuple instead of a set
On a registry with 296 modules, this saves 700 kilobytes of memory,
which is about 5% of the registry's memory footprint.
2021-05-10 14:29:05 +00:00
Raphael Collet f7c9cb2b60 [IMP] core: share fields by not duplicating them on model class
Also speed up the basic setup of fields that are always duplicated
top-level (on the model's registry class).  This saves time and memory,
as we also discard field.args and field._base_fields on toplevel fields
(those values are no longer useful after setup).
2021-05-03 12:33:29 +00:00
Raphael Collet 81ff2717cf [REM] core: remove optimization sharing fields across registries
The optimization will be reintroduced later in a different form.
2021-05-03 12:33:29 +00:00
Raphael Collet 960cd72774 [REF] core: add method get_currency_field() on Monetary to retrieve it
This replaces the setup of the attribute 'currency_field', which depends
on the presence of other fields on the model, and may therefore vary
from one registry to another.  This is necessary to make monetary fields
shareable across registries.
2021-05-03 12:33:29 +00:00
Raphael Collet 34d6f87d54 [REF] core: put field.depends on registry and make field.recursive explicit
The attributes field.depends and field.depends_context are problematic
for sharing fields across registries, because they depend on the model's
registry class, which may vary from one registry to another.  In order
to make computed fields shareable, we have to move those values away
from fields.

For the same reason, field.recursive should not be inferred, because its
value may depend on the registry, although it is generally not the case.
Moreover, the flag recursive=True is set on a field when field triggers
are determined (on the registry).  A compute method may be called before
the flag is set (if no update has been done yet), and that can lead to
incorrect computations.

This happened in test TestUsers2.test_reified_groups in module 'base'.
The user groups view was apparently determined without the flag being
set, and the view depends on the recursive field 'trans_implied_ids',
which was not correctly computed.

We thus force developers to be explicit about recursive computed fields.
The code now logs a warning when the flag is not set up properly.
2021-05-03 12:33:29 +00:00
Raphael Collet 1ca2c7e457 [REF] core: new field setup API
The basic setup of fields is now made in Field.__set_name__(), which
automatically makes this information shared across registries, because
it is done when importing Odoo modules.
2021-05-03 12:33:29 +00:00
Raphael Collet f934b29a8a [REF] core: speed up the retrieval of fields on models
Fields are no longer retrieved with inspect.getmembers(), which is
time-consuming.  Instead, the classes defining models automatically
collect their own fields (via Field.__set_name__()), and the field
retrieval is done by using those collections.  Also, a field no longer
needs to introspect its model class to find its field definitions.

A class that defines a model automatically determines its '_name',
'_inherit' and '_module' upon creation, and the fields determine their
'name', 'model_name' and '_module' via Field.__set_name__().

Loading a registry with 290 modules from scratch is now 25% faster.
2021-05-03 12:33:29 +00:00
Xavier Morel 01875541b1 [CHG] core, web: deprecate t-raw
Add a big fat warning when the qweb compiler finds a `t-raw`.

`t-esc` should now be used everywhere, the use-case for `t-raw` should
be handled by converting the corresponding values to `Markup`
objects. Even though it's convenient, this constructor *should never
be made available in the qweb rendering context* (maybe that should be
checked for explicitely?).

Replace `werkzeug.escape` by `markupsafe.escape` in
`odoo.tools.html_escape`, this means the output of `html_escape` is
markup-safe.

Updated qweb to work correctly with escaping and `Markup`, amongst
other things QWeb bodies should be markup-safe internally (so that a
`t-set` value can be fed into a `t-esc`). See at the bottom for the
attributes handling as it's a bit complicated.

`to_text` needed updating: `markupsafe.Markup` is a subclass of `str`,
but `str` is not a passthrough for strings. So `Markup` instances
going through would be converted to normal `str`, losing their safety
flag. Since qweb internally uses `to_text` on pretty much
everything (in order to handle None / False), this would then cause
almost every `Markup` to get mistakenly double-escaped.

Also mark a bunch of APIs as markup-safe by default

* html_sanitize output.
* HTML fields content, sanitization is applied on intake (so stripped
  by the trip through the database) and if the field is unsanitised
  the injection is very much intentional, probably. Note: this
  includes automatically decoding bytes as a number of default values
  & computes yield bytes, which Markup will happily accept... by
  repr-ing them which is useless. This is hard to notice without `-b`.
* Script-safe json, it's rather the point (though it uses a
  non-standard escaping scheme).
* Note that `nl2br`, kinda: it should work correctly whether or not
  the input is markup-safe, this means we should not need to escape
  values fed to `nl2br`, but it doesn't hurt either.

Update some qweb field serialisations to mark their output as
markup-safe when necessary (e.g. monetary, barcode,
contact). Otherwise either using proper escaping internally or doing
nothing should do the trick.

Also update qweb to return markup-safe bytes: we want qweb to return
markup-safe contents as a common use-case is to render something with
one template, and inject its content in an other one (with Python code
inbetween, as `t-call` works a bit differently and does not go through
the external rendering interface).

However qweb returns `bytes` while `Markup` extends `str`. After a
quick experiment with changing qweb rendering to return `str` (rather
unmitigated failure I fear), it looks like the safest tack is to add a
somewhat similar bytes-based type, which decodes to a `Markup` but
keeps to bytes semantics.

For debugging and convenience reasons, MarkupSafeBytes does *not*
stringify and raises an error instead (`__repr__` works fine). This is
to avoid implicit stringifications which do the wrong thing (namely
create a string `"b'foo'"`).

Also add some configuration around BytesWarning (which still has to be
enabled at the interpreter level via `-b`, there's no way to enable it
programmatically smh), and monkeypatch `showwarning` to show warning
tracebacks, as it's common for warnings to be triggered in the bowels
of the application, and hard to relate to business logic without the
complete traceback.

`t-out`
=======

`t-esc` is a bit confusing for the new behaviour of "maybe escape
maybe not", so add a `t-out` alias with the same behaviour.

Unlike `t-raw`, `t-esc` is only soft-deprecated for now: there are
thousands of instances, so editing all the templates is not
great. Eventually we'll add a `ci/style` to prevent addition of new
ones, and eventually we might do a bulk-replace and hard-deprecate.

Attributes handling
===================

There are a few issues with respect to attributes. The first issue is
that markup-safe content is not necessarily attributes-safe
e.g. markup-safe content can contain unescaped `<` or double-quotes
while attributes can not. So we must forcefully escape the input, even
if it's supposedly markup-safe already.

This causes a problem for script-safe JSON: it's markup-safe but
really does its own thing. So instead of escaping it up-front and
wrapping it in Markup, make script-safe JSON its own type which
applies JSON-escaping *during the `__html__` call.

This way if a script-safe JSON object goes through `markupsafe.escape`
we'll apply script-safe escaping, otherwise it'll be treated as a
regular strings and eventually escaped the normal way.

A second issue was the processing of format-valued
attributes (`t-attf`): literal segments should always be markup-safe,
while non-literal may or may not be. This turns out to be an issue if
the non-literal segment *is* markup-safe: in that case when the
literal and non-literal segments get concatenated the literal segments
will get escaped, then attributes serialization will escape
them *again* leading to doubly-escaped content in attributes.

The most visible instance of this was the `snippet_options` template,
specifically:

    <t t-set="so_content_addition_selector" t-translation="off">blockquote, ...</t>
    <div id="so_content_addition"
        t-att-data-selector="so_content_addition_selector"
        t-attf-data-drop-near="p, h1, h2, h3, .row > div > img, #{so_content_addition_selector}"
        data-drop-in=".content, nav"/>

Here `so_content_addition_selector` is a qweb body therefore
markup-safe, When concatenated with the literal part of
`t-atff-data-drop-near` it would cause the HTML-escaping of that
yielding a new Markup object. Normal attributes processing would then
strip the markup flag (using `str()`) and escape it again, leading to
doubly-escaped literals.

The original hack around was to unescape() `Markup` content before
stringifying it and escaping it again, in the attribute serialization
method (`_append_attributes`).

That's pretty disgusting, after some more consideration & testing it
looks like a much better and safer fix is to ensure the
expression (non-literal) segments of format strings always result in
`str`, never `Markup`, which is easy enough: just all `str()` on the
output of strexpr. We could also have concatenated all the bits using
`''.join` instead of repeated concatenation (`+`).

Also add a check on the type of the format string for safety, I think
it should always be a proper str and the bytes thing is only when
running in py2 (where lxml uses bytestrings as a space optimization
for ascii-only values) but it should not hurt too much to perform a
single typecheck assertion on the value... instead of performing one
per literal segment.

Note: we may need to implement unescape anyway, because it's still
possible to get double-escaping with the current scheme: given an
explicitly escape-ed `foo` and `t-att-foo="foo"`, `foo` will be
re-escaped.

fixup! [CHG] core, web: deprecate t-raw
2021-04-29 05:34:19 +00:00
Raphael Collet f7d5d11238 [IMP] core: do not add magic and inherited fields on abstract models
Magic and inherited fields are not really useful on abstract models.
The _inherits specification is used anyway by models that inherit from
those abstract models.

The main goal of this change is to prepare a refactoring of models where
fields are no longer duplicated on the registry classes, but fields
defined on classes are used directly.  But this new design cannot be
applied to all fields: a field being overridden simply cannot be used
directly.  This branch improves the situation by avoiding unnecessary
field overridings.

closes odoo/odoo#69372

Related: odoo/upgrade#2409
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-04-22 08:59:36 +00:00
Adrian TorresandRaphael Collet b4487e8ea4 [IMP] core: allow grouping by m2m fields in read_group
With this commit, it is now possible to group the records of a model by
a Many2many field of said model.

The result of a read_group grouped by a m2m will return as many entries
or "groups" as there are different records in the comodel that are
linked to the model through the m2m, plus a null/false group, for records
of the model that have no linked records of the comodel or for records
of the comodel for which the current user has no read access to due to
ir.rules.

For a more illustrated explanation, the tests should cover all cases in
detail.

Note that this commit only introduces this change at the ORM level, the
frontend does not yet handle grouping by m2m fields but it is planned
in the near future.

Task-2428971

closes odoo/odoo#68958

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2021-04-20 12:08:35 +00:00