Commit Graph
119 Commits
Author SHA1 Message Date
Rémy Voet (ryv) 87a1ebb338 [FIX] core: fix create/write on binary fields
When invoking create() or write() with a binary field, the cache of the
field was incorrect if bin_size=True was in context.  Force context with
bin_size=False when putting a binary value in cache.  It is particularly
important to have coherent values in the cache for `web_save`.

Also, because an environment with bin_size=False won't return the same
context cache key as one with bin_size=None, it leads to have a cache
inconstistency when we write with bin_size=False.  Change Environment
method cache_key() to return the same cache key when bin_size is absent,
bin_size=None or bin_size=False.

Tests on binary fields have been updated to not rely on flush and
invalidate.  We also created specific tests for write() on binary
fields.

Part-of: odoo/odoo#160708
2024-04-26 17:17:52 +00:00
Rémy Voet (ryv) a0bf434960 [FIX] core: add invalidation of Environment's _cache_key.
Changing the environment in method create() to force bin_size=False
looks harmless, but it actually breaks many tests, in particular in
module account.  The reason is that company_dependent fields are read at
the wrong place in the cache.  And this is because `env._cache_key` can
be polluted with old data.

Make sure that `_cache_key` is cleared when resetting all the lazy
properties on the environment.  Only the change in res_user.py makes it
work, but let's not tempt the devil.

Side note: I hate caches.

Part-of: odoo/odoo#160708
2024-04-26 17:17:52 +00:00
Thomas Lefebvre (thle) 176f5ec4a2 [FIX] api: assert cr is a cursor when creating a new environment
`Environment.__new__` expects a Cursor.
Ensure developers pass a Cursor and not another kind of unexpected object.
Passing another object with the same attributes would work during
the creation of the new environment, but then would fail later,
when using the created environment with the wrong `cr` attribute,
with a less comprehensive error.

Task-3796479

closes odoo/odoo#80644

Signed-off-by: Raphael Collet <rco@odoo.com>
2024-02-13 09:07:38 +00:00
Raphael Collet b00d135599 [FIX] core: optimization of Cache.set() to avoid multiple record.id
The expression record.id relies on a Python descriptor that has some
overhead, which is small but not negligible when used in a low-level
method of the ORM.  We simply factor out this expression in order to
evaluate it once for the method.

closes odoo/odoo#149624

Signed-off-by: Raphael Collet <rco@odoo.com>
2024-02-01 09:49:50 +00:00
Rémy Voet (ryv) 65a2c2ebc8 [FIX] core: new record shouldn't force fetching inverse x2many fields
In e0297bdac4, the creation of a new
record always patches the inverse fields of relational fields in order
to make the cache of those inverse fields consistent.

For instance, when creating a new record like

    user = model.new({'group_ids': [Command.link(group.id)]})

The inverse of field 'group_ids' on the new record having 'group' as
origin is patched so that its value includes record.  A side effect of
this mechanism is that it fetches group.user_ids in order to patch the
value of new_group.user_ids, where 'new_group' is the new record having
'group' as origin.

The side effect described above is problematic when that inverse field
has huge cardinality, like hundreds of thousands of records, and this
performance overhead is unacceptable when the inverse field is actually
not used at all.

We address this performance issue by patching the value of x2many fields
only when they are used.  If the value of the field is not in cache yet,
the patch is applied once a value is put in cache.  If the field is not
used, the patch is simply never applied.

Part-of: odoo/odoo#149624
2024-02-01 09:49:50 +00:00
Denis Ledoux ec0cd75328 [IMP] core: disable x2many manipulation as sudo for sensitive models
This revision adds a flag
`_allow_sudo_commands` to which models can opt-in
to protect themselves against malicious
manipulation of one2many or many2many fields
through an environment using `sudo` or a more priviledged user.

Flagging a model `_allow_sudo_commands = False`
disable `sudo` and `with_user(...)`
when manipulating a one2many or many2many
field targeting this model.

task-3695103
2023-11-01 08:05:16 +00:00
bve-odoo 72c1a4f96a [IMP] *: replace type where possible
do not compare types, for exact checks use `is` / `is not`,
for instance checks use `isinstance()`Flake8(E721)

Signed-off-by: Rémy Voet (ryv) <ryv@odoo.com>
2023-11-07 17:09:34 +00:00
Raphael Collet 425a6054ec [FIX] core: modified() for context-dependent non-stored recursive fields
When recursing on non-stored recursive fields, modified() only considers
the records that have some value in cache.  But when fields are context-
dependent, we may miss some records because we look up for cache values
in the wrong context.  Instead, consider cache values in all contexts
for that matter.

closes odoo/odoo#143644

X-original-commit: f19732c199e6bb2b8bdd55808658cb82c8cd44b4
Signed-off-by: Rémy Voet (ryv) <ryv@odoo.com>
Signed-off-by: Raphael Collet <rco@odoo.com>
2023-11-25 06:43:38 +00:00
Chong Wang (cwg) 2d08f97c07 [IMP] core: support delay translation
Jsonb data structure of model_terms translated fields' columns
'{
	"en_US": "<div>Apple</div>"
	"fr_FR": "<div>Pomme</div><div>Banane</div>"
	"_fr_FR": "<div>Pomme</div>"
}'::jsonb

"column" IS NULL OR "column"->>'en_US' IS NOT NULL

"column"->>'lang'
1. stores last confirmed value
2. logically fallbacks to
    COALESCE(
        "column"->>'lang',
        "column"->>'en_US'
    )

"column"->>'_lang'
1. stores translations and the last written html/xml structure
2. shares the same html/xml structure with other "column"->>'_lang'
3. logically fallbacks to
    COALESCE(
        "column"->>'_lang',
        "column"->>'lang',
        "column"->>'_en_US',
        "column"->>'en_US'
    )

Context:
1. `delay_translations`(new) only write values to _langs while keeping
translations for langs when the written value has at least one translatable term
2. `check_translations`(new) read _langs values to create translation mapping
for the translation dialog
3. `edit_translations` read values whose translatable terms are wrapped by
`<span></span>` with term information for the TRANSLATE mode of website

Potential issue
since the record value of a model terms translated field is logical content
dependent (check_translations, edit_translations), all computed field computed
from any model terms translated field should also be marked.
@api.depends_context('lang', 'edit_translations', 'check_translations')

Data flow for column value, cache value and record value
(make sure the window is wide enough to see the graph)

                             +- - - - - - - - - - - - - - - - - - - - - - - - - - - -+
                             ' Record(str):                                          '
                             '                                                       '
                             ' +---------------------------------------------------+ '               +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
                             ' | "<div><span data-oe-model='model'                 | '               {            read fr_FR             {
                             ' | data-oe-id='id' ...>French</span></div>"          | ' <------------ } context.get('edit_translations')  } <+
                             ' +---------------------------------------------------+ '               +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+  |
                             ' +---------------------------------------------------+ '               +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+  |
                             ' | "<div>French</div>"                               | '               {            read fr_FR             {  |
                             ' |                                                   | ' <------------ } context.get('check_translations') } <+
                             ' +---------------------------------------------------+ '               +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+  |
+~~~~~~~~~~~~+               ' +---------------------------------------------------+ '                                                      |
{ read fr_FR { ------------> ' | "<a>French</a>"                                   | '                                                      |
+~~~~~~~~~~~~+               ' +---------------------------------------------------+ '                                                      |
  ^                          '                                                       '                                                      |
  |                          +- - - - - - - - - - - - - - - - - - - - - - - - - - - -+                                                      |
  |                                                                                                                                         |
  |                                                                                                                                         |
  |                                                                                                                                         |
  |                          +- - - - - - - - - - - - - - - - - - - - - - - - - - - -+                                                      |
  |                          ' Cache(dict):                                          '                                                      |
  |                          '                                                       '                                                      |
  |                          ' +---------------------------------------------------+ '                                                      |
  |                          ' |                                                   | ' -----------------------------------------------------+
  |                          ' | "_fr_FR": "<div>French</div>"                     | '
  |                          ' |                                                   | ' <----------------------------------------------------+
  |                          ' +---------------------------------------------------+ '                                                      |
  |                          ' +---------------------------------------------------+ '               +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
  +------------------------- ' |                                                   | '               {              fetch fr_FR               {
                             ' | "fr_FR": "<a>French</a>"                          | '               }    context.get('edit_translations')    }
  +------------------------> ' |                                                   | '               {  or context.get('check_translations')  {
  |                          ' +---------------------------------------------------+ '               +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
+~~~~~~~~~~~~~~~~~+          '                                                       '                                                      ^
{   fetch fr_FR   {          +- - - - - - - - - - - - - - - - - - - - - - - - - - - -+                              COALESCE(               |
+~~~~~~~~~~~~~~~~~+                                                                                                     "c"->>'_fr_FR',     |
  ^                                                                                                                     "c"->>'fr_FR',      |
  | COALESCE(                                                                                                           "c"->>'_en_US',     |
  |     "c"->>'fr_FR',       +- - - - - - - - - - - - - - - - - - - - - - - - - - - -+                                  "c"->>'en_US'       |
  |     "c"->>'en_US'        ' Database(jsonb):                                      '                              )                       |
  | )                        '                                                       '                                                      |
  |                          ' +---------------------------------------------------+ '                                                      |
  |                          ' | "_fr_FR": "<div>French</div>"                     | ' -----------------------------------------------------+
  |                          ' +---------------------------------------------------+ '                                                      |
  |                          ' +---------------------------------------------------+ '                                                      |
  +------------------------- ' | "fr_FR": "<a>French</a>"                          | ' -----------------------------------------------------+
  |                          ' +---------------------------------------------------+ '                                                      |
  |                          ' +---------------------------------------------------+ '                                                      |
  |                          ' | "_en_US": "<div>English</div>"                    | ' -----------------------------------------------------+
  |                          ' +---------------------------------------------------+ '                                                      |
  |                          ' +---------------------------------------------------+ '                                                      |
  +------------------------- ' | "en_US": "<div>English1</div><div>English2</div>" | ' -----------------------------------------------------+
                             ' +---------------------------------------------------+ '
                             '                                                       '
                             +- - - - - - - - - - - - - - - - - - - - - - - - - - - -+

Task: 3043340
Part-of: odoo/odoo#139819
2023-10-27 11:35:03 +00:00
Victor Feyens 7a02a6a81e [IMP] core: docstring improvements
* clean and improve docstrings in orm
* fix typos found with codespell
* rely on the Environment class docstring instead of doc content (and
therefore move part of the doc inside the class docstring)

closes odoo/odoo#102969

Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
2023-08-07 09:36:23 +00:00
Victor Feyens 8ce3291eaf [IMP] core: docstring improvements
* clean and improve docstrings in orm
* fix typos found with codespell
* rely on the Environment class docstring instead of doc content (and
therefore move part of the doc inside the class docstring)

closes odoo/odoo#102969

Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
2023-08-09 17:56:35 +00:00
Xavier Morel 39be91b48e [IMP] core: clean up the ACL tracebacks some
Currently if a record is not in the cache it raises a `CacheMiss` the
handling of which leads to loading the record from the database (or
something).

If an issue occurs during that loading, because the loading is an
`except` scope the cache miss gets linked with the new one via

> During handling of the above exception, another exception occurred:

This is both noise (the cache miss is not actually relevant) and
misleading, because the wording makes it look like an unrelated error
occurred during handling.

- move the loading of the record out of the `except` to limit the
  scope of the `KeyError` and avoid "inheriting" it
- try to `from None` a few specific errors to remove implicit linkage,
  as the new error should have all relevant information, the key error
  / cache miss is just an implementation detail

closes odoo/odoo#139680

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2023-10-25 19:37:31 +00:00
Raphael Collet 2c09f5ed68 [IMP] core: introduce _field_to_sql() to replace _inherits_join_calc()
The new method should be used to generate an SQL object that represents
the value of a field in an SQL query.  Later this method will add
metadata to SQL objects, and that metadata will be used to determine
which fields to flush before executing some SQL code.

Part-of: odoo/odoo#138019
2023-10-20 09:35:18 +00:00
Xavier Morel 9d3de1190b [FIX] core: make model decorators work better with decorator 5
Decorator 5 changed the default decoration method from a transparent
exec-ing to wrapper functions. This makes the decorators visible to
the profiler, and breaks one of the profiler tests as the stack traces
now differ between using decorator 4 and decorator 5. Amongst other
concerns, this is an issue because debian bookworm has updated
decorator to 5 (.1.1), and the next ubuntu LTS (which should be 24.04
hopefully codenamed Nefarious Nematode) will do the same (Ubuntu has
been providing decorator 5 since 23.04).

5.1 added a `decoratorx` function which corresponds to the old
exec-based `decorator`, however it doesn't have a `decorate`
version. So we have to flag the wrappers, instead of decorate-ing the
original method with them. This seems to have the same semantics so
why we were using `decorate` is not entirely clear why we were not
doing that previously (neither
b1e83fd7b8 nor #25383 really provide
explanations). Possibly because pylint is a dum-dum and requires
ignoring one of its rules?

Implement in 14 since it doesn't hurt even though the test in question
does not exist yet.

closes odoo/odoo#137323

X-original-commit: 52e92904eca3cdeaee734b0a83e2d3fb463e8b56
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2023-10-03 06:16:49 +00:00
Stanislas Sobieski 436503e1f4 [IMP] core: readable representation of the cache
When the cache is filled with binary field, it can fill the logs with
unreadable and (probably) non compressible data

closes odoo/odoo#130741

X-original-commit: 6afba8fafe79dcf9bc28710a7edbd33b6a5b459f
Signed-off-by: Rémy Voet (ryv) <ryv@odoo.com>
2023-08-04 13:42:07 +02:00
Rémy Voet (ryv) cd7a10d06d [FIX] core,account_sequence: avoid add_to_compute no-store no-compute fields
Since https://github.com/odoo/odoo/pull/121355,
add_to_compute on no-store compute field, lead to recompute the field
at the end of request and it can lead to some non deterministic bug
(compute with a bad context by example).
Avoid to add_to_compute no-store or no-compute fields.

closes odoo/odoo#122973

X-original-commit: 533193106e6c9460d741e504babc7591cb935939
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Christophe Simonis (chs) <chs@odoo.com>
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
2023-05-31 09:23:18 +02:00
Xavier-Do 69d68296d8 [IMP] core: introduce OrderedWeakSet
The Weakset Used for envs can lead to unpredictible behaviour when
calling flush on the transaction because we iterate on the `envs`.

Since WeakSet.data is a set, the iteration will depends on the python
hash seed.

This commit proposes to replace Transaction.envs.data by an OrderedSet
to solve this issue.

Some test demonstrates that it does not affect the garbage collection
and that the order is now deterministic.

The question to now if we should reverse the envs to find the most
suitable envs remains.

closes odoo/odoo#121604

Signed-off-by: Raphael Collet <rco@odoo.com>
2023-05-19 13:33:11 +02:00
Chong Wang (cwg) b92415023d [IMP] core: prefetch all translations
a new context `prefetch_langs=True` allows ORM to prefetch all translations of
translated fields while fetching.

For example
The activated languages are 'fr_FR' and 'nl_NL'
In the database the value is '{"en_US": "English", "fr_FR": "French"}'::jsonb
after fetch with `prefetch_langs=True` the raw cache value will become
{'en_US': 'English', 'fr_FR': 'French', 'nl_NL': 'English'}

closes odoo/odoo#116947

Signed-off-by: Raphael Collet <rco@odoo.com>
2023-04-26 18:12:08 +02:00
Louis Wicket (wil) 9afe7c74c9 [IMP] *: remove "French spacing" 👺
According to Wiktionary, French spacing is "the archaic practice (though
still current in French) of inserting a space around colons, semicolons,
question marks, and exclamation marks". This is not standard practice in
English and most languages of the world.

The purpose of this commit is to start purging the code from this typo,
as it may reflect poorly on the software for some people.

closes odoo/odoo#114533

Related: odoo/enterprise#37853
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
2023-03-14 15:52:10 +01:00
Rémy Voet (ryv) 3353dfdb29 [REM] core,base,*: deprecated some methods to be remove correctly later
- Deprecated `norecompute` (on Environment class) because it is useless
and do nothing.
- Deprecated `cache_restart` (`res.company`) because `clear_caches` do
the stuff.
- Deprecated `write_company_and_print_report` (`res.company`) because
since https://github.com/odoo/odoo/pull/33863/, it is unused
- Deprecated `open_company_edit_report` (`res.company`) because since
25f7040998, it is unused.

Part-of: odoo/odoo#99550
2022-11-02 15:34:30 +01:00
william bebfa9b52c [FIX] base,core: order of self.env.companies
Using a `frozenset` leads to non deterministic issues.
Because when nothing is set in the context key `allowed_company_ids`,
`self.env.companies` will fallback on `self['res.company'].browse(user_company_ids)`
Since the browsing is done on a non ordered set, the recordset doesn't
follow the `_order` set on the model, by definition.

This has the effect of not being deterministic when iterating on
`self.env.companies`.

For instance
https://runbot.odoo.com/runbot/build/20539598

closes odoo/odoo#104538

X-original-commit: bfee55b14c918c44ebbb2563d5dc0e102edf396e
Signed-off-by: Rémy Voet <ryv@odoo.com>
Signed-off-by: William André (wan) <wan@odoo.com>
2022-10-28 20:39:50 +02:00
Xavier Morel 6c6ca16626 [REM] core: deprecated Environment methods
Part-of: odoo/odoo#98138
2022-10-26 19:03:46 +02:00
Victor Feyens 9ded78ede0 [IMP] core: docstring improvements
* clean and improve docstrings in orm
* fix typos found with codespell
* rely on the Environment class docstring instead of doc content (and
therefore move part of the doc inside the class docstring)

closes odoo/odoo#102969

X-original-commit: 8250cd4b210005d223a4cdb8afa4014425ca6fa3
Related: odoo/documentation#2803
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
2022-10-10 19:55:37 +02:00
Denis Ledoux 6ed63b9fee [IMP] base: possibility to archive companies
It is technically nearly impossible to delete a company
when it actually dealed with customers
(emitted invoices, received payments, ...).

Hence, if you want to get rid of a company,
for instance because you closed one of your subsidiaries,
giving the possibility to archive your company,
thanks to an active field, would be the best way to go.

We are hesitating to do a related field to the
`partner_id.active`, but we are a bit afraid
some people archive the partner linked to their company
for other valid reasons than get rid of their subsidiary
(such as avoid changing the address of their company
by mistake through the Contacts app),
while still wanting the company itself to be active.

So, we make it an independant column at the moment,
so we have the actual stored column in case we need it,
and will do a related stored to the partner
later on if we change our mind.

Manual forward-port of #102801

closes odoo/odoo#102586

Signed-off-by: Olivier Dony <odo@odoo.com>
2022-10-10 11:56:51 +02:00
ef00294e71 [IMP] core: store translated fields as JSONB columns
Translated fields no longer use the model ir.translation.  Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table.  The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages).  Empty text is allowed in translation values, but not NULL.

Here are examples for a field with translate=True:

    NULL
    {"en_US": "Foo"}
    {"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
    {"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}

Like before, writing False to the field makes it NULL, i.e., False in all
languages.  However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.

Here are examples for a field with translate=xml_translate:

    NULL
    {"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}

Change for callable(translate) fields: one can now write any value in any
language on such a field.  The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values.  Basically the
structure of the value must remain the same in all languages, like before.

Reading a translated field is now both simpler and faster than the former
implementation.  We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:

    SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...

The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages).  For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.

Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language.  Before this change, only the source value of the
field ('en_US') could be indexed.

Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages.  We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.

Code translations are no longer stored into the database.  They become static,
and are extracted from the PO files when needed.  The worker simply uses a cache
with extracted code translations for performance.  This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker.  Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).

Performance summary:
 (+) reading 'model' translated fields is faster
 (+) reading 'model_terms' translated fields is much faster (no need to inject
     translations into the source value)
 (+) searching translated fields with operator 'ilike' is much faster when the
     field is indexed with 'trigram'
 (+) updating translated fields requires less ORM flushing
 (-) importing translations from PO files is 2x slower

Some extra fixes:
 - make field 'name' of ir.actions.actions translated; because of the PG
   inheritance, this is necessary to make the column definition consistent in
   all models that inherit from ir.actions.actions.
 - add some backend API for the web/website client for editing translations
 - move methods get_field_string() to model ir.model.fields
 - move _load_module_terms to model ir.module.module
 - adapt tests in test_impex, test_new_api
 - because env.lang is injected into SQL queries, its returned value is
   now guaranteed to correspond to a valid active language or None
 - remove wizard to insert missing translations (no longer makes sense)

task-id: 2081307

Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2022-09-15 22:37:50 +02:00
Rémy Voet (ryv) dfa34aa3fe [IMP] core: reduce cost of modified.
For non-stored computed fields, `_modified_triggers` will traverse the
tree (at the cost of extra queries) only to know which record to
invalidate in cache. But in most cases, these fields have no data in
cache, so they can be ignored from the start, which allows us to prune
entire subtrees from the merged tree.

By example:
With simple write on `show_operations` of one `stock.picking.type`, the
`_modified_triggers` will fetch every ids (from database) of `stock.picking`
and `stock.move` related to this `stock.picking.type`
(For `stock.move`, it is because of the
`show_operations = fields.Boolean(related='picking_id.picking_type_id.show_operations'`))
In that case, there isn't any data of `show_operations` (`stock.move`)
in cache, then there are nothing to invalidate and
the `_modified_triggers` cost
is high (extra queries/processing) for nothing.

Then we cut parts of the tree when we know that
they won't invalidate anything (=> if the cache is empty for the field).
Also refactor the way to merge trees to be more efficient.

Performance improvements:
- For all tests at-install done by the runbot, we gain -+ 3.5% of
queries.
- In the example above, we reduce the number of queries (potentially
bottleneck queries in large DB) from 7 to 4.
- In term of CPU (without counting time in SQL), the new version is -+
20 % faster (on install of stock,purchase,mrp and with
--test-tags=/stock,/purchase).

closes odoo/odoo#76322
task-2780812

closes odoo/odoo#99274

Related: odoo/enterprise#30951
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-09-02 23:16:06 +02:00
Raphael ColletandVincent Schippefilt 384fda2c2a [REF] core: replace towrite by dirty flag in cache
Merging both the memory of field values and suspended updates has
several advantages:
 - avoid inconsistencies between cache and towrite
 - cache updates can be made safer w.r.t. dirty flag

However, the dirty flag in cache does not go well with context-dependent
fields.  When a context-dependent field is dirty in cache, the value to
store in the database is accessible through some context values.  But
when the model is flushed, the context values on the current environment
may be different.  When this happens, the method flush() fails to
retrieve the data to flush.

The proposed solution is to store the "dirty" value in cache under
conventional context values, and to retrieve them under the same
conventional context values to flush them.  For instance, when storing
the value of a binary field, it will be stored once under the context
value `context.get('bin_size')`, and a second time under the context
value `None`.  The flush implementation will then retrieve the value
using the context value `None`.

Translated fields are also problematic when a value is put in cache with
an environment where lang=False, and the value is retrieved with another
environment where lang=None.  This issue is addressed by normalizing the
context key 'lang' to None when the context value is False.

Part-of: odoo/odoo#95325
Co-authored-by: Vincent Schippefilt <vsc@odoo.com>
2022-07-14 22:45:44 +02:00
Raphael Collet 9c3b9a4926 [FIX] core: automatically flush upon invalidation for cache consistency
Because method _read() no longer updates existing values in memory,
those values in memory must be consistent with the database.

On the other hand, if pending updates are not performed on the database
before fetching values, it means that the corresponding database values
cannot be put in cache.  This implies that one cannot empty the cache
without flushing the corresponding fields.

In order to avoid mistakes, flush automatically before invalidating the
cache.  This makes the invalidation methods safe by default, and avoids
cargo-culting which would systematically associate invalidation to
flushing, which may eventually be less performant.

Part-of: odoo/odoo#66938
2022-07-05 11:35:00 +02:00
Raphael Collet a91cb08c5d [IMP] core: avoid flushing fields to read
The idea is to avoid flushing the fields to fetch in method _read().
This delays UPDATE queries, and makes the prefetching mechanism simpler
and more effective.

We do this by not overwriting the cache values by the values fetched
from database.  This simple idea allows to fetch more fields and more
records without having to care about pending computations and updates.
But it requires the cache consistency to be much more strict, because
nothing will "fix" the cache inconsistencies "by chance".  And it also
requires pending updates to be present in cache.

Part-of: odoo/odoo#66938
2022-07-05 11:35:00 +02:00
Raphael Collet 9f6be9321c [IMP] core: reimplement cache check in pure SQL
This allows checking that the ORM keeps the cache consistent without
relying on it for the check.

Part-of: odoo/odoo#66938
2022-07-05 11:34:59 +02:00
Gorash 8c869769e9 [IMP] base: add a cache for company_ids on res.user
Part-of: odoo/odoo#88276
2022-06-03 16:40:39 +02:00
Raphael Collet 32bc28aa66 [IMP] core: better API for flush() and invalidate()
This provides a new API for those operations, in order to make the
distinction between the use cases more explicit.  The former API was
using obscure parameter combinations to correspond to various cases.

In the summary below, `fnames` is an iterable of field names.  If the
parameter is not given, it means "all fields" in the given context.
Note that method recompute() is now mostly private, as it should not be
used in business code.

    # process pending computations and updates
    records.env.flush_all()                # all fields of all models
    records.flush_model(fnames)            # the fields of all records of the model
    records.flush_recordset(fnames)        # the fields of the given records

    # process pending computations, became non-public methods
    records.env._recompute_all()           # all fields of all models
    records._recompute_model(fnames)       # the fields of all records of the model
    records._recompute_recordset(fnames)   # the fields of the given records

    # invalidate the cache of fields
    records.env.invalidate_all()           # all fields of all models
    records.invalidate_model(fnames)       # the fields of all records of the model
    records.invalidate_recordset(fnames)   # the fields of the given records

Part-of: odoo/odoo#87527
2022-05-25 18:00:46 +02:00
Xavier-Do cb9b93b5c4 [FIX] core: ordered to_compute
Set order is not randomized for integers (integer hash is the integers
itself and doesnt use pythonhashseed)
but the output may be ordered or not depending of the values:

In [2]: list(set(range(126,130)))
Out[2]: [128, 129, 126, 127]

In [3]: list(set(range(130,134)))
Out[3]: [130, 131, 132, 133]

This will lead to different results in query count tests depending on
the current value of the sequence.

The proposed solution is to use an Orderedset. We need to keep in
mind that the initialisation/operation on Orderedset may be slower but
this shouldn't represent any significant overhead in this case.

closes odoo/odoo#91962

X-original-commit: da66b8fc008f97680584e266135b7fa7d3dedeca
Signed-off-by: Rémy Voet <ryv@odoo.com>
2022-05-23 08:30:14 +02:00
Rémy Voet (ryv) 8b7c8d0b71 [REF] core: replace _browse() by __init__()
The creation of recordset was done by class method _browse() instead of
a regular call to the model's class.  As we removed the old usage of
method __init__(), we can now reuse it with a normal usage. After this
commit, we can create a recordset by simply calling its class:

    registry['model_name'](env, ids, prefetch_ids)

closes odoo/odoo#79563

Signed-off-by: Raphael Collet <rco@odoo.com>
2021-12-24 14:56:58 +00:00
Rémy Voet (ryv) cbb563fd98 [REM] core: remove deadcode from models.py
- Remove unused imports `AsIs` and `Collector`
- Remove unused logger _schema
- Remove unused function same_name
- Remove unused attribute `_needaction`
- Remove backward compatibility of `__new__` and `__init__`
- Remove backward compatibility of `__export_rows`

Also:
- remove deprecated warning in api.py
- remove the `__init__` from `ir.ui.menu`, it was useless
because the cleaning of cache (`clear_caches`) is global (a call of
`clear_caches` clean all cache method with a ormcache decoration)

Part-of: odoo/odoo#79563
2021-12-24 14:56:57 +00:00
Victor Feyens 57ced812f0 [REF] product: clean pricelist API
* Do not rely on context, everything should be cleary specified through
parameters
  Catch context keys in an unique targeted place to improve code clarity
* Drop strange old API
  * do not provide unused partner parameter anymore
  * do not provide products, qty as a list of tuple, we only request the
same qty for all products anyway
* Clear methods, add/adapt comments and docstrings
* Reduce potential side-effects of context content.
2021-12-23 15:30:23 +01:00
Yannick TivisseandVictor Feyens 18952cdc76 [IMP] *: Convert single create method into multi
Taskid: 2703085
Part-of: odoo/odoo#80824
Co-authored-by: Victor Feyens <vfe@odoo.com>
2021-12-14 19:13:18 +00:00
Xavier Morel bdc9d9d369 [FIX] core; base: lots of docstrings
* add configuration for `flake8[flake8-rst-docstring]`
* enable docstring-related checks
* fix invalid docstrings in odoo's core & `base`
* fix a few more bits (mostly missing or incorrect `:param:` info
  fields) are out of scope for the lint but my editor catches

closes odoo/odoo#74604

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2021-12-09 14:36:58 +00:00
Raphael Collet 0011823932 [FIX] core: cr.transaction no longer depends on registry
The cursor features a "transaction" object to manage application-
specific data in relation with cursor operations.  In its initial
implementation, the transaction object was created by the registry.
This created a requirement: in order to be used in environments, a
cursor had to be created by registry.cursor().

We now remove that unnecessary technical requirement by making
environments create the transaction object on demand.

closes odoo/odoo#80644

X-original-commit: e902713648bca329e6821859462b2128aad62c09
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-12-01 08:44:48 +00:00
Rémy Voet (ryv) c3399b5470 [FIX] base: fix performance scalability of create
Issue:
------
The `create` method (models.py) doesn't scale correctly
(with huge number of values) when compute store no-readonly
field(s) are in values (at most one).
It is due to the method `protecting` (api.py) which has a
bad complexity in the `create` situation
(list of (field, records) pairs as argument):

O(r²) with `r` = number of record containing the protected field:
List of `len(r)` send to `protecting`,
loop of this list (`r` factor), loop on fields (constant factor),
create a new frozen set with the previous one (`r` factor).

Fix:
----
Decrease the complexity to O(r) by creating a map of set of ids by
protected field which allows avoiding recreating a new frozenset
for each record (update with tuple of one inside => O(1)).

Performance gain:
----------
For a very simple model with only one compute store no-readonly field,
and all `create` `values` contains this field.
+--------------+---------------+---------------+---------------+----------------+
|   Batch ->   |     1000      |     10000     |    30000      |     80000      |
+--------------+---------------+---------------+---------------+----------------+
| Before (sec) | 0.083 ± 0.006 | 1.433 ± 0.091 | 8.764 ± 0.609 | 94.428 ± 3.021 |
| After (sec)  | 0.069 ± 0.003 | 0.706 ± 0.006 | 2.188 ± 0.076 | 5.875 ± 0.104  |
+--------------+---------------+---------------+---------------+----------------+

closes odoo/odoo#80437

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-11-29 11:22:24 +00:00
Stefan RijnhartandRaphael Collet 8f574a274a [14.0][FIX] Keep original kwargs intact for reuse on retry
closes odoo/odoo#79589

X-original-commit: 6387619d308c8969cbafe6acf77f6d22075ac6ba
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2021-11-10 10:43:17 +00:00
Raphael ColletandXavier Dollé 1595c0ee27 [REF] core: replace thread-local "envs" by cursor-bound "transaction"
Refactor the Environments object into a Transaction object, which is
bound to one cursor, and is no longer shared among several cursors.

The following methods/properties have been changed:
 - Environment.envs no longer works (because of the design change);
 - Environment.manage() is deprecated (no longer useful);
 - Environment.reset() is now an instance method;
 - env.clear_upon_failure() is deprecated in favor of cr.savepoint().

closes odoo/odoo#75598

Related: odoo/enterprise#20451
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
2021-09-03 15:45:46 +00:00
Martin Trigaux 9aedb999f5 [IMP] *: remove xmlid_to_object helper
Use env.ref instead of the xmlid_to_object
To make it more readable and be able to clean old helpers
2021-08-10 14:24:11 +02:00
Martin Trigaux c7bac3dee0 [IMP] *: make ir.model.data helper private
No reason to interfact with them directly in RPC
2021-08-10 13:49:04 +02:00
Raphael Collet 3deafa8ebc [FIX] core: enable callable in api.constrains()
This avoids a call to __init__() which no longer has access to the
model's class' parent classes, and was not even correct...

closes odoo/odoo#70400

Related: odoo/enterprise#18141
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2021-05-06 07:30:29 +00:00
Raphael Collet 34d6f87d54 [REF] core: put field.depends on registry and make field.recursive explicit
The attributes field.depends and field.depends_context are problematic
for sharing fields across registries, because they depend on the model's
registry class, which may vary from one registry to another.  In order
to make computed fields shareable, we have to move those values away
from fields.

For the same reason, field.recursive should not be inferred, because its
value may depend on the registry, although it is generally not the case.
Moreover, the flag recursive=True is set on a field when field triggers
are determined (on the registry).  A compute method may be called before
the flag is set (if no update has been done yet), and that can lead to
incorrect computations.

This happened in test TestUsers2.test_reified_groups in module 'base'.
The user groups view was apparently determined without the flag being
set, and the view depends on the recursive field 'trans_implied_ids',
which was not correctly computed.

We thus force developers to be explicit about recursive computed fields.
The code now logs a warning when the flag is not set up properly.
2021-05-03 12:33:29 +00:00
Adrian Torres 1c8a958098 [IMP] core: introduce api.ondelete decorator
With this commit, a new ORM api decorator is introduced:
`api.ondelete(*, at_uninstall)`.

This decorator is to be applied to Model methods that check for specific
business conditions when attempting to unlink a record via the
interface.

E.g. trying to unlink a validated journal entry

This decorator allows this logic to exist outside the `BaseModel.unlink`
method and is automatically bypassed when in uninstall mode, this means
that during an uninstall any and all data related to a module can and
will be removed easily and cleanly while still being able to apply
business logic to manual deletion of records.

This feature opens the gates to solving a very big problem with
uninstalls: records and tables that remain in a database despite the
relevant module being uninstalled, because if an override to unlink
raises an error, the data will never be deleted from the database.

Henceforth, overrides of unlink shall solely be used for data-cleaning
purposes, i.e. deletion or modification of data that is related to the
one currently being deleted but cannot be automatically deleted because
there are no proper SQL relations.

In certain very specific, low-level scenarios an unlink may be
overridden to raise an error, but this should only be done if you know
what the fuck you're doing, most of the time you'll want to resort to
`@api.ondelete`.

Note that this new decorator includes a keyword-only, required argument
called `at_uninstall`, in most business cases this argument shall be
False as this argument dictates whether or not this method should be
executed in the `unlink` call during uninstall. It should only be set to
True if you are certain of all the implications which most likely means
that the records of the model in which this ondelete function is defined
will NOT be removed during uninstall, they will forever linger in the DB
until manual intervention, this in turn can mean a wide range of
undefined problems due to crap left on the database.

Following commits will replace any `unlink` overrides that raise
business errors by methods decorated with `api.ondelete`, another commit
will introduce a pylint checker that will raise a warning any time that
an unlink override raises an error.
2020-12-04 09:15:51 +00:00
Victor Feyens f7782f6c76 [FIX] core: consistent environment for env.company(ies)
Before this commit, the company recordset returned by
env.company/env.companies
was sometimes in sudo and sometimes not.

If 'allowed_company_ids' was specified in the context, the returned
recordset wasn't sudoed.
If no company was specified through the context, company/companies was
using env.user.company_id(s)
as fallback, which is always sudoed since env.user is always sudoed.

With this commit, the behavior is now consistent, we enforce that the
returned recordset is only sudoed if the current environment is.
As everybody has read access on res.company records, there is no need
to return the company as a sudoed record.
When a specific operation has to be done on the company, then a sudo
should be called if needed.

closes odoo/odoo#62074

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-12-03 14:25:12 +00:00
Raphael Collet a40511cd29 [FIX] core: env.clear() also invalidates lazy properties
In unit tests, this avoids side effects from one test to another.  In
particular, a test modifying the user's company can make other tests
fail because `env.company` defaults to the user's company when nothing
else is available in the context.
2020-11-24 13:23:32 +00:00
Raphael Collet 9b7a4e1815 [IMP] core: smaller and faster field cache
Consider a context-dependent field and two possible cache structures for
storing the data of that field for N records and K context keys.  The
common use-case is to assume that N is much larger than K.  We look at
how many dicts are used in each cache structure, where we label dicts
with N entries as "large", and dicts with K entries as "small".

     Structure                 | Number of dicts
    ---------------------------+-------------------
     cache[field][id][key]     | 1 large + N small
     cache[field][key][id]     | 1 small + K large

It is known that "small" dicts have a large relative overhead when
compared to "large" dicts.  Given this fact, the second cache structure
will generate less memory overhead.  Here are numbers for N=16, showing
the size in bytes taken by the cache structure.  Larger values of N lead
to larger size differences, always in favor of the second structure.

     Structure             |  K=1 |  K=2 |  K=3 |  K=4 |  K=5 |  K=6
    -----------------------+------+------+------+------+------+------
     cache[field][id][key] | 4488 | 4488 | 4488 | 4488 | 4488 | 6536
     cache[field][key][id] |  888 | 1536 | 2184 | 2832 | 3480 | 4256

Moreover, some memory operations are performed on many records and a
single context key.  Those operations are clearly simpler and faster
with the second cache structure.
2020-07-24 08:30:00 +00:00