When looking to fetch grouped data from the database, `read_group`
should work(-ish), especially with support for `array_agg`. This also
means this need should be more or less solved around RPC: client can
either `read_group` or group on the client side.
This leaves a glaring hole in the fabric: there's no convenient tools
to apply grouping to recordsets. `itertools.groupby` exists, but it
requires that the input be grouped by the key function, and it yields
an iterable of items which is not the most convenient for recordsets.
This here utility:
- is exclusive to recordsets
- returns a mapping of grouping keys to subsets of the input recordset
- keeps the prefetching of the source recordset
- allows grouping on a field, or an arbitrary (callable) key
- should work even on "new records" (aka should be suitable for
onchanges / arbitrary compute functions)
While the method is RPC-compatible, it's not especially designed for
that, it is likely a much better idea for the client to `read` the
data they need then group that client side on whatever criteria they
are interested in, or `read_group` the aggregated information they
need directly if that's an option.
Part-of: odoo/odoo#101522
* clean and improve docstrings in orm
* fix typos found with codespell
* rely on the Environment class docstring instead of doc content (and
therefore move part of the doc inside the class docstring)
closesodoo/odoo#102969
X-original-commit: 8250cd4b210005d223a4cdb8afa4014425ca6fa3
Related: odoo/documentation#2803
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Before this commit the neutralize system introduced in v16 was using ORM
methods in order to change appropriate records. Although flexible, this approach
could lead to call some methods with side effects while neutralizing
(eg: overloads of write).
This patch converts the neutralize system to a safer "inert" SQL based approach
by migrating the generic method _neutralize to SQL files exposed in the
data folder.
Task id: 2961687closesodoo/odoo#102792
X-original-commit: e5dbded9bb363351feff7ca8a56c7f8a6860f492
Related: odoo/enterprise#32580
Signed-off-by: Fabien Meghazi <fme@odoo.com>
Purpose
=======
Check the existence of the relational properties (many2one / many2many)
in batch and prefetch the values in batch as well to reduce the number
of SQL queries.
Technical
=========
The existence is checked in the read method of the properties field,
because we have the entire recordset. Then, the non-existing ids are
remove from the properties values and the cache is updated.
Task-2965523
X-original-commit: dba9b684d29c0041a32a5508284573851b8dd097
Part-of: odoo/odoo#102243
By definition, views are useless without the web client.
The web client is in the module `web`.
Therefore, in a perfect world, the `get_views` model method
and any related method should be within the `web` module,
as without it they are useless.
Let's imagine you would like to use fully in command-line,
without web client, all those view related methods are useless.
Maybe excepted for reports, as you might still like to be
able to print reports while using odoo fully in command line.
Views related method therefore shouldn't be in `odoo/models.py`.
Also, when you think about it, `get_views` related methods do not
make sense without the model `ir.ui.view`, which is loaded after
the `get_views` related methods, which also doesn't make sense.
However, moving these methods fully in the `web` module is an harder
work. For instance, there are base models, such as res.partner,
already overriding `get_view` in the `base` module,
and therefore relying on these view related methods.
As a first step, we move view related methods direcly in the
`odoo/addons/base/models/ir_ui_view.py` file, where
the `ir.ui.view` model is loaded.
This is not only a design / cleaning change,
but a required change to be able to use content of
`odoo.tools.config`, which is loaded after `odoo/models.py`.
For instance, if you want to configure a conditional decorator based on
the config `odoo.tools.config['dev_mode']`, it is not possible
to do so in `odoo/models.py` because the config is parsed/loaded
after `odoo/models.py`.
The config is loaded here:
https://github.com/odoo/odoo/blob/31de2b0a7a0921cab3c6c54045d15da46c8e6d8a/odoo/cli/server.py#L127
While, within the same file, `odoo/models.py` gets loaded through the
`import odoo`
https://github.com/odoo/odoo/blob/31de2b0a7a0921cab3c6c54045d15da46c8e6d8a/odoo/cli/server.py#L26
And we would like to put such a decorator based on `odoo.tools.config['dev_mode']`
on `_get_view_cache`, to not cache the back-end views when `--dev xml` is
passed in the server arguments.
X-original-commit: 0901adc38a724aec75676285977f1905a84ed8ee
Part-of: odoo/odoo#102117
Bug
===
- create a new helpdesk ticket without properties
- create a new property
- change the customer
=> The added property is removed
The fix in the onchange of models.py needs to be more specific, and
should be used only when the definition record is changed.
Task-2965523
X-original-commit: 909ee004183068d9ffa94abe0d1472a8f0534f34
Part-of: odoo/odoo#101487
The cache currectly fails to correctly invalidate relational fields that depend
on a non-relational field. Two passes of invalidation are done, to reflect
dependencies on both the old and the new written values. In the first pass
only relational fields are considered, as explained in the comments:
> It is best explained with a simple example: consider two sales orders SO1 and
SO2. The computed total amount on sales orders indirectly depends on the
many2one field 'order_id' linking lines to their sales order. Now consider the
following code:
>
> line = so1.line_ids[0] # pick a line from SO1
> line.order_id = so2 # move the line to SO2
>
> In this situation, the total amount must be recomputed on *both* sales order:
the line's order before the modification, and the line's order after the
modification.
The written values can be seen as the roots of a dependency forest (a
collection of dependency trees). Before this commit all non-relational roots
and their corresponding trees were filtered out during the first pass. However,
this approach is wrong, as relational fields can also depend on non-relational
fields. Instead, the complete dependency forest has to be traversed, skipping
invalidation for non-relational fields during the first pass.
The test that was previously included accidentally succeeded because of a
separate and unrelated bug in the orm domain parser: in certain one2many or
many2many leafs the domain parser would not take into consideration the domain
included in the definition of the field. As a result, the test still passed
by accident, because the records that no longer matched the domain after the
write were still invalidated during the second pass.
The problem can clearly be demonstrated, however, when the dependency is
generated by a compute function.
closesodoo/odoo#101038
X-original-commit: d4a5827b42d80f0f830455dcd2056701eb09aed1
Signed-off-by: Rémy Voet <ryv@odoo.com>
Signed-off-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
- In `unlink`, since https://github.com/odoo/odoo/pull/66938
modified is called on self for each batch of 1_000.
But it should be called on the batched records.
- In `write`, remove useless `records_to_inverse`
(there from ORM refactor but never used)
- make `_modified_triggers` more deterministic by
changing a `set` into `OrderedSet`.
closesodoo/odoo#100472
Signed-off-by: Raphael Collet <rco@odoo.com>
A feature allows to set the modifiers invisible, readonly, required
according to a key in the context
e.g.
```xml
<field name="date_approve" invisible="context.get('quotation_only', False)" optional="show"/>
```
Following odoo/odoo#99417,
back-end views are now cached.
These expressions are currently evaluated server-side,
before serving the view to the web client.
Hence, the modifier becomes for instance `invisible="1"` or `invisible="0"`
according to the context passed when calling `get_view`.
The key used to cache the views doesn't take into account such context keys.
Hence, when you asked for a view using for instance
`invisible="context.get('quotation_only')"`
A first time using the context `{'quotation_only': True}`
and a second time using the context `{'quotation_only': False}`,
on the second time, you received the view from the first time you
requested the view, where the modifier is evaluated as if the the
context was `{'quotation_only': True}`.
As we do not want to store a cached version of the view for each
possible key in the context, postprocess the evaluation of the
modifiers using the context after retrieving the view from the cache.
A better alternative would be to delegate this evaluation to the
web client, because it already has the information it needs,
it has the context value.
Modifiers using domains are already evaluated client-side,
it would make sense modifiers using context would too.
Nevertheless, as this bug has been introduced by odoo/odoo#99417,
and as we are close to the release, solve this server-side,
to keep a similar behavior than before.
We might re-consider the implementation of this later on.
closesodoo/odoo#100130
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Oversight in odoo/odoo#100376
The web client sends `list` rather than `tree` when requesting
views.
However, once they went through `get_views`, they are converted back
to `tree`.
https://github.com/odoo/odoo/blob/master/odoo/models.py#L1653
So, here, we should check against `tree`, not `list`.
Could be seen in Lunch > Manager > Today's Order.
With the `id` field missing, an error is raised
```
Uncaught (in promise) Error: The following error occurred in onWillStart: "field is undefined"
```
closesodoo/odoo#100474
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Following odoo/odoo@4636620004
`get_views` only pass the model fields included in the view architecture,
except for the main model when the search views is requested.
Because the search views requires all fields for the user to be able
to make advanced filters and advanced group by using any fields of the
model.
However, other views requires more field descriptions as well than
just the fields included in their architecture:
- the graph view requires all integer and float fields,
to automatically add suggestions of measures in the measures dropdown
menu. It's a bit like the search view, the user should be able
to choose any measure available in the model
(as long as this is integer or float fields)
- the pivot view requires all groupable fields,
so the user can group by any groupable fields of the model.
The JS MockServer `getViews` is adapted to include the changes added by the above
revision as well as the current revision,
for the qunit tests suite to be able to reflect these API changes
from the server side.
closesodoo/odoo#100376
Related: odoo/enterprise#31427
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Translated fields no longer use the model ir.translation. Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table. The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages). Empty text is allowed in translation values, but not NULL.
Here are examples for a field with translate=True:
NULL
{"en_US": "Foo"}
{"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
{"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}
Like before, writing False to the field makes it NULL, i.e., False in all
languages. However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.
Here are examples for a field with translate=xml_translate:
NULL
{"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}
Change for callable(translate) fields: one can now write any value in any
language on such a field. The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values. Basically the
structure of the value must remain the same in all languages, like before.
Reading a translated field is now both simpler and faster than the former
implementation. We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:
SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...
The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages). For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.
Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language. Before this change, only the source value of the
field ('en_US') could be indexed.
Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages. We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.
Code translations are no longer stored into the database. They become static,
and are extracted from the PO files when needed. The worker simply uses a cache
with extracted code translations for performance. This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker. Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).
Performance summary:
(+) reading 'model' translated fields is faster
(+) reading 'model_terms' translated fields is much faster (no need to inject
translations into the source value)
(+) searching translated fields with operator 'ilike' is much faster when the
field is indexed with 'trigram'
(+) updating translated fields requires less ORM flushing
(-) importing translations from PO files is 2x slower
Some extra fixes:
- make field 'name' of ir.actions.actions translated; because of the PG
inheritance, this is necessary to make the column definition consistent in
all models that inherit from ir.actions.actions.
- add some backend API for the web/website client for editing translations
- move methods get_field_string() to model ir.model.fields
- move _load_module_terms to model ir.module.module
- adapt tests in test_impex, test_new_api
- because env.lang is injected into SQL queries, its returned value is
now guaranteed to correspond to a valid active language or None
- remove wizard to insert missing translations (no longer makes sense)
task-id: 2081307
Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
The main goal is to reduce the number of KB sent by the `get_views`
method to the web client by getting rid of things not used by the web
client.
This revision aims to only pass the `fields_get` info of fields
actually required by the web client.
Before this revision, for each view, the list of all fields
of all models implied in the view are passed.
For the main view, it's required to pass all the fields,
for the search view: The advanced search needs to know all fields
of the model to be able to let the user to do advanced filters and
groupbys on the model fields.
But, for the models of the subviews (one2many/many2many embbeded views),
this is not required. Only the fields included in the view are required.
Also, if the search view is not requested by the web client,
then it's also not required to pass all the fields of the main model.
After this revision, for each view, pass the `fields_get`
info of only the fields actually implied in the view.
For the search view, info of all fields are passed,
not only the ones implied in the view.
For instance, on the `get_views` of `account.move`,
this allows a gain of an extra 18,05KB,
reducing from 165.82 to 147.77KB.
In addition, before this revision, the whole content of `fields_get`
was stored in the cache of `_get_view_cache`, meaning per view.
If 3 views were requested on `get_views`
let's say kanban, list, form,
the same whole `fields_get` info, of all fields,
was stored 3 times in the cache.
This revision removes the content of fields_get of the cache
for `get_views`, to avoid storing multiple times the exact same
information in the cache, therefore gaining tremendous memory,
while not loosing that processing time.
The decrease of speed is only 1~3ms per call to `get_views`.
This is mainly because all the information gathered by `fields_get`
is already in the cache. For instance, it doesn't require any SQL
request (at the moment). e.g. the translations (string, help, selection)
are already cached.
closesodoo/odoo#99834
Related: odoo/enterprise#31167
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Instead of fetching all field attributes sent with the field list
to the web client,
restrict the attributes to the ones actually required by the web client
This allows, for instance,
to gain 44,75KB on each call on `get_views` for `account.move`,
from 208.78KB to 164.03KB,
with only `account_accountant` installed.
closesodoo/odoo#99660
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
The master plan finally comes to an end.
Thanks to:
- odoo/odoo#87522 refactoring `load_views`,
- odoo/odoo#94337 refactoring `common.Form` to prevent changing
invisible fields in unit tests using `Form` instances,
- odoo/odoo#95729 refactoring the behavior of `groups=` in views,
- odoo/odoo#98551 removing the need of the `groups_id` many2many field
on back-end views.
The result returned by `get_view`/`get_views` can now finally be easily
and efficiently cached, in order to cache back-end views.
The goal of this revision is to cache the model views and fields
already post-processed for the web client
(with the modifiers, etc., already computed)
without group restriction.
Then, from this cached version, post-process group related features,
such as removing nodes restricted with a `groups=` attribute,
set the create/write button according to the user access rights to models, ...
Not including the groups in the cache key allows:
- to have less cached versions,
(otherwise it would be one cached version per different group combination)
- to not have to fetch the user groups to compute the key
(with the current cache key,
there is nothing to fetch from the database to compute the key)
Besides, post-processing the groups features
after taking the view from the cache of the view doesn't take a tremendous time:
- parsing arch from/to string with etree is fast,
- removing the `groups=` nodes using etree is fast,
- adding the `create="False"`, `write="False"`, `delete="False"` on the view
root node according to the access right of the user on the model is fast.
This allows way faster calls to `get_views` by the web client,
as the server no longer need, for each call, to fetch the views in database,
combine the inherited views, post-process the modifiers attributes, etc.
Timing tests are available on the pull request of this revision.
Part-of: odoo/odoo#99417
The cache key of _get_bindings was not super efficient.
The result of _get_bindings is cached,
but its performance was altered by the cache
key which requires to fetch the user groups for each call to
_get_bindings.
Besides, as there is a lot of possible group
combination, this resulted in a lot of possible cache keys,
and therefore a lot of cached values.
This revision aims to make _get_bindings more efficient
by:
- do not use the groups in the cache keys (less cached values)
- filter out actions not available to the user groups after
retrieving them from the cache
- use has_group to do the above, which is itself cached as well,
and therefore do not need to fetch the user groups
at each call to get_bindings.
In addition, move get_bindings from `get_view`
to `get_views`. If there was 3 views asked by `get_views`
(let's say kanban, list, form)
`get_bindings` was being called 3 times, through `get_view`
with each time the same arguments and therefore the same result :-).
Moving it to `get_views` allows to call it only once for all view types
requested, and for the web client it doesn't change much,
as it always request the toolbar/get_bindings through `get_views` only.
In addition, add the lang to the cache of _get_bindings.
it was actually a bug not to put it: if you had 2 users
with the same group set, using 2 different languages,
the user accessing first the get_bindings would cache
the action names within his language, and then the second
user would see the action name within the language of the first user
:-).
Before
```py
In [1]: %time for i in range(1000): self.env['ir.actions.actions'].get_bindings('res.partner'); self.env.invalidate_all();
CPU times: user 790 ms, sys: 104 ms, total: 893 ms
Wall time: 1.7 s
```
After
```py
In [1]: %time for i in range(1000): self.env['ir.actions.actions'].get_bindings('res.partner'); self.env.invalidate_all();
CPU times: user 23.5 ms, sys: 9.12 ms, total: 32.7 ms
Wall time: 36.9 ms
```
Part-of: odoo/odoo#99417
For non-stored computed fields, `_modified_triggers` will traverse the
tree (at the cost of extra queries) only to know which record to
invalidate in cache. But in most cases, these fields have no data in
cache, so they can be ignored from the start, which allows us to prune
entire subtrees from the merged tree.
By example:
With simple write on `show_operations` of one `stock.picking.type`, the
`_modified_triggers` will fetch every ids (from database) of `stock.picking`
and `stock.move` related to this `stock.picking.type`
(For `stock.move`, it is because of the
`show_operations = fields.Boolean(related='picking_id.picking_type_id.show_operations'`))
In that case, there isn't any data of `show_operations` (`stock.move`)
in cache, then there are nothing to invalidate and
the `_modified_triggers` cost
is high (extra queries/processing) for nothing.
Then we cut parts of the tree when we know that
they won't invalidate anything (=> if the cache is empty for the field).
Also refactor the way to merge trees to be more efficient.
Performance improvements:
- For all tests at-install done by the runbot, we gain -+ 3.5% of
queries.
- In the example above, we reduce the number of queries (potentially
bottleneck queries in large DB) from 7 to 4.
- In term of CPU (without counting time in SQL), the new version is -+
20 % faster (on install of stock,purchase,mrp and with
--test-tags=/stock,/purchase).
closesodoo/odoo#76322
task-2780812
closesodoo/odoo#99274
Related: odoo/enterprise#30951
Signed-off-by: Raphael Collet <rco@odoo.com>
`_check_concurrency` was used to check if 2 people were modifying the
same record at the same time. It was doing so by setting a special
`__last_update` value inside the context that was later evaluted to
prevent some concurrency issues.
It was mostly unused, wasted a lot of cpu cycles and was not covering
all cases (e.g. pending write).
closesodoo/odoo#87756
Signed-off-by: Vincent Schippefilt (vsc) <vsc@odoo.com>
Purpose
=======
Make the onchange work for the properties fields. When changing the
container field, we need to update the properties definition.
Task-2852259
Part-of: odoo/odoo#95184
Purpose
=======
Add a new field "Properties" to be able to light customization of workflows
based on a parent model. Those properties acts in some ways like Odoo fields
without requiring specific columns e.g. add new properties on tasks of a
specific project.
Usage
=====
Define properties on a parent model (e.g. project) with
```
attributes_definition = fields.PropertiesDefinition('Message Properties')
```
It defines properties available on children: types, default, value, model for
relational properties,...
Use it on children records (e.g. task) with
```
attributes = fields.Properties(
string='Properties',
definition='parent_id.attributes_definition',
)
```
Technical
=========
Parent | Properties definition
------------------------------
The properties definition is stored on the parent, on a JSON field.
This definition contains the type of the properties, the default value,
the model of the many2one,...
```
[
{
'name': 'name',
'string': 'Name',
'type': 'char',
'default': 'Default Name',
}, {
'name': 'partner_id',
'string': 'Partner',
'type': 'many2one',
'comodel': 'res.partner',
},
]
```
Child | Properties values
-------------------------
The value is stored on the child, using a Properties field.
```
{
'name': 'Mitchel',
'partner_id': 1337,
}
```
When we read this field, we will automatically read the definition on
the parent, and merge both JSON into one, so the web client has the
value of each property, and their definition.
```
[
{
'name': 'name',
'string': 'Name',
'type': 'char',
'default': 'Default Name',
'value': 'Mitchel',
}, {
'name': 'partner_id',
'string': 'Partner',
'type': 'many2one',
'comodel': 'res.partner',
'value': 1337,
},
]
```
Integrity
---------
If we remove a property on the parent, we won't update the child value.
Instead, when we read the child properties, we will filter them based
on the parent. So the removed properties will be removed the next time
we write on the field.
In the same logic, the many2one existence is checked when we read the
field. There's no foreign key between the integer stored in the JSON
in the SQL row corresponding to the record in database.
Write
-----
We can write on the Properties field with a list of field definition
+ value.
Some types are not JSONifiable (like the date, datetime), they are
stored as string in database and parsed when we read the value.
In order to update the parent definition by writing on the child,
you need to add the dict key `definition_changed` or
`definition_deleted`. This is because we need to be able to know
if the definition has been changed without doing extra SQL queries.
Access rights
-------------
A user can add a many2one / many2many property to a model only if he
has the access rights to it.
Many2one / Many2many
--------------------
The model choice of a many2one / many2many properties was subject to
changes.
First implementation stored models in both parent and children to easily
spot changes and avoid complex queries when fetching records, trying to
synchronize them, ...
As this leads to storing a lot of duplicated content we choose to instead
reset the value on the child if the model has been change. We generate a
new name for the property. So it behaves like if we removed the property
and created a new one.
To be able to restore the old value (e.g. if by mistake we changed the
model, and go back to the old model), we store the initial states.
Task-2852259
Part-of: odoo/odoo#95184
The sequence field is special-cased early on in read_group (_raw): if
the caller requests the aggregation of ``sequence`` that request is
ignored:
if fspec == 'sequence':
continue
This was added a long time ago, probably due to the special-ish status
of `sequence` (summing sequence number doesn't really make much
sense).
The issue is that it's also possible to request ordering by
`sequence`, which requires `sequence` to be one of the aggregated
fields. This is checked in `_read_group_prepare` and triggers a
warning.
This leads to an inconsistent behavior, where the user requests
aggregating & ordering by `sequence`, we remove it from the aggregated
fields, then warn that they didn't aggregate on the field.
Make the behavior consistent by also ignoring requests to order by
`sequence`.
closesodoo/odoo#97409
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
In case several xml_ids target the same record, now we show an icon with
other xml_ids (not the first one) as tooltip.
task-2954293
closesodoo/odoo#98139
Signed-off-by: Jérémy Kersten <jke@odoo.com>
Consider two models A and B, where
- model A has a many2one_reference field 'res_id' with model field 'res_model';
- model B has an auto-join one2many field 'stuff_ids' to A using field 'res_id';
- the field 'res_model' is not flushed on some record.
model | A | B
-----------+-------------------+-------------------
memory | res_model = B |
-----------+-------------------+-------------------
database | res_model = NULL | id = 42
| res_id = 42 |
| foo = 'bar' |
Now, perform a search on model B that should return record with id=42 by
matching some condition on the unflushed record in model A, like:
B.search([('stuff_ids.foo', '=', 'bar')])
Before this patch, the search method would not flush the field
'res_model', which causes the method to return incorrect results. This
patch fixes the issue by ensuring that searches on one2many fields flush
all the fields on which the one2many field depends.
The issue was discovered while working on task 2735672.
closesodoo/odoo#96115
Signed-off-by: Raphael Collet <rco@odoo.com>
Merging both the memory of field values and suspended updates has
several advantages:
- avoid inconsistencies between cache and towrite
- cache updates can be made safer w.r.t. dirty flag
However, the dirty flag in cache does not go well with context-dependent
fields. When a context-dependent field is dirty in cache, the value to
store in the database is accessible through some context values. But
when the model is flushed, the context values on the current environment
may be different. When this happens, the method flush() fails to
retrieve the data to flush.
The proposed solution is to store the "dirty" value in cache under
conventional context values, and to retrieve them under the same
conventional context values to flush them. For instance, when storing
the value of a binary field, it will be stored once under the context
value `context.get('bin_size')`, and a second time under the context
value `None`. The flush implementation will then retrieve the value
using the context value `None`.
Translated fields are also problematic when a value is put in cache with
an environment where lang=False, and the value is retrieved with another
environment where lang=None. This issue is addressed by normalizing the
context key 'lang' to None when the context value is False.
Part-of: odoo/odoo#95325
Co-authored-by: Vincent Schippefilt <vsc@odoo.com>
There was an issue with the computed `read_group` `__range` when grouping on
the same date/datetime field on multiple granularities (i.e. month, week).
Since the range was stored with the field_name as a key, the last evaluated
range would override the previous ones.
Impacted Versions:
- master
(- exists since 15.0 but it does not impact the user directly so it has been
decided to fix this only in master, since the API is modified)
Steps to reproduce:
1. Open a list view and group by a date field with at least 2 granularities
2. Open the chrome debugger (network) and check a web_read_group rpc preview
3. Find the web_read_group for groups related to one of the largest
granularities and check the `__range`
Current behavior:
- `__range = {field_name: false}`
Expected behavior:
- `__range = {field_name: {from: range_start, to: range_end}`
Explanation
Since the smaller granularities are evaluated last, and the condition to update
`__range` is related to the field_name and not the granularity, the range is
always overriden by the smaller granularities (even if their value is False)
when grouping on the same field with multiple granularities.
Furthermore, there is a conceptual problem with the current solution: it does
not allow to store multiple ranges when the read_group is not lazy and when
grouping on the same field with multiple granularities.
Therefore, the proposed solution is to use the full groupby keys in the
`__range` to allow storing multiple ranges depending on granularity. The keys
in `__range` would thus match the group value keys and allow more flexibility
if a domain must be forged from the group(s) range(s).
Task-2894519
closesodoo/odoo#95193
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
The new feature (search count with limit) introduced by https://github.com/odoo/odoo/pull/95589 lacks of test and code readability:
- Add tests to check the result and query generate
- Increase the readability of SQL/python code.
- Change a little bit the SQL request generate: in the subquery, change `SELECT 1 FROM...` into `SELECT FROM` which avoid extra work in the postgreSQL side.
task-2761165
closesodoo/odoo#95641
Signed-off-by: Raphael Collet <rco@odoo.com>
Search count on large tables can be very slow (it takes 4s to
search_count the list counter of res.partner on our production DB)
This will allow to display a 10000+ counter for large DBs.
closesodoo/odoo#95589
Signed-off-by: Fabien Pinckaers <fp@odoo.com>
This allows to create a repair.order record without
the need to call the onchanges to set the uom and locations
or to set them manually during the `create` call.
For instance, this makes easier to create repair orders
using XMLRPC when you do not use multiple UOMs or multiple locations.
closesodoo/odoo#95321
Signed-off-by: Raphael Collet <rco@odoo.com>
Because method _read() no longer updates existing values in memory,
those values in memory must be consistent with the database.
On the other hand, if pending updates are not performed on the database
before fetching values, it means that the corresponding database values
cannot be put in cache. This implies that one cannot empty the cache
without flushing the corresponding fields.
In order to avoid mistakes, flush automatically before invalidating the
cache. This makes the invalidation methods safe by default, and avoids
cargo-culting which would systematically associate invalidation to
flushing, which may eventually be less performant.
Part-of: odoo/odoo#66938
The idea is to avoid flushing the fields to fetch in method _read().
This delays UPDATE queries, and makes the prefetching mechanism simpler
and more effective.
We do this by not overwriting the cache values by the values fetched
from database. This simple idea allows to fetch more fields and more
records without having to care about pending computations and updates.
But it requires the cache consistency to be much more strict, because
nothing will "fix" the cache inconsistencies "by chance". And it also
requires pending updates to be present in cache.
Part-of: odoo/odoo#66938
Move the definition of class Query to odoo.tools, in order to avoid
circular imports when importing Query in core Odoo modules.
Also reorganize imports in the impacted modules.
Part-of: odoo/odoo#66938
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
Having the parameter stacklevel=2 in warnings.warn() logs the warning
once per location that calls the method with the given warning. This is
what makes sense for warning calls to deprecated methods.
closesodoo/odoo#92411
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This provides a new API for those operations, in order to make the
distinction between the use cases more explicit. The former API was
using obscure parameter combinations to correspond to various cases.
In the summary below, `fnames` is an iterable of field names. If the
parameter is not given, it means "all fields" in the given context.
Note that method recompute() is now mostly private, as it should not be
used in business code.
# process pending computations and updates
records.env.flush_all() # all fields of all models
records.flush_model(fnames) # the fields of all records of the model
records.flush_recordset(fnames) # the fields of the given records
# process pending computations, became non-public methods
records.env._recompute_all() # all fields of all models
records._recompute_model(fnames) # the fields of all records of the model
records._recompute_recordset(fnames) # the fields of the given records
# invalidate the cache of fields
records.env.invalidate_all() # all fields of all models
records.invalidate_model(fnames) # the fields of all records of the model
records.invalidate_recordset(fnames) # the fields of the given records
Part-of: odoo/odoo#87527
If `_get_external_ids` is called in an onchange context on a newid
wrapper, the result map uses NewId keys but the assigned data uses
real ids, which leads to a mis-setting, and usually the call blowing
up immediately as `data['res_id']` is not one of the preallocated dict
entries.
Update the code to better handle this possible difference.
closesodoo/odoo#91957
X-original-commit: 5797fd80a63309269f15bcbe4948d4429a53eec2
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
* use the latest translation API (with fallback on english text if
translation doesn't include expected placeholders)
* indent/split translations to avoid excessive line lengths
closesodoo/odoo#91743
Signed-off-by: Julien Castiaux <juc@odoo.com>
When fetching the views of a one2many/many2many fields
in a form view, do not propagate the context keys
`form_view_ref`, `tree_view_ref`, ...
It was supposed to be already handled,
but the keys were not removed when the arg `view_id`
is passed to `_get_view`.
closesodoo/odoo#91489
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
On a `sale.order`, when hitting the `View Forecast` button,
a traceback occurred:
```
Traceback (most recent call last):
File "/home/odoo/src/odoo/master/odoo/http.py", line 1381, in _serve_db
return service_model.retrying(self._serve_ir_http, self.env)
File "/home/odoo/src/odoo/master/odoo/service/model.py", line 136, in retrying
result = func()
File "/home/odoo/src/odoo/master/odoo/http.py", line 1410, in _serve_ir_http
response = self.dispatcher.dispatch(rule.endpoint, args)
File "/home/odoo/src/odoo/master/odoo/http.py", line 1607, in dispatch
result = self.request.registry['ir.http']._dispatch(endpoint)
File "/home/odoo/src/odoo/master/addons/website/models/ir_http.py", line 220, in _dispatch
response = super()._dispatch(endpoint)
File "/home/odoo/src/odoo/master/addons/utm/models/ir_http.py", line 27, in _dispatch
return super()._dispatch(endpoint)
File "/home/odoo/src/odoo/master/odoo/addons/base/models/ir_http.py", line 137, in _dispatch
result = endpoint(**request.params)
File "/home/odoo/src/odoo/master/odoo/http.py", line 541, in route_wrapper
result = endpoint(self, *args, **params_ok)
File "/home/odoo/src/odoo/master/addons/web/controllers/dataset.py", line 42, in call_kw
return self._call_kw(model, method, args, kwargs)
File "/home/odoo/src/odoo/master/addons/web/controllers/dataset.py", line 33, in _call_kw
return call_kw(request.env[model], method, args, kwargs)
File "/home/odoo/src/odoo/master/odoo/api.py", line 457, in call_kw
result = _call_kw_model(method, model, args, kwargs)
File "/home/odoo/src/odoo/master/odoo/api.py", line 430, in _call_kw_model
result = method(recs, *args, **kwargs)
File "/home/odoo/src/odoo/master/odoo/models.py", line 1808, in fields_view_get
view = self.env['ir.ui.view'].sudo(result.pop('id'))
File "/home/odoo/src/odoo/master/odoo/models.py", line 5381, in sudo
assert isinstance(flag, bool)
AssertionError
```
This follows the `load_views` refactor odoo/odoo#87522.
`fields_view_get` has been deprecated. All calls to it must be replaced
by calls to `get_views`.
Leaving it there is an oversight:
https://github.com/odoo/odoo/blob/fc26cc1789b1309e54473cccbc545145ccbaee31/addons/stock/static/src/js/report_stock_forecasted.js#L116
However, this demonstrated an issue in the compatibility layer added
server-side to keep `fields_view_get`.
This revisions solves the compatibility layer bug.
In the next commit, the call to `fields_view_get` is replaced to
a call to `get_views`
Part-of: odoo/odoo#90750
Now that the method is deprecated, we think it's better to
keep showing it in the doc, but with a clear deprecation notice
in the docstring (instead of hiding it).
closesodoo/odoo#90634
Related: odoo/documentation#1908
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Fixing docstring of new get_view method (odoo/odoo#87522),
to include it in the doc (and replace reference to the deprecated
method fields_view_get).
models.py:docstring of odoo.models.BaseModel.get_view:8: WARNING: Unexpected indentation.
models.py:docstring of odoo.models.BaseModel.get_view:9: WARNING: Block quote ends without a blank line; unexpected unindent.
+ some little improvements to the docstring
closesodoo/odoo#90372
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
The base model is modified in order to be able to make comparisons between
recordsets contained in lazy values. The isinstance method will always
return an error, so we consider that we receive a recordset and try to
access the `_name` and `_ids`. If an error is triggered, it was not a
recordset.
This way of doing it involves little change in performance (improvement
when it is good and decrease when there is an error) because we do not
test if it is a recordset before comparing it.
closesodoo/odoo#89141
Signed-off-by: Rémy Voet <ryv@odoo.com>
- Filter in attributes rather than filter out.
The goal is to not gather attributes which are
costly in term of performance if they are not requested
in the first place.
e.g., with all modules installed:
- Before rev:
```py
In [1]: %time for _i in range(1000): self.env["res.partner"].fields_get(attributes=['readonly', 'required', 'states', 'invisible']);self.invalidate_cache()
CPU times: user 1.99 s, sys: 9.83 ms, total: 2 s
Wall time: 2.03 s
```
- After rev:
```py
In [2]: %time for _i in range(1000): self.env["res.partner"].fields_get(attributes=['readonly', 'required', 'states', 'invisible']);self.invalidate_cache()
CPU times: user 345 ms, sys: 0 ns, total: 345 ms
Wall time: 345 ms
```
- Use the `_description_` mechanism for the attributes `name` and `type`,
so its no longer needed to treat them as exception in
`field_get` and `get_description` respectively,
and make the code shorter and cleaner.
- Move out from `fields_get` the block
```py
has_access = functools.partial(self.check_access_rights, raise_exception=False)
readonly = not (has_access('write') or has_access('create'))
...
if readonly:
description['readonly'] = True
description['states'] = {}
```
because:
- It meant you had a different behavior using `fields_get` or `get_description`
for the keys `readonly` and `states`, meaning a field could be marked as `readonly`
by `fields_get` but not by `get_description`, which is confusing.
- It is actually used in only one place, the post-process of back-end views,
which mark the fields readonly for the web client if you do not have
the create or write access to their model.
closesodoo/odoo#87273
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Refactor the `load_views` API so it no longer sends multiple times the same
fields description.
e.g.
When `load_views` is called to get the kanban, tree and form views,
the list of fields of the model was sent 4 times:
- Once for each view, with only the fields used in the view,
in `['fields_views']['kanban']['fields']` for instance
- Once globally, with all the fields of the model, in `['fields']`
The goal of this revision is to change that so it sends the list of all fields
only once.
In addition, if a view contains x2many fields,
the fields description of the comodel is also sent.
It was sent in the `views` key of the view fields dict.
e.g.
When calling `load_views` of `res.partner` to get the kanban,
tree and form views,
the `res.partner` fields description was actually sent 6 times:
- Once for each view
- Once globally
- Once for each view of the many2many field `child_ids` of the form view, in
- `['fields_views']['form']['fields']['child_ids']['views']['kanban']['fields']`
- `['fields_views']['form']['fields']['child_ids']['views']['form']['fields']`
The change suggested in this revision is to:
- Remove the fields description for each view in `['fields_views']`.
As it no longer contains the fields,
the key becomes `['views']` instead of `['fields_views']`.
- Replace the dict key `['fields']` by `['models']`,
which is a dict with as key the model name and as values
the model fields description. It contains the fields description
for all models implied in the view:
the model of the main view and the model of all one2many and many2many fields.
With this change, the fields description will only be sent once by model
implied in the view.
In addition, the web client was getting the information about the fields
sometimes in the global fields description list (e.g. `['fields']`),
sometimes in the fields description list of the view type
(e.g. `['fields_views']['form']['fields']`),
making it a pain to try to make changes / performance gain
in these field description dictionaries, because you never knew in which dict
the web client was getting its info.
Now, as there is only one place to get the fields description from,
it's clearer and cleaner.
- one2many and many2many fields views are passed directly in the main view
architecture rather than being put in the `views` key
of the field description.
This is actually easier to treat by the web client,
and this will allow in a future work to cache an entire view in one block
of text rather than having to combine multiple cached blocks of text
to return one view.
- one2many and many2many fields which do not have directly embedded views
have their views directly injected in the architecture,
so the web client doesn't have to do RPC calls to `load_views`
for each one2many and many2many fields not having embedded views.
For instance, this allow to reduce the number of RPC calls to `load_views`
from 8 to 1 when loading the form of `product.product`.
Currently, this behavior is limited to 1 level deep but we consider making it
go all the way down in future works. We did not do it for the moment because
in certain cases it rises the processing time and the size (bytes) too much.
e.g. the sale.order view can be 5 levels deep,
meaning you can reach 4 dialogs on top the main view.
```
sale.order form > order_line > sale.order.line form > invoice_lines >
account.move.line form > asset_ids > account.asset form >
depreciation_move_ids > account.move form.
```
This will also benefit in future works to cache an entire view in one block
of text rather to having to combine multiple cached block of text
to get one view.
- `fields_view_get` becomes `get_view`.
As it no longer returns the fields description,
keeping the `fields` in the name `fields_view_get` no longer makes sense.
Hence removing `fields` from the method name, it becomes `view_get`.
As it gets renamed anyway, we take the opportunity to rename it `get_view`,
which is more in line with the general getter/setter guidelines
in the model object world.
- `_fields_view_get` becomes `_get_view`. For the same reasons than above.
- `load_views` becomes `get_views`.
This is not mandatory, there is no technical reason to rename `load_views` as
it practically sends the same info as before,
the view architectures and their fields description. Just in another way.
We just take the opportunity of this pull request to suggest a cleaner API:
`_get_view`, `get_view` and `get_views`.
- Arguments `toolbar=False, submenu=False` fo the methods
`_fields_view_get` and `fields_view_get` are converted to a kwargs `**options`
in `_get_view` and `get_view`.
The rationale is that submenu was already no longer used (deprecated)
and the mobile options is introduced.
The mobile options is necessary to tell the server to send the mobile views
for x2many fields (kanban instead of tree).
Instead of adding a new argument each time we add a new option to
`fields_view_get`, it seems wiser to have a kwargs `**options` to avoid
to re-write all overrides each time a new option is introduced.
- `_fields_view_get` returned a dict containing the arch in text and some of the
view information. Now, `get_view` returns a tuple with the view architecture
as an `etree` node, and the view as a browse record. The rationale is that all
overrides of `_fields_view_get` were about modifying the arch only
(e.g. changing the address format/re-organizing the address related field
nodes of the partner according to the company country).
To do so, all these overrides were doing `etree.fromstring` to parse the arch
which was sent in text to convert it to an `etree`,
then operations were done on the `etree`,
and then `etree.tostring` was called to convert back the arch to string.
With this change of signature to send the arch as an `etree`,
all these back and forth `etree.fromstring` -> `etree.tostring` are avoided,
allowing some performance gain and less code in the end.
- A cleanup of the keys returned in the dict of `fields_view_get`
has been performed in `get_view`:
- `fields` is removed, as explained above,
- `view_id` is renamed `id`,
- `name` is removed, it was unused by the web client,
- `type` is removed, it was unused by the web client,
- `field_parent` is removed, it was unused by the web client,
- `base_model` is removed, it was unused by the web client.
- `filters` is moved from the global dict returned by `load_views`
(now `get_views`) to the dict returned by `fields_view_get` (now `get_view`)
as it applies only to the `search` view type.
- Retro-compatible methods for the 3 methods
`fields_view_get`, `_fields_view_get` and `load_views` are provided,
with deprecation warnings in them.
- The web client could cache the model fields description
(as it already caches the views),
so it doesn't need to fetch them again if it asks for another view of a model
for which he already has the fields description.
If we do so, `get_views` could return only the list of models used by
the views, without the fields description as of now,
and the web client would then call `fields_get` independently only for
the models for which it doesn't have yet the fields description.
This would avoid the server to return the fields description
and to call `fields_get`, which is costly, for each `get_views`,
therefore gaining performances.
- Inject the views of the one2many and many2many fields all the way down,
unlimited depth level, as explained above.
- Cache with `ormcache` the architecture of back-end views.
This is already done for qweb views, it's not done for back-end views.
Therefore the postprocessing of the views is performed for each `get_views`,
which is costly, while the view architecture doesn't change for users
belonging to the same groups, according to the groups implied by the view.
This pull request is co-authored by
Aaron Bohy (aab) for the web client part and
Denis Ledoux (dle) for the server part.
Part-of: odoo/odoo#87522
This is a followup of #86567, where we change the computation of
display_name to match the values of name_get(). Forcing its value to
False instead of the empty string causes many regressions in frontend
applications like website_blog.
closesodoo/odoo#88438
X-original-commit: d467881f66b11021005005584854f30059971b6c
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Xavier Morel