The master plan finally comes to an end.
Thanks to:
- odoo/odoo#87522 refactoring `load_views`,
- odoo/odoo#94337 refactoring `common.Form` to prevent changing
invisible fields in unit tests using `Form` instances,
- odoo/odoo#95729 refactoring the behavior of `groups=` in views,
- odoo/odoo#98551 removing the need of the `groups_id` many2many field
on back-end views.
The result returned by `get_view`/`get_views` can now finally be easily
and efficiently cached, in order to cache back-end views.
The goal of this revision is to cache the model views and fields
already post-processed for the web client
(with the modifiers, etc., already computed)
without group restriction.
Then, from this cached version, post-process group related features,
such as removing nodes restricted with a `groups=` attribute,
set the create/write button according to the user access rights to models, ...
Not including the groups in the cache key allows:
- to have less cached versions,
(otherwise it would be one cached version per different group combination)
- to not have to fetch the user groups to compute the key
(with the current cache key,
there is nothing to fetch from the database to compute the key)
Besides, post-processing the groups features
after taking the view from the cache of the view doesn't take a tremendous time:
- parsing arch from/to string with etree is fast,
- removing the `groups=` nodes using etree is fast,
- adding the `create="False"`, `write="False"`, `delete="False"` on the view
root node according to the access right of the user on the model is fast.
This allows way faster calls to `get_views` by the web client,
as the server no longer need, for each call, to fetch the views in database,
combine the inherited views, post-process the modifiers attributes, etc.
Timing tests are available on the pull request of this revision.
Part-of: odoo/odoo#99417
The cache key of _get_bindings was not super efficient.
The result of _get_bindings is cached,
but its performance was altered by the cache
key which requires to fetch the user groups for each call to
_get_bindings.
Besides, as there is a lot of possible group
combination, this resulted in a lot of possible cache keys,
and therefore a lot of cached values.
This revision aims to make _get_bindings more efficient
by:
- do not use the groups in the cache keys (less cached values)
- filter out actions not available to the user groups after
retrieving them from the cache
- use has_group to do the above, which is itself cached as well,
and therefore do not need to fetch the user groups
at each call to get_bindings.
In addition, move get_bindings from `get_view`
to `get_views`. If there was 3 views asked by `get_views`
(let's say kanban, list, form)
`get_bindings` was being called 3 times, through `get_view`
with each time the same arguments and therefore the same result :-).
Moving it to `get_views` allows to call it only once for all view types
requested, and for the web client it doesn't change much,
as it always request the toolbar/get_bindings through `get_views` only.
In addition, add the lang to the cache of _get_bindings.
it was actually a bug not to put it: if you had 2 users
with the same group set, using 2 different languages,
the user accessing first the get_bindings would cache
the action names within his language, and then the second
user would see the action name within the language of the first user
:-).
Before
```py
In [1]: %time for i in range(1000): self.env['ir.actions.actions'].get_bindings('res.partner'); self.env.invalidate_all();
CPU times: user 790 ms, sys: 104 ms, total: 893 ms
Wall time: 1.7 s
```
After
```py
In [1]: %time for i in range(1000): self.env['ir.actions.actions'].get_bindings('res.partner'); self.env.invalidate_all();
CPU times: user 23.5 ms, sys: 9.12 ms, total: 32.7 ms
Wall time: 36.9 ms
```
Part-of: odoo/odoo#99417
For non-stored computed fields, `_modified_triggers` will traverse the
tree (at the cost of extra queries) only to know which record to
invalidate in cache. But in most cases, these fields have no data in
cache, so they can be ignored from the start, which allows us to prune
entire subtrees from the merged tree.
By example:
With simple write on `show_operations` of one `stock.picking.type`, the
`_modified_triggers` will fetch every ids (from database) of `stock.picking`
and `stock.move` related to this `stock.picking.type`
(For `stock.move`, it is because of the
`show_operations = fields.Boolean(related='picking_id.picking_type_id.show_operations'`))
In that case, there isn't any data of `show_operations` (`stock.move`)
in cache, then there are nothing to invalidate and
the `_modified_triggers` cost
is high (extra queries/processing) for nothing.
Then we cut parts of the tree when we know that
they won't invalidate anything (=> if the cache is empty for the field).
Also refactor the way to merge trees to be more efficient.
Performance improvements:
- For all tests at-install done by the runbot, we gain -+ 3.5% of
queries.
- In the example above, we reduce the number of queries (potentially
bottleneck queries in large DB) from 7 to 4.
- In term of CPU (without counting time in SQL), the new version is -+
20 % faster (on install of stock,purchase,mrp and with
--test-tags=/stock,/purchase).
closesodoo/odoo#76322
task-2780812
closesodoo/odoo#99274
Related: odoo/enterprise#30951
Signed-off-by: Raphael Collet <rco@odoo.com>
`_check_concurrency` was used to check if 2 people were modifying the
same record at the same time. It was doing so by setting a special
`__last_update` value inside the context that was later evaluted to
prevent some concurrency issues.
It was mostly unused, wasted a lot of cpu cycles and was not covering
all cases (e.g. pending write).
closesodoo/odoo#87756
Signed-off-by: Vincent Schippefilt (vsc) <vsc@odoo.com>
Purpose
=======
Make the onchange work for the properties fields. When changing the
container field, we need to update the properties definition.
Task-2852259
Part-of: odoo/odoo#95184
Purpose
=======
Add a new field "Properties" to be able to light customization of workflows
based on a parent model. Those properties acts in some ways like Odoo fields
without requiring specific columns e.g. add new properties on tasks of a
specific project.
Usage
=====
Define properties on a parent model (e.g. project) with
```
attributes_definition = fields.PropertiesDefinition('Message Properties')
```
It defines properties available on children: types, default, value, model for
relational properties,...
Use it on children records (e.g. task) with
```
attributes = fields.Properties(
string='Properties',
definition='parent_id.attributes_definition',
)
```
Technical
=========
Parent | Properties definition
------------------------------
The properties definition is stored on the parent, on a JSON field.
This definition contains the type of the properties, the default value,
the model of the many2one,...
```
[
{
'name': 'name',
'string': 'Name',
'type': 'char',
'default': 'Default Name',
}, {
'name': 'partner_id',
'string': 'Partner',
'type': 'many2one',
'comodel': 'res.partner',
},
]
```
Child | Properties values
-------------------------
The value is stored on the child, using a Properties field.
```
{
'name': 'Mitchel',
'partner_id': 1337,
}
```
When we read this field, we will automatically read the definition on
the parent, and merge both JSON into one, so the web client has the
value of each property, and their definition.
```
[
{
'name': 'name',
'string': 'Name',
'type': 'char',
'default': 'Default Name',
'value': 'Mitchel',
}, {
'name': 'partner_id',
'string': 'Partner',
'type': 'many2one',
'comodel': 'res.partner',
'value': 1337,
},
]
```
Integrity
---------
If we remove a property on the parent, we won't update the child value.
Instead, when we read the child properties, we will filter them based
on the parent. So the removed properties will be removed the next time
we write on the field.
In the same logic, the many2one existence is checked when we read the
field. There's no foreign key between the integer stored in the JSON
in the SQL row corresponding to the record in database.
Write
-----
We can write on the Properties field with a list of field definition
+ value.
Some types are not JSONifiable (like the date, datetime), they are
stored as string in database and parsed when we read the value.
In order to update the parent definition by writing on the child,
you need to add the dict key `definition_changed` or
`definition_deleted`. This is because we need to be able to know
if the definition has been changed without doing extra SQL queries.
Access rights
-------------
A user can add a many2one / many2many property to a model only if he
has the access rights to it.
Many2one / Many2many
--------------------
The model choice of a many2one / many2many properties was subject to
changes.
First implementation stored models in both parent and children to easily
spot changes and avoid complex queries when fetching records, trying to
synchronize them, ...
As this leads to storing a lot of duplicated content we choose to instead
reset the value on the child if the model has been change. We generate a
new name for the property. So it behaves like if we removed the property
and created a new one.
To be able to restore the old value (e.g. if by mistake we changed the
model, and go back to the old model), we store the initial states.
Task-2852259
Part-of: odoo/odoo#95184
The sequence field is special-cased early on in read_group (_raw): if
the caller requests the aggregation of ``sequence`` that request is
ignored:
if fspec == 'sequence':
continue
This was added a long time ago, probably due to the special-ish status
of `sequence` (summing sequence number doesn't really make much
sense).
The issue is that it's also possible to request ordering by
`sequence`, which requires `sequence` to be one of the aggregated
fields. This is checked in `_read_group_prepare` and triggers a
warning.
This leads to an inconsistent behavior, where the user requests
aggregating & ordering by `sequence`, we remove it from the aggregated
fields, then warn that they didn't aggregate on the field.
Make the behavior consistent by also ignoring requests to order by
`sequence`.
closesodoo/odoo#97409
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
In case several xml_ids target the same record, now we show an icon with
other xml_ids (not the first one) as tooltip.
task-2954293
closesodoo/odoo#98139
Signed-off-by: Jérémy Kersten <jke@odoo.com>
Consider two models A and B, where
- model A has a many2one_reference field 'res_id' with model field 'res_model';
- model B has an auto-join one2many field 'stuff_ids' to A using field 'res_id';
- the field 'res_model' is not flushed on some record.
model | A | B
-----------+-------------------+-------------------
memory | res_model = B |
-----------+-------------------+-------------------
database | res_model = NULL | id = 42
| res_id = 42 |
| foo = 'bar' |
Now, perform a search on model B that should return record with id=42 by
matching some condition on the unflushed record in model A, like:
B.search([('stuff_ids.foo', '=', 'bar')])
Before this patch, the search method would not flush the field
'res_model', which causes the method to return incorrect results. This
patch fixes the issue by ensuring that searches on one2many fields flush
all the fields on which the one2many field depends.
The issue was discovered while working on task 2735672.
closesodoo/odoo#96115
Signed-off-by: Raphael Collet <rco@odoo.com>
Merging both the memory of field values and suspended updates has
several advantages:
- avoid inconsistencies between cache and towrite
- cache updates can be made safer w.r.t. dirty flag
However, the dirty flag in cache does not go well with context-dependent
fields. When a context-dependent field is dirty in cache, the value to
store in the database is accessible through some context values. But
when the model is flushed, the context values on the current environment
may be different. When this happens, the method flush() fails to
retrieve the data to flush.
The proposed solution is to store the "dirty" value in cache under
conventional context values, and to retrieve them under the same
conventional context values to flush them. For instance, when storing
the value of a binary field, it will be stored once under the context
value `context.get('bin_size')`, and a second time under the context
value `None`. The flush implementation will then retrieve the value
using the context value `None`.
Translated fields are also problematic when a value is put in cache with
an environment where lang=False, and the value is retrieved with another
environment where lang=None. This issue is addressed by normalizing the
context key 'lang' to None when the context value is False.
Part-of: odoo/odoo#95325
Co-authored-by: Vincent Schippefilt <vsc@odoo.com>
There was an issue with the computed `read_group` `__range` when grouping on
the same date/datetime field on multiple granularities (i.e. month, week).
Since the range was stored with the field_name as a key, the last evaluated
range would override the previous ones.
Impacted Versions:
- master
(- exists since 15.0 but it does not impact the user directly so it has been
decided to fix this only in master, since the API is modified)
Steps to reproduce:
1. Open a list view and group by a date field with at least 2 granularities
2. Open the chrome debugger (network) and check a web_read_group rpc preview
3. Find the web_read_group for groups related to one of the largest
granularities and check the `__range`
Current behavior:
- `__range = {field_name: false}`
Expected behavior:
- `__range = {field_name: {from: range_start, to: range_end}`
Explanation
Since the smaller granularities are evaluated last, and the condition to update
`__range` is related to the field_name and not the granularity, the range is
always overriden by the smaller granularities (even if their value is False)
when grouping on the same field with multiple granularities.
Furthermore, there is a conceptual problem with the current solution: it does
not allow to store multiple ranges when the read_group is not lazy and when
grouping on the same field with multiple granularities.
Therefore, the proposed solution is to use the full groupby keys in the
`__range` to allow storing multiple ranges depending on granularity. The keys
in `__range` would thus match the group value keys and allow more flexibility
if a domain must be forged from the group(s) range(s).
Task-2894519
closesodoo/odoo#95193
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
The new feature (search count with limit) introduced by https://github.com/odoo/odoo/pull/95589 lacks of test and code readability:
- Add tests to check the result and query generate
- Increase the readability of SQL/python code.
- Change a little bit the SQL request generate: in the subquery, change `SELECT 1 FROM...` into `SELECT FROM` which avoid extra work in the postgreSQL side.
task-2761165
closesodoo/odoo#95641
Signed-off-by: Raphael Collet <rco@odoo.com>
Search count on large tables can be very slow (it takes 4s to
search_count the list counter of res.partner on our production DB)
This will allow to display a 10000+ counter for large DBs.
closesodoo/odoo#95589
Signed-off-by: Fabien Pinckaers <fp@odoo.com>
This allows to create a repair.order record without
the need to call the onchanges to set the uom and locations
or to set them manually during the `create` call.
For instance, this makes easier to create repair orders
using XMLRPC when you do not use multiple UOMs or multiple locations.
closesodoo/odoo#95321
Signed-off-by: Raphael Collet <rco@odoo.com>
Because method _read() no longer updates existing values in memory,
those values in memory must be consistent with the database.
On the other hand, if pending updates are not performed on the database
before fetching values, it means that the corresponding database values
cannot be put in cache. This implies that one cannot empty the cache
without flushing the corresponding fields.
In order to avoid mistakes, flush automatically before invalidating the
cache. This makes the invalidation methods safe by default, and avoids
cargo-culting which would systematically associate invalidation to
flushing, which may eventually be less performant.
Part-of: odoo/odoo#66938
The idea is to avoid flushing the fields to fetch in method _read().
This delays UPDATE queries, and makes the prefetching mechanism simpler
and more effective.
We do this by not overwriting the cache values by the values fetched
from database. This simple idea allows to fetch more fields and more
records without having to care about pending computations and updates.
But it requires the cache consistency to be much more strict, because
nothing will "fix" the cache inconsistencies "by chance". And it also
requires pending updates to be present in cache.
Part-of: odoo/odoo#66938
Move the definition of class Query to odoo.tools, in order to avoid
circular imports when importing Query in core Odoo modules.
Also reorganize imports in the impacted modules.
Part-of: odoo/odoo#66938
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
Having the parameter stacklevel=2 in warnings.warn() logs the warning
once per location that calls the method with the given warning. This is
what makes sense for warning calls to deprecated methods.
closesodoo/odoo#92411
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This provides a new API for those operations, in order to make the
distinction between the use cases more explicit. The former API was
using obscure parameter combinations to correspond to various cases.
In the summary below, `fnames` is an iterable of field names. If the
parameter is not given, it means "all fields" in the given context.
Note that method recompute() is now mostly private, as it should not be
used in business code.
# process pending computations and updates
records.env.flush_all() # all fields of all models
records.flush_model(fnames) # the fields of all records of the model
records.flush_recordset(fnames) # the fields of the given records
# process pending computations, became non-public methods
records.env._recompute_all() # all fields of all models
records._recompute_model(fnames) # the fields of all records of the model
records._recompute_recordset(fnames) # the fields of the given records
# invalidate the cache of fields
records.env.invalidate_all() # all fields of all models
records.invalidate_model(fnames) # the fields of all records of the model
records.invalidate_recordset(fnames) # the fields of the given records
Part-of: odoo/odoo#87527
If `_get_external_ids` is called in an onchange context on a newid
wrapper, the result map uses NewId keys but the assigned data uses
real ids, which leads to a mis-setting, and usually the call blowing
up immediately as `data['res_id']` is not one of the preallocated dict
entries.
Update the code to better handle this possible difference.
closesodoo/odoo#91957
X-original-commit: 5797fd80a63309269f15bcbe4948d4429a53eec2
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
* use the latest translation API (with fallback on english text if
translation doesn't include expected placeholders)
* indent/split translations to avoid excessive line lengths
closesodoo/odoo#91743
Signed-off-by: Julien Castiaux <juc@odoo.com>
When fetching the views of a one2many/many2many fields
in a form view, do not propagate the context keys
`form_view_ref`, `tree_view_ref`, ...
It was supposed to be already handled,
but the keys were not removed when the arg `view_id`
is passed to `_get_view`.
closesodoo/odoo#91489
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
On a `sale.order`, when hitting the `View Forecast` button,
a traceback occurred:
```
Traceback (most recent call last):
File "/home/odoo/src/odoo/master/odoo/http.py", line 1381, in _serve_db
return service_model.retrying(self._serve_ir_http, self.env)
File "/home/odoo/src/odoo/master/odoo/service/model.py", line 136, in retrying
result = func()
File "/home/odoo/src/odoo/master/odoo/http.py", line 1410, in _serve_ir_http
response = self.dispatcher.dispatch(rule.endpoint, args)
File "/home/odoo/src/odoo/master/odoo/http.py", line 1607, in dispatch
result = self.request.registry['ir.http']._dispatch(endpoint)
File "/home/odoo/src/odoo/master/addons/website/models/ir_http.py", line 220, in _dispatch
response = super()._dispatch(endpoint)
File "/home/odoo/src/odoo/master/addons/utm/models/ir_http.py", line 27, in _dispatch
return super()._dispatch(endpoint)
File "/home/odoo/src/odoo/master/odoo/addons/base/models/ir_http.py", line 137, in _dispatch
result = endpoint(**request.params)
File "/home/odoo/src/odoo/master/odoo/http.py", line 541, in route_wrapper
result = endpoint(self, *args, **params_ok)
File "/home/odoo/src/odoo/master/addons/web/controllers/dataset.py", line 42, in call_kw
return self._call_kw(model, method, args, kwargs)
File "/home/odoo/src/odoo/master/addons/web/controllers/dataset.py", line 33, in _call_kw
return call_kw(request.env[model], method, args, kwargs)
File "/home/odoo/src/odoo/master/odoo/api.py", line 457, in call_kw
result = _call_kw_model(method, model, args, kwargs)
File "/home/odoo/src/odoo/master/odoo/api.py", line 430, in _call_kw_model
result = method(recs, *args, **kwargs)
File "/home/odoo/src/odoo/master/odoo/models.py", line 1808, in fields_view_get
view = self.env['ir.ui.view'].sudo(result.pop('id'))
File "/home/odoo/src/odoo/master/odoo/models.py", line 5381, in sudo
assert isinstance(flag, bool)
AssertionError
```
This follows the `load_views` refactor odoo/odoo#87522.
`fields_view_get` has been deprecated. All calls to it must be replaced
by calls to `get_views`.
Leaving it there is an oversight:
https://github.com/odoo/odoo/blob/fc26cc1789b1309e54473cccbc545145ccbaee31/addons/stock/static/src/js/report_stock_forecasted.js#L116
However, this demonstrated an issue in the compatibility layer added
server-side to keep `fields_view_get`.
This revisions solves the compatibility layer bug.
In the next commit, the call to `fields_view_get` is replaced to
a call to `get_views`
Part-of: odoo/odoo#90750
Now that the method is deprecated, we think it's better to
keep showing it in the doc, but with a clear deprecation notice
in the docstring (instead of hiding it).
closesodoo/odoo#90634
Related: odoo/documentation#1908
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Fixing docstring of new get_view method (odoo/odoo#87522),
to include it in the doc (and replace reference to the deprecated
method fields_view_get).
models.py:docstring of odoo.models.BaseModel.get_view:8: WARNING: Unexpected indentation.
models.py:docstring of odoo.models.BaseModel.get_view:9: WARNING: Block quote ends without a blank line; unexpected unindent.
+ some little improvements to the docstring
closesodoo/odoo#90372
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
The base model is modified in order to be able to make comparisons between
recordsets contained in lazy values. The isinstance method will always
return an error, so we consider that we receive a recordset and try to
access the `_name` and `_ids`. If an error is triggered, it was not a
recordset.
This way of doing it involves little change in performance (improvement
when it is good and decrease when there is an error) because we do not
test if it is a recordset before comparing it.
closesodoo/odoo#89141
Signed-off-by: Rémy Voet <ryv@odoo.com>
- Filter in attributes rather than filter out.
The goal is to not gather attributes which are
costly in term of performance if they are not requested
in the first place.
e.g., with all modules installed:
- Before rev:
```py
In [1]: %time for _i in range(1000): self.env["res.partner"].fields_get(attributes=['readonly', 'required', 'states', 'invisible']);self.invalidate_cache()
CPU times: user 1.99 s, sys: 9.83 ms, total: 2 s
Wall time: 2.03 s
```
- After rev:
```py
In [2]: %time for _i in range(1000): self.env["res.partner"].fields_get(attributes=['readonly', 'required', 'states', 'invisible']);self.invalidate_cache()
CPU times: user 345 ms, sys: 0 ns, total: 345 ms
Wall time: 345 ms
```
- Use the `_description_` mechanism for the attributes `name` and `type`,
so its no longer needed to treat them as exception in
`field_get` and `get_description` respectively,
and make the code shorter and cleaner.
- Move out from `fields_get` the block
```py
has_access = functools.partial(self.check_access_rights, raise_exception=False)
readonly = not (has_access('write') or has_access('create'))
...
if readonly:
description['readonly'] = True
description['states'] = {}
```
because:
- It meant you had a different behavior using `fields_get` or `get_description`
for the keys `readonly` and `states`, meaning a field could be marked as `readonly`
by `fields_get` but not by `get_description`, which is confusing.
- It is actually used in only one place, the post-process of back-end views,
which mark the fields readonly for the web client if you do not have
the create or write access to their model.
closesodoo/odoo#87273
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Refactor the `load_views` API so it no longer sends multiple times the same
fields description.
e.g.
When `load_views` is called to get the kanban, tree and form views,
the list of fields of the model was sent 4 times:
- Once for each view, with only the fields used in the view,
in `['fields_views']['kanban']['fields']` for instance
- Once globally, with all the fields of the model, in `['fields']`
The goal of this revision is to change that so it sends the list of all fields
only once.
In addition, if a view contains x2many fields,
the fields description of the comodel is also sent.
It was sent in the `views` key of the view fields dict.
e.g.
When calling `load_views` of `res.partner` to get the kanban,
tree and form views,
the `res.partner` fields description was actually sent 6 times:
- Once for each view
- Once globally
- Once for each view of the many2many field `child_ids` of the form view, in
- `['fields_views']['form']['fields']['child_ids']['views']['kanban']['fields']`
- `['fields_views']['form']['fields']['child_ids']['views']['form']['fields']`
The change suggested in this revision is to:
- Remove the fields description for each view in `['fields_views']`.
As it no longer contains the fields,
the key becomes `['views']` instead of `['fields_views']`.
- Replace the dict key `['fields']` by `['models']`,
which is a dict with as key the model name and as values
the model fields description. It contains the fields description
for all models implied in the view:
the model of the main view and the model of all one2many and many2many fields.
With this change, the fields description will only be sent once by model
implied in the view.
In addition, the web client was getting the information about the fields
sometimes in the global fields description list (e.g. `['fields']`),
sometimes in the fields description list of the view type
(e.g. `['fields_views']['form']['fields']`),
making it a pain to try to make changes / performance gain
in these field description dictionaries, because you never knew in which dict
the web client was getting its info.
Now, as there is only one place to get the fields description from,
it's clearer and cleaner.
- one2many and many2many fields views are passed directly in the main view
architecture rather than being put in the `views` key
of the field description.
This is actually easier to treat by the web client,
and this will allow in a future work to cache an entire view in one block
of text rather than having to combine multiple cached blocks of text
to return one view.
- one2many and many2many fields which do not have directly embedded views
have their views directly injected in the architecture,
so the web client doesn't have to do RPC calls to `load_views`
for each one2many and many2many fields not having embedded views.
For instance, this allow to reduce the number of RPC calls to `load_views`
from 8 to 1 when loading the form of `product.product`.
Currently, this behavior is limited to 1 level deep but we consider making it
go all the way down in future works. We did not do it for the moment because
in certain cases it rises the processing time and the size (bytes) too much.
e.g. the sale.order view can be 5 levels deep,
meaning you can reach 4 dialogs on top the main view.
```
sale.order form > order_line > sale.order.line form > invoice_lines >
account.move.line form > asset_ids > account.asset form >
depreciation_move_ids > account.move form.
```
This will also benefit in future works to cache an entire view in one block
of text rather to having to combine multiple cached block of text
to get one view.
- `fields_view_get` becomes `get_view`.
As it no longer returns the fields description,
keeping the `fields` in the name `fields_view_get` no longer makes sense.
Hence removing `fields` from the method name, it becomes `view_get`.
As it gets renamed anyway, we take the opportunity to rename it `get_view`,
which is more in line with the general getter/setter guidelines
in the model object world.
- `_fields_view_get` becomes `_get_view`. For the same reasons than above.
- `load_views` becomes `get_views`.
This is not mandatory, there is no technical reason to rename `load_views` as
it practically sends the same info as before,
the view architectures and their fields description. Just in another way.
We just take the opportunity of this pull request to suggest a cleaner API:
`_get_view`, `get_view` and `get_views`.
- Arguments `toolbar=False, submenu=False` fo the methods
`_fields_view_get` and `fields_view_get` are converted to a kwargs `**options`
in `_get_view` and `get_view`.
The rationale is that submenu was already no longer used (deprecated)
and the mobile options is introduced.
The mobile options is necessary to tell the server to send the mobile views
for x2many fields (kanban instead of tree).
Instead of adding a new argument each time we add a new option to
`fields_view_get`, it seems wiser to have a kwargs `**options` to avoid
to re-write all overrides each time a new option is introduced.
- `_fields_view_get` returned a dict containing the arch in text and some of the
view information. Now, `get_view` returns a tuple with the view architecture
as an `etree` node, and the view as a browse record. The rationale is that all
overrides of `_fields_view_get` were about modifying the arch only
(e.g. changing the address format/re-organizing the address related field
nodes of the partner according to the company country).
To do so, all these overrides were doing `etree.fromstring` to parse the arch
which was sent in text to convert it to an `etree`,
then operations were done on the `etree`,
and then `etree.tostring` was called to convert back the arch to string.
With this change of signature to send the arch as an `etree`,
all these back and forth `etree.fromstring` -> `etree.tostring` are avoided,
allowing some performance gain and less code in the end.
- A cleanup of the keys returned in the dict of `fields_view_get`
has been performed in `get_view`:
- `fields` is removed, as explained above,
- `view_id` is renamed `id`,
- `name` is removed, it was unused by the web client,
- `type` is removed, it was unused by the web client,
- `field_parent` is removed, it was unused by the web client,
- `base_model` is removed, it was unused by the web client.
- `filters` is moved from the global dict returned by `load_views`
(now `get_views`) to the dict returned by `fields_view_get` (now `get_view`)
as it applies only to the `search` view type.
- Retro-compatible methods for the 3 methods
`fields_view_get`, `_fields_view_get` and `load_views` are provided,
with deprecation warnings in them.
- The web client could cache the model fields description
(as it already caches the views),
so it doesn't need to fetch them again if it asks for another view of a model
for which he already has the fields description.
If we do so, `get_views` could return only the list of models used by
the views, without the fields description as of now,
and the web client would then call `fields_get` independently only for
the models for which it doesn't have yet the fields description.
This would avoid the server to return the fields description
and to call `fields_get`, which is costly, for each `get_views`,
therefore gaining performances.
- Inject the views of the one2many and many2many fields all the way down,
unlimited depth level, as explained above.
- Cache with `ormcache` the architecture of back-end views.
This is already done for qweb views, it's not done for back-end views.
Therefore the postprocessing of the views is performed for each `get_views`,
which is costly, while the view architecture doesn't change for users
belonging to the same groups, according to the groups implied by the view.
This pull request is co-authored by
Aaron Bohy (aab) for the web client part and
Denis Ledoux (dle) for the server part.
Part-of: odoo/odoo#87522
This is a followup of #86567, where we change the computation of
display_name to match the values of name_get(). Forcing its value to
False instead of the empty string causes many regressions in frontend
applications like website_blog.
closesodoo/odoo#88438
X-original-commit: d467881f66b11021005005584854f30059971b6c
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Most of the extensions of `_name_get` are very similar and only want to
search for the given string in multiple fields.
A lot of extensions also don't take into account the negative operators.
Some implementations were also really outdated and needlessly
complicated.
closesodoo/odoo#86588
Related: odoo/enterprise#25608
Signed-off-by: Raphael Collet <rco@odoo.com>
Postgres is aligning columns to 4 or 8 bytes, depending on their type.
So, consecutive fixed-length columns of differing size will be padded
with empty bytes due to the alignment requirements.
Before this patch, columns where created in their definition order. Now,
they are ordered based on their size, in order to minimize the padding.
As an example, before each row uses 36 bytes (+24b header):
attname | typname | typlen
-------------+-----------+--------
id | int4 | 4
create_uid | int4 | 4
create_date | timestamp | 8
write_uid | int4 | 4 -> 4 bytes padding
write_date | timestamp | 8
active | bool | 1 -> 3 bytes padding
After each row uses 32 bytes (4 bytes saved per row):
attname | typname | typlen
-------------+-----------+--------
id | int4 | 4
create_uid | int4 | 4
write_uid | int4 | 4
active | bool | 1 -> 3 bytes padding
create_date | timestamp | 8
write_date | timestamp | 8
This saving scheme applies to all rows in all tables. We save between 4
and 8 bytes per row just on the usual create_uid, create_date,
write_uid, write_date.
closesodoo/odoo#87896
Signed-off-by: Raphael Collet <rco@odoo.com>
Because of 4347491670, the method
name_get() can now return records with False as label, and this makes
code crash when it expects a string. Make sure that name_get() always
return strings as labels, while keeping False as empty value for the
field 'display_name'.
closesodoo/odoo#86688
X-original-commit: ff37a2fef89df292b52b1501c28fcf841e1628f5
Signed-off-by: Raphael Collet <rco@odoo.com>
The field prefetching mechanism was poorly customizable. Before this,
we could only tell if a field was prefetched with other fields or not at
all. We have no way to inform the framework, like: "When I need data of
that field, prefetch these other fields, which are likely be used in the
same transaction".
From now on, the `prefetch` attribute is used as a grouping key for
prefetching fields. When a field is fetched, all the fields with the
same value for `prefetch` are taken for prefetching.
For example, consider a small set of fields that are rarely used, except
for one flow A using them. You want to prefetch those fields only in
the flow A, and you want to fetch them in a single query. With the new
feature, simply set `prefetch=A` for some string `A` on those fields,
and they will be grouped for prefetching.
closesodoo/odoo#85220
Signed-off-by: Rémy Voet <ryv@odoo.com>
When we read data of one record, the method `recompute` (`_fetch_field`
-> `_read` -> `flush` -> `recompute`) can take more than 40 % of the
time of the `_read`, due to a huge number of recordset creation (from
`records_to_compute` and `records & recs`). Avoid that waste of time by
postponing the test on records.
For example, on a database with modules crm, mrp, purchase, website,
sale_management, reading the prefetchable fields of the current company
took 1.45 ms ± 60.3 µs, and now takes 1.17 ms ± 110 µs (more than 20%
speedup).
closesodoo/odoo#83818
Related: odoo/enterprise#24645
Signed-off-by: Raphael Collet <rco@odoo.com>
Part-of: odoo/odoo#85220
The public read_group method is used by back-end code as well as RPC calls to receive grouped data.
Most of the time, if the data isn't shown to the user, it doesn't need to be sorted on the names of
the many2one, and can just be sorted on their ID instead, avoiding extra joins in the resulting queries
This is a first commit that will reduce the features of _read_group to the strict minimum necessary
to obtain correct grouped data that is never shown to a user
Task-id: 2479334
Part-of: odoo/odoo#84908
Just before deleting records, we mark fields on dependent records as "to
compute", and their actual computation should happen after the records
have been deleted. The problem is that checking security rules may
trigger the computations that have been prepared. So we should check
security rules before marking dependent records.
closesodoo/odoo#85574
X-original-commit: 79908bc757a239c97bd2b4cda505b7b8df769325
Signed-off-by: Raphael Collet <rco@odoo.com>
* website, website_blog (perf)
If the source values is only composed of empty elements (list of `None`
values), don't try to translate it, as there is nothing to translate
anyway.
While the translation method being called (xml_translate,
html_translate) won't be costly as they will return instantly if the
provided value is falsy, there is still an SQL Query made to get the
callable translation method, which we won't actually use.
This commit avoids that query if we already know we won't do anything
with the translation method.
task-2774979
X-original-commit: e1eb5a6e1610a9550a59ba229a83b9e70e10aa88
Part-of: odoo/odoo#85456
When we read data of one record, the method `recompute` (`_fetch_field`
-> `_read` -> `flush` -> `recompute`) can take more than 40 % of the
time of the `_read`, due to a huge number of recordset creation (from
`records_to_compute` and `records & recs`). Avoid that waste of time by
postponing the test on records.
For example, on a database with modules crm, mrp, purchase, website,
sale_management, reading the prefetchable fields of the current company
took 1.45 ms ± 60.3 µs, and now takes 1.17 ms ± 110 µs (more than 20%
speedup).
closesodoo/odoo#83818
Related: odoo/enterprise#24645
Signed-off-by: Raphael Collet <rco@odoo.com>
The field prefetching mechanism was poorly customizable. Before this,
we could only tell if a field was prefetched with other fields or not at
all. We have no way to inform the framework, like: "When I need data of
that field, prefetch these other fields, which are likely be used in the
same transaction".
From now on, the `prefetch` attribute is used as a grouping key for
prefetching fields. When a field is fetched, all the fields with the
same value for `prefetch` are taken for prefetching.
For example, consider a small set of fields that are rarely used, except
for one flow A using them. You want to prefetch those fields only in
the flow A, and you want to fetch them in a single query. With the new
feature, simply set `prefetch=A` for some string `A` on those fields,
and they will be grouped for prefetching.
Part-of: odoo/odoo#83818
Denis Ledoux