During the installation of a module, an IntegrityError can be raised,
e.g. if some module data already exists in the database
(as in the case of a module which uninstallation did not remove its data).
In Python2 it was possible to use such a psycopg2.IntegrityError with i[0]
to get its error message, while this raises an exception in Python3,
i.args[0] is needed.
As a result the log would contain an exception within an exception instead of
displaying an error to the user.
opw 1916374
closesodoo/odoo#29787
This revision moves the `cache_key` to the first level
dict of the cache, instead of the last one.
Doing so, we reduce the number of times the reference
to the cache key is stored in the dict.
For instance,
for 100.000 records, 20 fields and 2 env (e.g. with and without sudo)
formerly, there were 100.000 * 20 * 2 occurences of cache key references
now, there is only 2 references.
Storing references to an object consumes memory.
Therefore, by reducing the number of object references
in the cache, we reduce the memory consumed by the cache.
Also, we reduce the time to access a value in the cache
as the cache size is smaller.
The time and memory consumption are therefore improved,
while keeping the advantages of revision
d7190a3fd0
which was about sharing the cache of fields
which do not depends on the context, but
only on the cursor and user id.
This revision relies on the fact there are less different references
to the cache key then references to fields/records.
Indeed, this is more likely to have 100.000 different records stored
in the cache rather than 100.000 different environments.
Here is the Python proof of concept that was used
to make the conclusion that setting the cache_key
in the first level dict of the cache is more efficient.
```Python
import os
import psutil
import time
from collections import defaultdict
cr = object()
uid = 1
fields = [object() for i in range(20)]
number_items = 500000
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache_key = (cr, uid)
cache = defaultdict(lambda: defaultdict(dict))
for field in fields:
for i in range(number_items):
cache[field][i][cache_key] = 5.0
# cache[cache_key][field][i] = 5.0
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Using `cache[field][i][cache_key]`:
- Time: 3.17s
- Memory: 3138MB
- Using `cache[cache_key][field][i]`:
- Time: 1.43s
- Memory: 756MB
Even worse, when the cache key tuple is instantiated inside the loop,
for the former cache structure (e.g. `cache[field][i][(cr, uid)]`),
the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB
Here is the same proof of concept, but using the Odoo API and Cache:
```Python
import os
import psutil
import time
from odoo.api import Cache
model = env['res.users']
records = [model.new() for i in range(100000)]
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache = Cache()
char_fields = [field for field in model._fields.values() if field.type == 'char']
for field in char_fields:
for record in records:
cache.set(record, field, 'test')
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop):
- Time: 4.12s
- Memory: 810MB
- After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used):
- Time: 1.63s
- Memory: 125MB
This can be played in an Odoo shell, for instance
by storing it in `/tmp/test.py`, and then
piping it to the Odoo shell:
`cat /tmp/test.py | ./odoo-bin shell -d 12.0`
closesodoo/odoo#29676
Suppose calling get_fiscal_year(date, day=29, month=2) with date(year=2017, month=2, day=28)
The computation steps are:
date_from = date(year=2017, month=2, day=28) - 1 year = date(year=2016, month=2, day=28)
date_from = date_from + 1 day = date(year=2016, month=2, day=29)
It's incorrect because the value of date_from must be date(year=2016, month=3, day=1)
closesodoo/odoo#29416
instead of updating the cache by records.
We use `cr.fetchall()` and `zip` to get the values by field, which is more
convenient to be stored in cache.
Result of `cr.fetchall()`:
```
[
(3, 'Marc Demo', 'demo@example.com', '032'),
(1, 'Mitchell Admin', 'admin@example.com', '001'),
]
```
After being grouped by field:
```
[
[3, 1],
['Marc Demo', 'Mitchell Admin],
['demo@example.com', 'admin@example.com'],
['032', '001'],
]
```
That way we can update the cache of a field for all records at once, by calling
the builtin `update` method of `dict` with the `zip` of record ids and
corresponding values.
This significantly speeds up the storage of field values in the cache.
closesodoo/odoo#30817
Instead of iterating on self and then putting the values in the right place in
the `data` dict, iterate over data directly.
This removes the need of the indexing `data[record]`, which takes a significant
time when dealing with hundred of thousands of records.
Calling `len` on a long list and checking that its value is 1 is slower than
just unpacking the list. The point is to check whether there is just one item
in the list, and counting all items just for that is a bit overkill.
Use `env.registry` instead of `env` to get the comodel as a class instead of an
instance, which saves one call to `_browse` each time we access a relational
field. This saves time because of the cost of `object.__new__` and prefetch
update in `_browse`.
The cache is quite time critical, as it is accessed millions times when reading
multiple fields on hundred of thousands of records.
Avoiding indirections and `if` statement when possible speeds up the cache
access time.
If module A define an sql constraints and module B, which depends on
module A, override the sql constraint, we must ensure having only the
last definition in constraints list. Having the two versions in the list
makes the ORM continuously trying to drop the second version by the
first one, then recreate the second version, when performing operations
like an upgrade a module B.
An illustration of this is the login_key on res.users, which is unique
per login in base, but unique per (login, website) in website.
closesodoo/odoo#30334
Following commit 983c7e8152.
`_compute_display_name` is overridden in partner to force `show_adress` and some
other context variables to None. When those values are in the context, calling
`display_name` can give a different result than calling `name_get()[0][1]`.
This commit reverts the changes of the mentioned commit where the behavior is
actually different, that is, when `show_address` is in the context.
It also adds a comment on the `_compute_display_name` method to avoid further
mistakes.
closesodoo/odoo#29510
In `test_enable` mode, `render_qweb_pdf` fallbacked to `render_qweb_html`. We add a context key to be able to force the rendering of the pdf, even in `test-enable` mode.
The definition of `attachment_ids` on model `email_template.preview` is wrong,
because its table/columns refer to the model `mail.template`. As the field is
only used to preview a result in a wizard form, it does not need to be stored.
closesodoo/odoo#29349
*: tools, web, website, website_forum, mail, im_livechat
This commit refactors ir_http to make it more readable
and flexible.
Move the resize function of web/image to odoo.tools
Co-authored-by: XavierDo <xdo@odoo.com>
Co-authored-by: Antony Lesuisse <al@openerp.com>
closes: #28563
task: #1908896
Moved the field 'active', 'res_model_name' and 'thumbnail' from
ir.attachment to enterprise's documents.document.
Moved the pdf split feature to enterprise's Documents module.
task: #1908896
This reverts commit 5f4a945182.
The cure is worse than the original disease.
The main installation cursor often holds exclusive locks (due to DDL
changes) on vital tables such as res_users. As a result, using
another cursor to perform changes while the main cursor is waiting
is extremely deadlock-prone. And these deadlocks can't be detected by
PostgreSQL as they mix Python-SQL locking, which leads to deadlocked
HTTP workers.
Related to:
- opw-1916918
- #29528
Steps to reproduce the bug:
- Write a model in lunch.py called "my.model"
- Install module Lunch
- Erase "my.model" in lunch.py
- Upgrade the module Lunch
A error was raised when processing function "_process_end" because
the ir.model.data record of "my.model" was not deleted
opw:1908813
closesodoo/odoo#29247
Nans Lefebvre