This revision moves the `cache_key` to the first level
dict of the cache, instead of the last one.
Doing so, we reduce the number of times the reference
to the cache key is stored in the dict.
For instance,
for 100.000 records, 20 fields and 2 env (e.g. with and without sudo)
formerly, there were 100.000 * 20 * 2 occurences of cache key references
now, there is only 2 references.
Storing references to an object consumes memory.
Therefore, by reducing the number of object references
in the cache, we reduce the memory consumed by the cache.
Also, we reduce the time to access a value in the cache
as the cache size is smaller.
The time and memory consumption are therefore improved,
while keeping the advantages of revision
d7190a3fd0
which was about sharing the cache of fields
which do not depends on the context, but
only on the cursor and user id.
This revision relies on the fact there are less different references
to the cache key then references to fields/records.
Indeed, this is more likely to have 100.000 different records stored
in the cache rather than 100.000 different environments.
Here is the Python proof of concept that was used
to make the conclusion that setting the cache_key
in the first level dict of the cache is more efficient.
```Python
import os
import psutil
import time
from collections import defaultdict
cr = object()
uid = 1
fields = [object() for i in range(20)]
number_items = 500000
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache_key = (cr, uid)
cache = defaultdict(lambda: defaultdict(dict))
for field in fields:
for i in range(number_items):
cache[field][i][cache_key] = 5.0
# cache[cache_key][field][i] = 5.0
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Using `cache[field][i][cache_key]`:
- Time: 3.17s
- Memory: 3138MB
- Using `cache[cache_key][field][i]`:
- Time: 1.43s
- Memory: 756MB
Even worse, when the cache key tuple is instantiated inside the loop,
for the former cache structure (e.g. `cache[field][i][(cr, uid)]`),
the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB
Here is the same proof of concept, but using the Odoo API and Cache:
```Python
import os
import psutil
import time
from odoo.api import Cache
model = env['res.users']
records = [model.new() for i in range(100000)]
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache = Cache()
char_fields = [field for field in model._fields.values() if field.type == 'char']
for field in char_fields:
for record in records:
cache.set(record, field, 'test')
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop):
- Time: 4.12s
- Memory: 810MB
- After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used):
- Time: 1.63s
- Memory: 125MB
This can be played in an Odoo shell, for instance
by storing it in `/tmp/test.py`, and then
piping it to the Odoo shell:
`cat /tmp/test.py | ./odoo-bin shell -d 12.0`
closesodoo/odoo#29676
instead of updating the cache by records.
We use `cr.fetchall()` and `zip` to get the values by field, which is more
convenient to be stored in cache.
Result of `cr.fetchall()`:
```
[
(3, 'Marc Demo', 'demo@example.com', '032'),
(1, 'Mitchell Admin', 'admin@example.com', '001'),
]
```
After being grouped by field:
```
[
[3, 1],
['Marc Demo', 'Mitchell Admin],
['demo@example.com', 'admin@example.com'],
['032', '001'],
]
```
That way we can update the cache of a field for all records at once, by calling
the builtin `update` method of `dict` with the `zip` of record ids and
corresponding values.
This significantly speeds up the storage of field values in the cache.
closesodoo/odoo#30817
The cache is quite time critical, as it is accessed millions times when reading
multiple fields on hundred of thousands of records.
Avoiding indirections and `if` statement when possible speeds up the cache
access time.
This revision moves the `cache_key` to the first level
dict of the cache, instead of the last one.
Doing so, we reduce the number of times the reference
to the cache key is stored in the dict.
For instance,
for 100.000 records, 20 fields and 2 env (e.g. with and without sudo)
formerly, there were 100.000 * 20 * 2 occurences of cache key references
now, there is only 2 references.
Storing references to an object consumes memory.
Therefore, by reducing the number of object references
in the cache, we reduce the memory consumed by the cache.
Also, we reduce the time to access a value in the cache
as the cache size is smaller.
The time and memory consumption are therefore improved,
while keeping the advantages of revision
d7190a3fd0
which was about sharing the cache of fields
which do not depends on the context, but
only on the cursor and user id.
This revision relies on the fact there are less different references
to the cache key then references to fields/records.
Indeed, this is more likely to have 100.000 different records stored
in the cache rather than 100.000 different environments.
Here is the Python proof of concept that was used
to make the conclusion that setting the cache_key
in the first level dict of the cache is more efficient.
```Python
import os
import psutil
import time
from collections import defaultdict
cr = object()
uid = 1
fields = [object() for i in range(20)]
number_items = 500000
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache_key = (cr, uid)
cache = defaultdict(lambda: defaultdict(dict))
for field in fields:
for i in range(number_items):
cache[field][i][cache_key] = 5.0
# cache[cache_key][field][i] = 5.0
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Using `cache[field][i][cache_key]`:
- Time: 3.17s
- Memory: 3138MB
- Using `cache[cache_key][field][i]`:
- Time: 1.43s
- Memory: 756MB
Even worse, when the cache key tuple is instantiated inside the loop,
for the former cache structure (e.g. `cache[field][i][(cr, uid)]`),
the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB
Here is the same proof of concept, but using the Odoo API and Cache:
```Python
import os
import psutil
import time
from odoo.api import Cache
model = env['res.users']
records = [model.new() for i in range(100000)]
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache = Cache()
char_fields = [field for field in model._fields.values() if field.type == 'char']
for field in char_fields:
for record in records:
cache.set(record, field, 'test')
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop):
- Time: 4.12s
- Memory: 810MB
- After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used):
- Time: 1.63s
- Memory: 125MB
This can be played in an Odoo shell, for instance
by storing it in `/tmp/test.py`, and then
piping it to the Odoo shell:
`cat /tmp/test.py | ./odoo-bin shell -d 12.0`
closesodoo/odoo#29676closesodoo/odoo#30554
Consider a many2one field `foo_id` on model `bar`, with an inverse one2many
field `bar_ids` on model `foo`. During an onchange, the statement
bar.foo_id = foo
puts a special value in cache to add `bar` to the value of `foo.bar_ids`
without explicitly reading `foo.bar_ids`.
Executing the above statement a second time, the cache of `foo.bar_ids` is no
longer empty. This causes the actual value of `foo.bar_ids` to be read and
updated. The issue is that this can be slow for large values of `foo.bar_ids`.
Avoid reading the value of the one2many field by handling the case where the
cache contains the special value: simply update the special value to take into
account the second assignment.
closesodoo/odoo#28982
Followup from the previous commit: before this, _write takes 32% of
total runtime, of which 13.7% is ultimately assignable to add_todo.
Turns out for the test case we mostly keep adding records to recorsets
where they're already present, so the 13.7% of runtime in add_todo are
mostly spent creating orderedsets (11.17) then converting those back
into recordsets (1.75%) with some time spent creating lists and
appending records (already present) in them.
Checking if the records are already present before merging them in
decreases add_todo's runtime cost to 0.5%, and ultimately _write to
20%.
As many of the ignorable calls were merging a singleton or an empty
recorset into the parent recordset, optimising for these cases was
added to BaseModel's <= and >=.
Today, Odoo is really tricky to use without seeing the screen, it must be improved to be usable.
This PR forbid to use labels without a "for" attribute, add some title, rule and aria attributes in HTML. With that, Odoo will be fully usable with a screen reader.
* [IMP] Labels must have a for attribute. Improve accessibility.
* [IMP] Better error message when trying to read a missing cached value
* [FIX] Add some aria-label and title attributes for screen readers.
* [FIX] Template name is not included in the error message in case of SyntaxError in QWeb
* [FIX] Improve the Tour failed at step error message to be more explicit.
* [IMP] Add aria-labels
* [FIX] Add missing aria-label on failing test
* [IMP] aria-hidden means hidden. Fix all bad aria-hidden and hide aria-hidden for all.
* [IMP] Color names on kanban views and many2many tags
* [IMP] Add some checks on views for accessibility.
* [IMP] Add `alt` attribute on `img` tags.
* [IMP] Add aria-label and title on non-described icons
* [IMP] Add button role to widgets with btn class
* [IMP] Translate aria and formatted attributes.
* [IMP] Remove wrong aria-labelledby
* [IMP] Add menu role on dropdowns
* [IMP] Buttons must be focusable
* [IMP] Add aria attributes on progress bars
* [IMP] Improve accessibility of basic widgets
* [IMP] Change main layout to more semantic tags
* [IMP] Add menuitem role when missing
* [IMP] Remove wrong role='presentation'
* [IMP] Improve accessibility of tab panels
* [IMP] Add aria-invalid on invalid fields
* [IMP] Add aria-sort on ordered columns
* [IMP] Add role on alerts
* [IMP] Use dialog role, header, main and footer tags for modals
* [IMP] Add labels on o_status
* [IMP] Improve accessibility of kanban view with feeds and articles
* [IMP] Add alerts in case of new messages
* [IMP] Add widget, navigation or img role to aria-labelled items
Instead of searching for ids in the whole cache, search for the ones we will potentially prefetch.
Example: imagine you have 1K records in cache, and only 3 records in your prefetch set. Instead of retrieving the 1K records that have a value in cache, retrieve which of the 3 records that have no value in cache.
Optimize the copy of the cache being made by related fields to compute
themselves. Instead of traversing records data in cache, follow the cache
structure itself and copy everything!
OPW 816125
Assuming that `field.cache_key(record)` is independent from `record.id`,
compute the cache key once for all records, and avoid using `browse` on all
record ids. This makes `get_records` about 20 times faster.
This has a good performance impact on `field.modified_draft()`, which is used
during onchanges for invalidating fields.
* Fix a bunch of ill-documented/incomplete/incorrect method docs
* add start of Sphinx extension to extract & integrate jsdoc into
Sphinx documentation:
- parse JS files (and don't blow up), uses a fork of pyjsparser as
the project currently does not parse comments
- extract cross-module dependency information
- parse JsDoc comments using pyjsdoc and infer structure from code &
jsdoc
- ``ast`` CLI printing a simplified AST of the input files
- ``dependencies`` creating a dependency graph of either all modules
in the provided input files or the modules matching the specified
filters (warning: will not work if missing dependencies),
generates a .dot file
- ``extractor`` generating a plain text module documentation (mix of
rst and markdown styles, not anything formal)
* sphinx extension with an "automodule" directive taking a module name
and generating the documentation for it
The record cache is indexed by `(field, record_id, key)`, where `key` depends
on the environment. The key is either `(cr, uid, context)` or `(cr, uid)`,
depending on whether the field's value is context-dependent. As most fields
are not context-dependent, this should avoid some cache prefetching when
swiching context.
Make `RecordCache` work on a single record only, and implement a mapping from
field names (only) to values.
Also add explicit methods on `RecordCache` to wrap special behavior in cache,
and simplify all special values as a wrapper for a getter function.
The record cache may contain regular and special values. Modify `RecordCache`
so that only methods ending with `_value` check for regular values. The
dictionary methods considers both regular and special values as equivalent.
name in record._cache # test if cache has a value
iter(record._cache) # iterate on values
record._cache[name] # get value
record._cache.get(name) # get value or default
record._cache.has_value(name) # test if cache has a regular value
record._cache.get_value(name) # get regular value or default
It may be confusing that a constraint is not triggered in a name_create call.
Testing the absence of value should be done in an override of create, not in a
api.constrains.
Closes#18382
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
Raphael Collet