This revision moves the `cache_key` to the first level
dict of the cache, instead of the last one.
Doing so, we reduce the number of times the reference
to the cache key is stored in the dict.
For instance,
for 100.000 records, 20 fields and 2 env (e.g. with and without sudo)
formerly, there were 100.000 * 20 * 2 occurences of cache key references
now, there is only 2 references.
Storing references to an object consumes memory.
Therefore, by reducing the number of object references
in the cache, we reduce the memory consumed by the cache.
Also, we reduce the time to access a value in the cache
as the cache size is smaller.
The time and memory consumption are therefore improved,
while keeping the advantages of revision
d7190a3fd0
which was about sharing the cache of fields
which do not depends on the context, but
only on the cursor and user id.
This revision relies on the fact there are less different references
to the cache key then references to fields/records.
Indeed, this is more likely to have 100.000 different records stored
in the cache rather than 100.000 different environments.
Here is the Python proof of concept that was used
to make the conclusion that setting the cache_key
in the first level dict of the cache is more efficient.
```Python
import os
import psutil
import time
from collections import defaultdict
cr = object()
uid = 1
fields = [object() for i in range(20)]
number_items = 500000
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache_key = (cr, uid)
cache = defaultdict(lambda: defaultdict(dict))
for field in fields:
for i in range(number_items):
cache[field][i][cache_key] = 5.0
# cache[cache_key][field][i] = 5.0
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Using `cache[field][i][cache_key]`:
- Time: 3.17s
- Memory: 3138MB
- Using `cache[cache_key][field][i]`:
- Time: 1.43s
- Memory: 756MB
Even worse, when the cache key tuple is instantiated inside the loop,
for the former cache structure (e.g. `cache[field][i][(cr, uid)]`),
the time goes from 3.17s to 25.63s and the memory from 3138MB to 3773MB
Here is the same proof of concept, but using the Odoo API and Cache:
```Python
import os
import psutil
import time
from odoo.api import Cache
model = env['res.users']
records = [model.new() for i in range(100000)]
p = psutil.Process(os.getpid())
m = p.memory_info().rss
s = time.time()
cache = Cache()
char_fields = [field for field in model._fields.values() if field.type == 'char']
for field in char_fields:
for record in records:
cache.set(record, field, 'test')
print('Memory: %s' % (p.memory_info().rss - m,))
print('Time: %s' % (time.time() - s,))
```
- Before (`cache[field][record_id][cache_key]` and cache_key tuple instantiated in the loop):
- Time: 4.12s
- Memory: 810MB
- After (`cache[cache_key][field][record_id]` and cache_key tuple stored in the env and re-used):
- Time: 1.63s
- Memory: 125MB
This can be played in an Odoo shell, for instance
by storing it in `/tmp/test.py`, and then
piping it to the Odoo shell:
`cat /tmp/test.py | ./odoo-bin shell -d 12.0`
closesodoo/odoo#29676closesodoo/odoo#30554
Consider a many2one field `foo_id` on model `bar`, with an inverse one2many
field `bar_ids` on model `foo`. During an onchange, the statement
bar.foo_id = foo
puts a special value in cache to add `bar` to the value of `foo.bar_ids`
without explicitly reading `foo.bar_ids`.
Executing the above statement a second time, the cache of `foo.bar_ids` is no
longer empty. This causes the actual value of `foo.bar_ids` to be read and
updated. The issue is that this can be slow for large values of `foo.bar_ids`.
Avoid reading the value of the one2many field by handling the case where the
cache contains the special value: simply update the special value to take into
account the second assignment.
closesodoo/odoo#28982
Instead of searching for ids in the whole cache, search for the ones we will potentially prefetch.
Example: imagine you have 1K records in cache, and only 3 records in your prefetch set. Instead of retrieving the 1K records that have a value in cache, retrieve which of the 3 records that have no value in cache.
Optimize the copy of the cache being made by related fields to compute
themselves. Instead of traversing records data in cache, follow the cache
structure itself and copy everything!
OPW 816125
Assuming that `field.cache_key(record)` is independent from `record.id`,
compute the cache key once for all records, and avoid using `browse` on all
record ids. This makes `get_records` about 20 times faster.
This has a good performance impact on `field.modified_draft()`, which is used
during onchanges for invalidating fields.
* Fix a bunch of ill-documented/incomplete/incorrect method docs
* add start of Sphinx extension to extract & integrate jsdoc into
Sphinx documentation:
- parse JS files (and don't blow up), uses a fork of pyjsparser as
the project currently does not parse comments
- extract cross-module dependency information
- parse JsDoc comments using pyjsdoc and infer structure from code &
jsdoc
- ``ast`` CLI printing a simplified AST of the input files
- ``dependencies`` creating a dependency graph of either all modules
in the provided input files or the modules matching the specified
filters (warning: will not work if missing dependencies),
generates a .dot file
- ``extractor`` generating a plain text module documentation (mix of
rst and markdown styles, not anything formal)
* sphinx extension with an "automodule" directive taking a module name
and generating the documentation for it
The record cache is indexed by `(field, record_id, key)`, where `key` depends
on the environment. The key is either `(cr, uid, context)` or `(cr, uid)`,
depending on whether the field's value is context-dependent. As most fields
are not context-dependent, this should avoid some cache prefetching when
swiching context.
Make `RecordCache` work on a single record only, and implement a mapping from
field names (only) to values.
Also add explicit methods on `RecordCache` to wrap special behavior in cache,
and simplify all special values as a wrapper for a getter function.
The record cache may contain regular and special values. Modify `RecordCache`
so that only methods ending with `_value` check for regular values. The
dictionary methods considers both regular and special values as equivalent.
name in record._cache # test if cache has a value
iter(record._cache) # iterate on values
record._cache[name] # get value
record._cache.get(name) # get value or default
record._cache.has_value(name) # test if cache has a regular value
record._cache.get_value(name) # get regular value or default
It may be confusing that a constraint is not triggered in a name_create call.
Testing the absence of value should be done in an override of create, not in a
api.constrains.
Closes#18382
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
Denis Ledoux