One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.
Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.
The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.
Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.
The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.
We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.
Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.
Note that this poc is inspired from the long term cache but not all
use case where applie yet.
Part-of: odoo/odoo#119813
The `ormcache` decorator fails to create the key method
(`determine_key`) when the method signature contains any annotation.
Fix it by removing annotation of the signature.
closesodoo/odoo#109777
Signed-off-by: Raphael Collet <rco@odoo.com>
The `check` decorator in `sql_db.py` was on a lot of `Cursor` methods.
It checks if the cursor is close before be using a the method.
Remove it because:
- It is completly redundant because `psycopg2` do already the job
to check the cursor before usage.
- It complicated the call stack and lead to a small overhead of
highly use methods (can be more than 1% of the execute call)
- Also the nature of
the Error isn't correct: raise `OperationalError`
(https://www.psycopg.org/docs/module.html#psycopg2.OperationalError)
instead of `InterfaceError`
(https://www.psycopg.org/docs/module.html#psycopg2.InterfaceError).
Part-of: odoo/odoo#80961
Before this, invalidations to the UID cache is not synchronised
between workers because it's an ad-hoc solution (so a user changing
their password or an admin disabling a user would only lock out an
attacker currently using the API of one of possibly several
workers). Shift the entire thing to ormcache which already has proper
support for synchronising cache invalidation between workers.
Also simplify the cache invalidation mess in Users.write because the
caches have been unified into a single registry-level LRU, so the
half-dozen cache clears on specific ormcached methods & models is
pretty much the same as repeatedly calling clear_caches on the current
model.
**However** registry.cache is trivially accessible from server actions
and safe_eval as long as they provide access to a model (through
`model.pool.cache`). Which is common, and an issue given we're very
much putting sensible data in there.
Fix this by renaming `Registry.cache` to `Registry.__cache`, this
requires few editions and mangled names are not accessible from
safe_eval contexts.
The alternative would have been to add more bespoke handling of the
uid cache to hook it into the cache invalidation propagation
machinery.
After discussion with (@)odony, fixing LRU access and using that seems
cleaner and less error-prone.
Note on lazy_property
=====================
Make Registry.cache / Registry.__cache into a regular attribute: the
overhead of the LRU is not that high (compared to that of the registry
itself), it's rare that we *don't* need it, and it's assumed to be a
persisted attribute (it's not just a cache) so making it a normal
attribute seems fine; and lazy_property doesn't work for mangled
names: the name of the property is mangled using the name of the
definition class, but the name of the symbol (fget) is not mangled so
lazy_property would set the __cache attribute but then Python would
lookup _Registry__cache, creating a new cache every access.
And we can't (always) mangle things correctly on `__get__(obj,
owner)`: `owner` is just `type(obj)`, meaning in the case of
inheritance the type we get is the type through which the property is
accessed rather than the one it's defined on. So it would work in the
cases where no inheritance is involved (such as Registry.__cache) but
not in general (lest we want to play around walking the MRO ourselves
to find the definition source, which doesn't seem worth it).
lazy_property *could* be made to work properly on Python 3.6+: the
descriptor protocol gains `__set_name__(name, owner)`, which is called
with the properly mangled name — and with the definition class to boot
(though there might still be issues when overriding lazy properties as
the override will be mangled & named differently... or maybe that's a
feature?). However we're still supporting 3.5 at this point, AFAIK, so
that's not an option. Plus it feels unnecessary / not very useful.
However add an assertion to `lazy_property` so it signals when we try
to use it on a mangled method (as otherwise it kinda sorta work in the
sense that the property / object is accessible but is in effect a
slower way to write a regular property).
Since commit 1b30a78b, LRU is now based on OrderedDict and keys can be found
on dictionnary 'd' directly.
closesodoo/odoo#55541
X-original-commit: 3f093da75765d66ed1e2d828e8b41ed05d936d0b
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Iteration methods on LRU were removed because they were not
thread-safe and it's not clear that making them thread-safe is the
correct thing to do, so not providing them seems saner.
I thought I'd looked for usages of the LRU but apparently didn't look
hard enough as I missed that it's used by the cron workers (apparently
using the threaded server we only run crons for dbs currently living
in the registry cache, the more you know).
Convert these to iterating on the LRU's internal mapping, and also
don't iterate on the LRU to clear its entries one by one when we can
just clear the entire thing safely, although Registry.delete_all
really seems completely unused.
closesodoo/odoo#49023
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Those two decorators are removed/deprecated since recent commits but
some references remained in the documentation.
api.guess and api.noguess is deprecated and removed since c552fb7a61closesodoo/odoo#35332
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Following the revamp / contextualisation of multi company, the list of
allowed companies from the context is now used as part of the ir_rule
caching key.
Sadly that list is a list, and thus not hashable, meaning
_compute_domain was not cached anymore. Sadlier, type errors would be
silently ignored (just increasing the error counter of the cache stats).
Fix both issues:
* Store lists from the context as tuple. Technically could be a
frozenset but that might not always be the case so it doesn't feel
future-proof, a tuple might lead to slightly less hits but seems safer
* Log a proper warning in case of an ormcache TypeError (non-hashable
type used as cache key). It should never routinely happen.
closesodoo/odoo#34330
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
The adaptation to Python 3 was done with two mistakes: the iteration on the lru
cache does not return keys, and functions are not sortable.
closesodoo/odoo#28870
Returning a recordset from a cached method will raise an
`psycopg2.OperationalError` once the cursor is closed.
Add documentation to `ormcache` method indicating such
See OCA/stock-logistics-barcode#93 and #8795 for example of issues
Closes#19113
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
* remove references to basestring & unicode (use relevant pycompat
helpers)
* remove some str calls (either entirely or replaced by relevant
helper, either text or native)
* use better API to avoid unnecessary conversions
* remove some XML declarations in views
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
Problem: the update of custom models/fields is not fully transactional, and may
potentially lead to an inconsistent database. An other problem is creating two
custom fields by writing on a model: if the second one fails, the first one has
been committed without notice. Retrying the request will give an unexpected
error (duplicate field name).
Solution: never commit in the middle of a request. If the changes have an
impact on the registry, then mark it as invalid (with a new flag), and signal
registry invalidation after everything has been committed. If the request
fails, reset the registry. Both registry and cache invalidation are handled
the same way.
Introspection attributes on function and methods were originally
prefixed with func_ or im_ e.g. im_class or func_name. For coherence with the
rest of the data model, Python 3 added dunder attributes (__func__,
__code__) and removed the old style, the dunder attributes were
backported to Python 2.6.
Use dunder attributes everywhere we're currently using func_* or im_*
attributes.
Fixers:
lib2to3.fixes.fix_funcattrs
lib2to3.fixes.fix_methodattrs
#8530
Chong Wang (cwg)