Translated fields no longer use the model ir.translation. Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table. The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages). Empty text is allowed in translation values, but not NULL.
Here are examples for a field with translate=True:
NULL
{"en_US": "Foo"}
{"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
{"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}
Like before, writing False to the field makes it NULL, i.e., False in all
languages. However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.
Here are examples for a field with translate=xml_translate:
NULL
{"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}
Change for callable(translate) fields: one can now write any value in any
language on such a field. The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values. Basically the
structure of the value must remain the same in all languages, like before.
Reading a translated field is now both simpler and faster than the former
implementation. We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:
SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...
The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages). For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.
Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language. Before this change, only the source value of the
field ('en_US') could be indexed.
Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages. We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.
Code translations are no longer stored into the database. They become static,
and are extracted from the PO files when needed. The worker simply uses a cache
with extracted code translations for performance. This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker. Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).
Performance summary:
(+) reading 'model' translated fields is faster
(+) reading 'model_terms' translated fields is much faster (no need to inject
translations into the source value)
(+) searching translated fields with operator 'ilike' is much faster when the
field is indexed with 'trigram'
(+) updating translated fields requires less ORM flushing
(-) importing translations from PO files is 2x slower
Some extra fixes:
- make field 'name' of ir.actions.actions translated; because of the PG
inheritance, this is necessary to make the column definition consistent in
all models that inherit from ir.actions.actions.
- add some backend API for the web/website client for editing translations
- move methods get_field_string() to model ir.model.fields
- move _load_module_terms to model ir.module.module
- adapt tests in test_impex, test_new_api
- because env.lang is injected into SQL queries, its returned value is
now guaranteed to correspond to a valid active language or None
- remove wizard to insert missing translations (no longer makes sense)
task-id: 2081307
Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
We added trigram index for char fields since https://github.com/odoo/odoo/pull/83015.
But if `unaccent` is installed in the database (and isn't force to
`False` on field), these new trigram indexes are pointless and cost a
lot for nothing (almost nothing, it still can be used for equality
operator but in this case a btree will be far more efficient).
The simple way to fix it is to add `unaccent(<column>)` in the index
trigram definition, but unfortunately `unaccent` is not immutable and
may therefore not be indexed. In order to make `unaccent` indexable, we
must declare it as immutable (see
https://stackoverflow.com/questions/11005036/does-postgresql-support-accent-insensitive-collations/11007216#11007216
for more information and how to do that).
With this patch, trigram indexes are created with `unaccent(<column>)`
if the function `unaccent` is available in the database, and for the
fields that are not declared with `unaccent=False`. Moreover, we issue
a warning when `unaccent` is available but is not immutable, in which
case most trigram indexes will be useless.
odoo/upgrade#3736
task-2551518
closesodoo/odoo#95943
Signed-off-by: Rémy Voet <ryv@odoo.com>
Those were not accounted for, leading to fstrings passing through
unflagged.
Also update the SQL checker to be stricter but smarter:
The previous version would "fail open", unknown nodes would be allowed
through hence f-strings not being flagged when they started appearing
in arg0 position, should now fail-closed, anything that's not allowed
is forbidden.
This flags a few more cases, all of which seem acceptable upon review.
However the previous version would also only resolve arg0 (in case it
had a `NAME`, to see if that resolved to an acceptable form of
query-building). The new version performs resolution during
`_check_concatenation` and should thus allow e.g. format strings to be
separate variables (though not e.g. module-level constants, yet
anyway).
In resolution, replace the ad-hoc process by astroid's built-in
`lookup` which seems to provide the same information. Slightly more in
fact, as it yields every assignment in case of e.g. conditionals, but
making use of that would require a lot more changes in the checker so
leaving the behaviour as-is for now.
It's important to *not* use `ilookup` here, because ilookup is not
"iterable" but "inferring", and we don't want values, we want
expression ASTs for analysis.
NOTE: previous improvements as well as fixes to existing code were
only implemented in 14.0, hence this being merged in 14.0 not 13.0
despite 13.0 still being supported.
closesodoo/odoo#81721
X-original-commit: 376ccf0944dae1bc53ae9c5385977c4e6b23e083
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
The http.addons_manifest is a map {module: manifest_dict} that is
populated upon the first http request. This map is basically a module
manifest cache with an extra `addons_path` key, the path of the module
on the file-system. This cache is eagerly populated upon the first http
request, the map is empty in non-http contextes (e.g. cron) which have
been a source of bugs (e.g. 50c8eb1).
A manifest cache is necessary because reading and parsing python files
from the file-system is not that cheap but there is no reason that cache
is located in `odoo.http`. A thin cache layer now wraps
`load_information_from_description_file()`/`load_manifest()` and is
lazily populated.
The `http.addons_manifest` have been removed. The extra `addons_path`
key is now present in the "normal" manifest. The `read_manifest()` was
hardly used so it has been deprecated. The only way to retrieve a
manifest is now `load_information_from_description_file()` which was
renamed `load_manifest()` (no cache) and `get_manifest()` (cache).
Side note about performances, the cache is necessary. Addons manifest
are read-only and reading + parsing python files from the file system is
not a cheap operation. Running the e-commerce tour
`@website_sale.test_04_admin_website_sale_tour` without cache on
`load_manifest()` requires 68,29 secs to complete on my laptop,
exceeding the default 1-minute time frame allowed in tests. Using a
cache the time is down to 36,53 secs. The performance impact is huge.
Part-of: odoo/odoo#79977
Before this commit searched terms were the ones entered by the end-user.
After this commit if the searched term is not found then the search is
performed on a resembling term.
The similarity is obtained from a Levenshtein distance combined with
ratios of common and different letters.
By default the dictionary against which the search term is matched is
based on the content of the search fields containing a word that starts
with the first letter of the search term.
If the `pg_trgm` Postgresql extension is installed the dictionary is
built from matches using the `<%%` operator (similar to the
`word_similarity` function).
Several approaches were benchmarked during development, those results
are available through the task record.
task-2379555
https://github.com/odoo/odoo/pull/65871
Part-of: odoo/odoo#65871
An unintended side-effect of #29431 is apparently that auto_install
applications automatically get reinstalled when uninstalled, which was
not the goal.
Move the auto_install selection back into db.py, this version seems to
work even though the previous attempt was apparently unsuccessful.
closesodoo/odoo#35180
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Context:
After a module update, the records present in ir_model_data table but not in
self.pool.loaded_xmlids are considered as no longer needed and should be
removed.
An exception is done with noupdate=true entries.
Bug 1:
Entries with noupdate=NULL are not considered in the evaluation and are ignored
while it may be worth deleting.
Bug 2:
Some records with their external id being automatically created are not present
in self.pool.loaded_xmlids and may get removed after updating a module.
By a lucky coincidence, bug 1 make it so that records targeted by bug 2 are
ignored and not deleted (the "automagically" created ir.model.data often lack
a noupdate value).
Fix Bug 1:
Use a COALESCE to find both records with noupdate=NULL and noupdate=false
Fix Bug 2:
Depends on the source of the generated external id:
- ir.model:
The entries created through _reflect_model were not loaded in
self.loaded_xmlids
Use the proper ORM method _update_xmlids that correctly populates
self.pool.loaded_xmlids
- ir.model.category:
The categories were generated when the db was initalised, doing SQL was not
avoidable.
Create the categories in noupdate to avoid it being considered for removal.
- ir.property:
Are always created in noupdate in data files but in stock_account it was
manually created without being in noupdate
closesodoo/odoo#32881
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Currently, auto_install is triggered when all dependencies get
installed, but there are cases where one would want such trigger on
only a subset thereof.
e.g. we want `website_sale_dashboard` to auto-install when
`website_sale` is installed. Currently, it requires `web_dashboard` to
also be auto-installed otherwise `website_sale_dashboard` would "wait"
for both dependencies to be explicitly installed before the
auto-install triggers. That's despite `web_dashboard` not being very
useful on its own. More generally this is an issue with technical
modules which need to be marked as auto_install so as not to block
e.g. bridge modules from automatically installing.
This change allows setting `auto_install` to a subset of `depends`:
* if auto_install is set to `False`, the module does not get
automatically installed (no change in semantics)
* if auto_install is set to `True`, the module gets automatically
installed if and only if all its dependencies are installed (also no
change in semantics)
* if auto_install is set to a list of dependencies, the module will be
installed when all *these* dependencies are installed, other
dependencies (excluded from auto_install) will be installed
alongside as a consequence
* auto_install can be set to an empty list, in this case the module
will always be automatically installed regardless of its
dependencies (and will force their installation).
So after this change, `web_dashboard`'s auto_install can be set to
`False` (such that it's not installed if no module defining dashboards
is installed) and `website_sale_dashboard`'s manifest can be edited
to:
'auto_install': ['website_sale']
possibilities:
# no automatic installation
'depends': ['a', 'b'],
'auto_install': False
# automatic installation if both a and b are installed
'depends': ['a', 'b'],
'auto_install': True
# automatic installation if both a and b are installed (explicit)
'depends': ['a', 'b'],
'auto_install': ['a', 'b']
# automatic installation if b is installed, a will get forcefully
# installed if it isn't yet
'depends': ['a', 'b'],
'auto_install': ['b']
# always automatically installed, will cause the installation of
# its dependencies even if they're not marked explicitly
'depends': ['a', 'b'],
'auto_install': []
Task 1851328
closesodoo/odoo#29431
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Try to remove cr.commit (and rollback) from module and db install:
* put a savepoint around test data loading
* remove a bunch of commits sprinkled throughout
* remove rollback on data loading failure (assuming it bubbles up, the
entire module's installation should be rolled back)
* add commit right before the tests are run, so they can run isolated
and still see whatever was done when installing their module
* convert a few explicit closing to context managers
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
