Commit Graph
899 Commits
Author SHA1 Message Date
Benoit Socias 0bc3ae0ed5 [FIX] base: ignore spaces around text content when matching translation
When the same text appears several times inside a translated field but
nested in different HTMLs, the matching for each one is done
independently if various spacing appear in the HTML.

This commit strips the spaces around the matched texts so that texts
that are synchronized on purpose do not become desynchronized.
Doing this leads to collisions on the keys of `text2term`, it therefore
also has to replace it with a dictionary of text to list of terms.

opw-3098819

closes odoo/odoo#109798

X-original-commit: 6cec590a2dad43063d3bb747838393a8df13aa04
Signed-off-by: Benoit Socias (bso) <bso@odoo.com>
2023-01-13 17:32:35 +01:00
Jordan D. (Joda) a57b214c7d [FIX] tools: add SQL comparaison to fix module translation
When installing both documents and fleet and then, add a translation to
the database. Both fleet and document will be translated the same way.
This is due to the fact that both of them share a lot of similar value
inside of the PSQL database.

opw-3120320

closes odoo/odoo#109700

X-original-commit: fff53f845398ee0f8632cf7cb392f108388f3dc4
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
2023-01-12 12:27:10 +01:00
Julien Castiaux c59750d824 [IMP] core: smarter geoip
Maxmind offers multiple ip-geolocalization databases, historically we
have been using the City database which contains records on a
city-basis. Many years later it turns out we are primary using geoip to
know the country of the user. Geolocalization in the City database is
considered slow by our standard and we have been clever in order not to
geolocate each request by saving the info in the session.

On the other hand, the Country database that is offered by Maxmind is
much more lightweight and geoip using that country is considered a fast
operation by our standard.

In this work we make Odoo compatible with both the City and the Country
databases. Using multiple database at the same time, we can be smart and
only query each of the two on-demand. If a user ask for its country,
we'll use the fast Country db. If a user ask for its city/timezone we'll
use the slower City db.

By default it loads both database from the `/usr/share/GeoIP/` folder,
respectively the files `GeoLite2-City.mmdb` and `GeoLite2-Country.mmdb`,
you can provide alternative paths using the `--geoip-city-db` and
`--geoip-country-db` CLI options.

In the same mindset as #86015, geoip is still lazy. It is done on-demand
and the result is cached on the current request. The different with the
related PR is that as we know consider geoip to be fast, we no longer
cache the result in the session.

Task: 2848206
Part-of: odoo/odoo#91337
2023-01-03 13:16:02 +01:00
Nshimiyimana Séna 3aeed55764 [FIX] tools: recognize SVG with explicit namespace
Currently, odoo's mimetype guesser does not recognize SVGs with explicit
namespace.

closes odoo/odoo#108938

X-original-commit: a1257cad9d5a63d940da041f355c13ea3eba90ce
Signed-off-by: Julien Castiaux <juc@odoo.com>
2023-01-02 20:39:49 +01:00
Nshimiyimana Séna 68f1529490 [FIX] tools: set correct mimetype for XML files
Currently, the mimetype detection for XML files always returns
`image/svg+xml`.

opw-3062131

closes odoo/odoo#108644

X-original-commit: 6ea9e75d5a14cfe4a846be69e05c3c18f1daffcb
Signed-off-by: Julien Castiaux <juc@odoo.com>
2022-12-24 02:35:46 +01:00
Alexandre Kühn 7139ae416e [IMP] tools: put QUnit.only test in module wrap
Before this commit, when filtering tests by odoo module,
tests registered with `QUnit.test` and `QUnit.debug` were
listed, but not `QUnit.only`.

closes odoo/odoo#108591

Signed-off-by: Géry Debongnie <ged@odoo.com>
2022-12-23 15:08:18 +01:00
Olivier Dony 9bd642de7f [FIX] tools/transpiler: avoid nested quantifiers
It made the regex much slower in case of non-match, due to backtracking.

closes odoo/odoo#108046

X-original-commit: 56b1b17d6fdf493368ce219f981f94d9efdc9b7f
Signed-off-by: Simon Genin (ges@odoo) <ges@odoo.com>
2022-12-21 11:26:32 +01:00
Loan (lse) d08d812a83 [IMP] tools: show query count and timings in thread dump
Before this commit:
 Thread details gave information regarding the database
 the UID and the URL.

After this commit:
 It also informs on the "performance times":
  - `qc` = query count
  - `qt` = query time
  - `pt` = python time. It should in theory be "remaining time", but it won't be use as its acronym `rt` is too ambiguous/confusing

closes odoo/odoo#106610

Signed-off-by: Olivier Dony (odo) <odo@odoo.com>
2022-12-19 20:11:13 +01:00
FrancoisGe 8e3de77042 [IMP] tools: adds the dependencies in the odoo.define
After this commit, the transpiler will add the dependencies of a file
to the second argument of the odoo.define(...).

For example:
odoo.define("@test/test", ['@test/dep1', '@test/dep2'], async function (require)

With this change, the code for the module system in the browser will get
directly the dependencies instead of having to stringify the function
defining a module, and grep inside for the "require" words.

closes odoo/odoo#105651

Taskid: 3062390
Related: odoo/enterprise#34895
Signed-off-by: Géry Debongnie <ged@odoo.com>
2022-12-15 16:47:05 +01:00
Nicolas Martinelli aa8de3d4d2 [IMP] config: increase limit request
A typical Odoo worker is able to handle ~20 requests per second. With a
default limit to 8192, it means that the worker is recycled after
8192 / 20 = 409s ~ 7 minutes.

There is no reason to recycle a worker that often, since there are
other means of limiting the resources allocated to a worker such as the
memory limit.

We increase the default limit request to 65536, therefore the lifetime
of a worker should be extended to ~1 hour in peak times.

closes odoo/odoo#107970

X-original-commit: a24d32003af334d63f04d6892dbc2e4383a65301
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
2022-12-15 14:43:44 +01:00
Ayman Adil 6a3444e5bd [FIX] convert.py: skip the creation of a forcecreate=0 record when a reference is missing
When a record is `forcecreate=0` and a reference of one of its fields is missing, the update of the corresponding module fails. In this case, we can skip the creation of this record.

An example of this issue is this [record](https://github.com/odoo/enterprise/blob/6411ae071ace980834befeb8040c94b1a05c8034/documents_account/data/data.xml#L17) in `documents_account` module (enterprise) when the reference of this [field](https://github.com/odoo/enterprise/blob/6411ae071ace980834befeb8040c94b1a05c8034/documents_account/data/data.xml#L20) is missing.

This can be reproduced as follows:
1. Create a DB with `documents_account` module
2. Uninstall `documents_account` module
3. Remove `documents.documents_finance_status`
4. Reinstall `documents_account`

```
Traceback (most recent call last):
  File "/home/ayman/src/odoo/15.0/odoo/tools/convert.py", line 680, in _tag_root
    f(rec)
  File "/home/ayman/src/odoo/15.0/odoo/tools/convert.py", line 567, in _tag_record
    f_val = self.id_get(f_ref)
  File "/home/ayman/src/odoo/15.0/odoo/tools/convert.py", line 663, in id_get
    res = self.model_id_get(id_str, raise_if_not_found)
  File "/home/ayman/src/odoo/15.0/odoo/tools/convert.py", line 669, in model_id_get
    return self.env['ir.model.data']._xmlid_to_res_model_res_id(id_str, raise_if_not_found=raise_if_not_found)
  File "/home/ayman/src/odoo/15.0/odoo/addons/base/models/ir_model.py", line 1943, in _xmlid_to_res_model_res_id
    return self._xmlid_lookup(xmlid)[1:3]
  File "<decorator-gen-35>", line 2, in _xmlid_lookup
  File "/home/ayman/src/odoo/15.0/odoo/tools/cache.py", line 90, in lookup
    value = d[key] = self.method(*args, **kwargs)
  File "/home/ayman/src/odoo/15.0/odoo/addons/base/models/ir_model.py", line 1936, in _xmlid_lookup
    raise ValueError('External ID not found in the system: %s' % xmlid)
ValueError: External ID not found in the system: documents.documents_finance_status
```

closes odoo/odoo#107766

X-original-commit: 2e4f0667397d2d6670df50b9088fa9c23f1f9f5a
Signed-off-by: Christophe Simonis <chs@odoo.com>
2022-12-12 18:16:15 +01:00
Denis Ledoux 15948702e5 [IMP] tools: locate_node, remove useless code
`if isinstance(node, SKIPPED_ELEMENT_TYPES):`
---------------------------------------------

It has been added in
https://github.com/odoo/odoo/commit/31b9bee00676e179c124f3d42539103305cf7423

It's hard to tell why it has been added. Though it seems not useful:
- Has we iter the nodes using the tag of the specification,
  `arch.iter(spec.tag)`,
  this is normally impossible to iter on a node type from that
  SKIPPED_ELEMENT_TYPES node type list
  (`etree._Comment, etree._ProcessingInstruction, etree.CommentBase, etree.PIBase, etree._Entity`),
  as those node types do not have a tag (`spec.tag`).
  In addition, in the place calling `locate_node`,
  we already do this same check on the specification node:
  https://github.com/odoo/odoo/blob/48109c15d319ae69a8fcdd4a6c3b19172a2de6d9/odoo/tools/template_inheritance.py#L144-L150

Version spec should match parent's root element's version
---------------------------------------------------------

It has also been added in the same revision:
https://github.com/odoo/odoo/commit/31b9bee00676e179c124f3d42539103305cf7423

This has been added in 7.0 to handle `version="7.0"` in xml views.

However, these `version="7.0"` have been removed in 8.0, in revision
odoo/odoo@faa09da325
After the above revision, there were still some leftovers,
but which have all been removed in 11.0 in revision
odoo/odoo@70942e4cfb
during an improvement of the RNG validator.

This is therefore enough evidence to say this code
checking for the `version` attribute in `locate_node`
is no longer useful since a quite long time now.
It can therefore be removed.

closes odoo/odoo#107340

Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
2022-12-12 11:22:32 +01:00
Chong Wang (cwg) eab341aec3 [FIX] core: drop mismatched and illegal model terms
The model terms translations assume the number of terms in each language of a
model_term translated field should be the same.
However, sometimes the assumption cannot be promised for sake of bad
translations

This commit
1. drops illegal model term translations while translating
2. drops mismatched terms at run-time in case the database has been contaminated

closes odoo/odoo#107373

X-original-commit: f9ab5ca3e99b2883ada18a8f6deb12d45608b43f
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-12-07 18:29:18 +01:00
Chong Wang (cwg) a0bd7d0d72 [IMP] core: improve translation import speed
this commit improves the performance of importing translations by
1. groupup udpating data for different languages and different modules
2. read model_terms translations data directly with xmlid to get rid of fetching
   data from ir_model_data

for modules:sale_management, point_of_sale, industry_fsm, helpdesk, mrp, mrp_plm
(62 installed modules)
time for test_language_install is 45.14% less

for all modules:
time for test_language_install is 68.03% less

closes odoo/odoo#107332

X-original-commit: 727ef9e0a2ca93089b472eb1313117fee8dc876d
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-12-06 17:12:21 +01:00
jard@odoo.com d65f3e64b2 [FIX] tools.image: Allow conversion of P to JPEG
While upgrade of a customer database, A P-type image
was being converted to JPEG image. It didn't allow to save
as direct conversion from P-type(palette) to JPEG. So, fix
covers conversion of such corner cases to first convert it
to RGB and then RGB can convert it to desired output format
opw-3043418

closes odoo/odoo#107300

X-original-commit: d0db7a14dec7ee6a983ede3bf7b968675f83979d
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
2022-12-06 17:12:02 +01:00
william-andre 88879f9f27 [IMP] core: allow to have virtual SQL constraints
Sometimes a constraint is too complex to be expressed in the general
`_sql_contraint`.
This allows to hook on a constraint if it matches the name in order to
display a user friendly error message.

X-original-commit: e1f06479a526c703ccabc441b1e194646206b966
Part-of: odoo/odoo#106325
2022-11-23 16:14:44 +01:00
abd-msyukyu-odoo 51b5ee8d5f [FIX] web_editor, knowledge: prevent modification of data-oe-protected
Change class="oe-blackbox" to data-oe-protected="true" (same purpose), because
the name was decided in a rush before 16.0

When the editor history management and sanitization (sanitize.js) handle nodes
of an embedded view, OWL can become corrupted, which will lead to a traceback.

This commit uses the attribute data-oe-protected="true" to prevent
historyRollbacks or sanitization, and ensure that protected elements children
are removed during the cleanForSave operation.

The collaborative DOMPurify sanitization is untouched, for security reasons. To
properly use `data-oe-protected` in collaborative mode, a more complex
mechanism should be put in place to ensure that the protected element children
are properly reconstructed for each client, and that it is not part of the
collaboration. (i.e.: embedded view in knowledge do not support the
collaborative mode currently).

This commit also blocks the selection handling in tables which would be in a
data-oe-protected="true" node.

Task-3012552

closes odoo/odoo#106255

X-original-commit: 394e4fe567bb2eecbd0865de82914aa7396fde63
Related: odoo/enterprise#34245
Signed-off-by: David Beguin (dbe) <dbe@odoo.com>
Signed-off-by: Abeloos Damien (abd) <abd@odoo.com>
2022-11-23 11:26:12 +01:00
FrancoisGe b1f87e1088 [IMP] tools: transpiler adds a QUnit.module to js test files
The purpose of this improvement is to easily identify the module of a
test and to be able to run only the tests of a module.

closes odoo/odoo#105530

Task-id: 3062058
Signed-off-by: Aaron Bohy (aab) <aab@odoo.com>
2022-11-21 11:50:05 +01:00
Vincent Schippefilt 2fb1ed9175 [FIX] profiler: fix typos
closes odoo/odoo#105928

Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2022-11-17 13:57:35 +01:00
Chong Wang (cwg) dc15f81d08 [IMP] core: new comments for translations
before this commit:
If a translation is used for web client, the translation has comment
'openerp-web'.

after this commit:
If a translation is used for web client, the translation has comment
'odoo-javascript'
If a translation is used for python, the translation has comment
'odoo-python'

PO/POT files with old comments still works. The only overhead is: translation
only for web client will also be stored as code translation for python.

closes odoo/odoo#105764

X-original-commit: 13a5fa56ac7a102ad38f7ecc11874786bdb2fe85
Signed-off-by: Wang Chong (cwg) <cwg@odoo.com>
2022-11-15 16:16:54 +01:00
Chong Wang (cwg) 424774e081 [FIX] core: fix export web client translations
before the commit: web client code translations cannot be exported in the po.

If a web client only translation (not used for python translation and has
comment openerp-web) has its translation in the po file of /i18n, when it is
exported in a po file from Settings -> Translations -> Export Translation,
its corresponding translation is missing.

after this commit: web client code translations can be exported in the po.

X-original-commit: c2bf7665363b988d7c7e8bff43e71b7b1097ad84
Part-of: odoo/odoo#105764
2022-11-15 16:16:54 +01:00
Rémy Voet (ryv) 29e9d99364 [FIX] core: field indexed by trigram cannot become translated
Field indexed by trigram cannot become translated because the
`convert_column_translatable` is based on the old index naming convention
(changed in https://github.com/odoo/odoo/pull/100736).
Then it generates a PostgreSQL error:
`psycopg2.errors.DatatypeMismatch: operator class "gin_trgm_ops" does not accept data type jsonb`

closes odoo/odoo#105295

Signed-off-by: Raphael Collet <rco@odoo.com>
2022-11-08 13:36:05 +01:00
Chong Wang (cwg) d57d8534bb [FIX] core: fix search with accent
For a trigram indexed translated field field_x,
domain leaf ('field_x', '=', value) and ('field_x', 'ilike', pattern)
will miss records if the pattern contains accent characters

This commit set ensure_ascii=False for json.dumps to avoid escaping special
characters

opw-3048182

closes odoo/odoo#105240

X-original-commit: 26b5c8c2baeaee9fcd5384aafa5f9b39543d6903
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Wang Chong (cwg) <cwg@odoo.com>
2022-11-07 21:37:44 +01:00
Rémy Voet (ryv) 0787f150b1 [FIX] core: make index naming without conflicts
There are two issues with the index naming convention used by the ORM:

Problem 1: it is possible to have naming conflict for indexes.  For
instance, the name 'slide_channel_tag_group_sequence_index' is used for
both fields slide.channel.tag.group.sequence and
slide.channel.tag.group_sequence.  Only the first index will be created.

Solution 1: we separate the model and field names with a double
underscore instead of a single one, which is the same strategy as with
LEFT JOIN aliases.  This is correct because model names don't contain
such double underscores or underscores as prefix or suffix (it is not
forbiden but model name should follow the 'dot notation'.)

Problem 2: index names can be longer than 63 chars, but PostgreSQL
silently truncates it.  This doesn't actually break anything (PostgreSQL
also truncates values when we check the existence of indexes) but it can
lead to using the same name twice.  There is hopefully not any example
in our code.

Solution 2: if the name is too large, we truncate it and pad it with a
hash of the complete name to match 63 characters, which is also the
strategy used for LEFT JOIN aliases.

task-2984730

closes odoo/odoo#100736

Related: odoo/upgrade#3957
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-11-07 15:57:28 +01:00
Krzysztof Magusiak 167cf8fff8 [FIX] tools.image: Update IMAGE_MAX_RESOLUTION
New phones go up to 48MP (Samsung Galaxy A22).
Also the error message was saying 4.5 instead of 45.

opw-3020614, opw-3020502

Close #104424

closes odoo/odoo#104546

X-original-commit: 8b19107c69c648dbfba2a9304c04f0f55aac4b46
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2022-11-02 09:45:40 +01:00
xmo-odoo 84e6be4150 [REM] core: deprecated osv-memory-age-limit option
Part-of: odoo/odoo#98138
2022-10-26 19:03:47 +02:00
Xavier Morel 18446f23f1 [REM] core: support for the deprecated <report> and <act_window> tags
Part-of: odoo/odoo#98138
2022-10-26 19:03:46 +02:00
Chong Wang (cwg) 105e0b9ef2 [FIX] core: search translated fields with characters needed to be escaped
The trigram index function jsonb_path_query_array("column_name", '$.*')::text
uses all translations' representations to build the indexed text. So the
original text needs to be JSON-escaped correctly to match it.

X-original-commit: 7547df664945dddcb839e4903068f7f25ecfc08c
Part-of: odoo/odoo#103031
2022-10-11 14:10:16 +02:00
Chong Wang (cwg) bde30899a4 [FIX] core: new PO file overrides old ir_translation
Before: while upgrading to 16.0, the PO file is loaded before migrating
ir_translate. As a result, old translations from ir_translation
override the new translations imported by new PO files.

After: translations imported by the PO file take priority over the
translations migrated from the ir_translation table.

X-original-commit: 178270eff48221537a4efba1ad8fa0b53a1516f2
Part-of: odoo/odoo#103031
2022-10-11 14:10:15 +02:00
Antoine Dupuis 5a09449ddc [FIX] tools: fix filename resolution in XSD imports
With PRs odoo/89145 and enterprise/26393 came new methods for
retrieving XSD files and using them for XML validation.

The new retrieval method expects modules to provide a 'prefix' that
is prepended to the XSD's filename. For example, l10n_cl_edi will
name its XSD files 'l10n_cl_edi.<filename>.xsd'.

However, this messes things up when one XSD file needs to import
another. For example, l10n_cl_edi.DTE_v10.xsd has the statement
'<xs:include schemaLocation="SiiTypes_v10.xsd"/>'

Currently, the filename resolver has no way of knowing that this
should resolve to 'l10n_cl_edi.SiiTypes_v10.xsd', not
'SiiTypes_v10.xsd'.

In addition, the new retrieval method saves the ZIP archives
received over the network under the '<filename.xsd>'. Thus
'SiiTypes_v10.xsd' might actually be a ZIP-encoded file.

So, we need to do something to fix the imports.
Here are two possible solutions:
1. We scrap this 'prefix' stuff and either save the ZIP files under
   a different name, or we just don't save them.
2. Or, we provide a mechanism for indicating a prefix to the filename
   resolver.

Personally, I don't see the point in saving the ZIP files, and this
'prefix' stuff seems pointless. So I prefer solution 1.

But, because I assume there must be a reason to all of that 'prefix'
stuff, here is an implementation of solution 2.

I'd be keen to know the reason, btw.

EDIT:
In addition to the first issue described above, we have the second
issue that some XSD files returned by the Chilean SII are encoded
using ISO-8859-1 encoding (e.g. SiiTypes_v10.xsd). If we leave them
in this encoding, then LXML isn't able to parse them when performing
imports.

closes odoo/odoo#102601

Solution: convert the files to UTF-8 before storing them.
X-original-commit: 75555df56475b457331938453657c1f73d231e33
Related: odoo/enterprise#32482
Signed-off-by: Josse Colpaert <jco@odoo.com>
2022-10-07 14:37:14 +02:00
Walid HANNICHE (waha) 82265c52ca [FIX] tools: fix underline and italic in logged notes
Steps to reproduce:
- in any app log a note using full composer
- add styles to the text (underline, strike-through, italic)

Bug:
styles except bold are removed

Fix:
added missing styles to the whitlelist

opw-2956374

closes odoo/odoo#102056

X-original-commit: eb07ab104d5f47574e14db811108a7d0cbbbe4b2
Signed-off-by: David Monjoie (dmo) <dmo@odoo.com>
2022-10-04 16:01:06 +02:00
Damien Abeloos 1f6005070f [IMP] tools: keep additional data when sanitizing
Purpose is to allow to have ``data-behavior-props`` and ``data-prop-name``
on HTML nodes and keep it even when HTML content is sanitized. Those attributes
will be used notably in Knowledge application to store some structured content
e.g. used to populate props.

Prepares Task-2796156
Prepares odoo/enterprise#29423

X-original-commit: 7ddb4685567c115ca4e14fa7e5f845b0cb33fd7c
Part-of: odoo/odoo#101694
2022-09-30 10:33:01 +02:00
Lucas Lefèvre f07cd645c9 [FIX] tools: don't extract missing spreadsheet data source name
Previous versions of spreadsheet files don't have a name
for list and pivot datasources.

The spreadsheet template "pipeline dashboard" in `document_spreadsheet_crm`
is such a file.
As a consequence, exporting the source terms from this module fails
with a `KeyError: 'name'`

Part-of: odoo/odoo#101659
2022-09-30 09:31:52 +02:00
Lucas Lefèvre 723d8f8b50 [FIX] tools: extract translatable terms from spreadsheet dashboard
This commit extract translatable content from spreadsheet data.
- chart titles and description
- pivot and list names
- argument of _t() functions
- labels of links

X-original-commit: d210a55f48b316fdf38fcbe5ae5a70cf0e2c3814
Part-of: odoo/odoo#101120
2022-09-26 14:43:21 +02:00
Martin Trigaux 5150d003fd [I18N] *: export 16.0 source terms
closes odoo/odoo#101053

X-original-commit: 7d4c9d9736cc2ff0ee91dc17ea0f952c9a3dc65a
Related: odoo/enterprise#31752
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2022-09-24 15:00:56 +02:00
Martin Trigaux ffbd1ff062 [IMP] tools: do not export attribute content
Nodes like
<attribute name="t-on-dragenter.stop.prevent">() => state.dropzoneVisible = true</attribute>
should not be exported in the translations files but
<attribute name="string">Hello World</attribute>
should be kept

It was already the case in the xml processing method for server side
templates.
Since 16.0, client side QWeb views also use the same inheritance
mechanism than the one on the server side so the exception needs to be
replicated.

X-original-commit: e05d1192481935b2cd7aae7e939f98e60e002899
Part-of: odoo/odoo#101053
2022-09-24 15:00:56 +02:00
Xavier-Do 33b44ce84b [FIX] config: don't save deprecated longpolling-port
Longpolling port is replaced with gevent port and is deprecated
This commit avoid saving the value.

Not really usefull but when saved the value was None leadind to an error
when casting to int. The default value should be an int.

closes odoo/odoo#100991

X-original-commit: adac9a9ceaa173c0181a4fc57d0ea83b7a37fc71
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2022-09-23 14:37:17 +02:00
Chong Wang (cwg) a61c0e3fc4 [FIX] tools: export pot files
Before this commit, it was not possible to export the .pot files

Part-of: odoo/odoo#100573
2022-09-20 13:48:49 +02:00
ef00294e71 [IMP] core: store translated fields as JSONB columns
Translated fields no longer use the model ir.translation.  Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table.  The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages).  Empty text is allowed in translation values, but not NULL.

Here are examples for a field with translate=True:

    NULL
    {"en_US": "Foo"}
    {"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
    {"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}

Like before, writing False to the field makes it NULL, i.e., False in all
languages.  However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.

Here are examples for a field with translate=xml_translate:

    NULL
    {"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}

Change for callable(translate) fields: one can now write any value in any
language on such a field.  The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values.  Basically the
structure of the value must remain the same in all languages, like before.

Reading a translated field is now both simpler and faster than the former
implementation.  We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:

    SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...

The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages).  For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.

Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language.  Before this change, only the source value of the
field ('en_US') could be indexed.

Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages.  We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.

Code translations are no longer stored into the database.  They become static,
and are extracted from the PO files when needed.  The worker simply uses a cache
with extracted code translations for performance.  This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker.  Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).

Performance summary:
 (+) reading 'model' translated fields is faster
 (+) reading 'model_terms' translated fields is much faster (no need to inject
     translations into the source value)
 (+) searching translated fields with operator 'ilike' is much faster when the
     field is indexed with 'trigram'
 (+) updating translated fields requires less ORM flushing
 (-) importing translations from PO files is 2x slower

Some extra fixes:
 - make field 'name' of ir.actions.actions translated; because of the PG
   inheritance, this is necessary to make the column definition consistent in
   all models that inherit from ir.actions.actions.
 - add some backend API for the web/website client for editing translations
 - move methods get_field_string() to model ir.model.fields
 - move _load_module_terms to model ir.module.module
 - adapt tests in test_impex, test_new_api
 - because env.lang is injected into SQL queries, its returned value is
   now guaranteed to correspond to a valid active language or None
 - remove wizard to insert missing translations (no longer makes sense)

task-id: 2081307

Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2022-09-15 22:37:50 +02:00
Raphael Collet 50767ef90e [IMP] core: simplify code for column conversion in fields 2022-09-15 22:30:56 +02:00
Raphael Collet 70c776f21f [FIX] core: CLI --load-language was broken 2022-09-15 22:30:56 +02:00
Rémy Voet (ryv) 4087bcdc5e [IMP] core: add unaccent to trigram indexes
We added trigram index for char fields since https://github.com/odoo/odoo/pull/83015.
But if `unaccent` is installed in the database (and isn't force to
`False` on field), these new trigram indexes are pointless and cost a
lot for nothing (almost nothing, it still can be used for equality
operator but in this case a btree will be far more efficient).

The simple way to fix it is to add `unaccent(<column>)` in the index
trigram definition, but unfortunately `unaccent` is not immutable and
may therefore not be indexed.  In order to make `unaccent` indexable, we
must declare it as immutable (see
https://stackoverflow.com/questions/11005036/does-postgresql-support-accent-insensitive-collations/11007216#11007216
for more information and how to do that).

With this patch, trigram indexes are created with `unaccent(<column>)`
if the function `unaccent` is available in the database, and for the
fields that are not declared with `unaccent=False`.  Moreover, we issue
a warning when `unaccent` is available but is not immutable, in which
case most trigram indexes will be useless.

odoo/upgrade#3736
task-2551518

closes odoo/odoo#95943

Signed-off-by: Rémy Voet <ryv@odoo.com>
2022-09-05 18:34:53 +02:00
std-odoo 87307a9010 [IMP] base: add new "Properties" fields
Purpose
=======

Add a new field "Properties" to be able to light customization of workflows
based on a parent model. Those properties acts in some ways like Odoo fields
without requiring specific columns e.g. add new properties on tasks of a
specific project.

Usage
=====

Define properties on a parent model (e.g. project) with

```
    attributes_definition = fields.PropertiesDefinition('Message Properties')
```

It defines properties available on children: types, default, value, model for
relational properties,...

Use it on children records (e.g. task) with

```
    attributes = fields.Properties(
        string='Properties',
        definition='parent_id.attributes_definition',
    )
```

Technical
=========

Parent | Properties definition
------------------------------
The properties definition is stored on the parent, on a JSON field.
This definition contains the type of the properties, the default value,
the model of the many2one,...
```
[
    {
        'name': 'name',
        'string': 'Name',
        'type': 'char',
        'default': 'Default Name',
    }, {
        'name': 'partner_id',
        'string': 'Partner',
        'type': 'many2one',
        'comodel': 'res.partner',
    },
]
```

Child | Properties values
-------------------------
The value is stored on the child, using a Properties field.
```
{
    'name': 'Mitchel',
    'partner_id': 1337,
}
```

When we read this field, we will automatically read the definition on
the parent, and merge both JSON into one, so the web client has the
value of each property, and their definition.

```
[
    {
        'name': 'name',
        'string': 'Name',
        'type': 'char',
        'default': 'Default Name',
        'value': 'Mitchel',
    }, {
        'name': 'partner_id',
        'string': 'Partner',
        'type': 'many2one',
        'comodel': 'res.partner',
        'value': 1337,
    },
]
```

Integrity
---------
If we remove a property on the parent, we won't update the child value.

Instead, when we read the child properties, we will filter them based
on the parent. So the removed properties will be removed the next time
we write on the field.

In the same logic, the many2one existence is checked when we read the
field. There's no foreign key between the integer stored in the JSON
in the SQL row corresponding to the record in database.

Write
-----
We can write on the Properties field with a list of field definition
+ value.

Some types are not JSONifiable (like the date, datetime), they are
stored as string in database and parsed when we read the value.

In order to update the parent definition by writing on the child,
you need to add the dict key `definition_changed` or
`definition_deleted`. This is because we need to be able to know
if the definition has been changed without doing extra SQL queries.

Access rights
-------------
A user can add a many2one / many2many property to a model only if he
has the access rights to it.

Many2one / Many2many
--------------------

The model choice of a many2one / many2many properties was subject to
changes.

First implementation stored models in both parent and children to easily
spot changes and avoid complex queries when fetching records, trying to
synchronize them, ...

As this leads to storing a lot of duplicated content we choose to instead
reset the value on the child if the model has been change. We generate a
new name for the property. So it behaves like if we removed the property
and created a new one.

To be able to restore the old value (e.g. if by mistake we changed the
model, and go back to the old model), we store the initial states.

Task-2852259

Part-of: odoo/odoo#95184
2022-08-29 23:46:06 +02:00
Raphael Collet 07744c64e9 [FIX] core: in ir.model.fields, discard deleted fields from registry
This patch discard the fields to delete from data structures like field
triggers, field inverses, and fields to compute.

This is quite useful when uninstalling modules.  This commit fixes the
uninstallation of modules base_setup, bus, mail, web_tour.

closes odoo/odoo#98668

X-original-commit: 49398a769b2ec4965bf148f10b5e13d0a6c1acff
Signed-off-by: Raphael Collet <rco@odoo.com>
2022-08-25 22:46:04 +02:00
Pratik Raval bb24757d08 [IMP] crm: detect leads based on similar phone/mobile number
Currently, the 'similar lead detection' mechanism only considers email,
contact name, and partner name while finding duplicate leads in CRM.

After this commit, it will also consider the mobile/phone number for
the same. Note that the mobile numbers and phone numbers both will be
matched with each other while finding duplicate leads.

Task-2817884

closes odoo/odoo#88372

Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2022-08-25 19:56:48 +02:00
Florian Charlier db7c9ba7c5 [IMP] tools, website_{*}: increment multiple fields at once
* = blog, forum, slides

For performance reasons, allow to increment multiple fields of the same record
within the same query.

With python tests.

Task-2663320
Part of odoo/odoo#79615
2022-08-25 14:24:53 +02:00
tsm-odoo 29468b1272 [IMP] config: deprecate longpolling port in favor of gevent port
This commit is part of the websocket integration in Odoo.
The longpolling port does not make sense anymore: longpolling has been dropped.
This commit deprecate the `--longpolling-port` option and replace it by the
`--gevent-port` option.

closes odoo/odoo#75510

Related: odoo/enterprise#23184
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
2022-08-23 17:55:11 +02:00
tsm-odoo f8e794a864 [IMP] bus, tools: add rate limiting to incoming websocket messages
This commit is part of the websocket integration in Odoo.
It focuses on implementing rate limiting to the incoming
websocket messages. When opening a connection, a burst of messages
is allowed. When requests are received too fast, an exception is
raised and the websocket is closed.

Two config parameters are added to customize the rate limtier
behavior:
   - websocket_rate_limit_delay: Integer specifying the seconds that
     should space out two requests.
   - websocket_rate_limit_burst: Integer specifying how many websocket
     frames can be accepted in excess of the specified rate.

Part-of: odoo/odoo#75510
2022-08-23 17:55:09 +02:00
tsm-odoo e06bb9a42d [ADD] bus: add websocket implementation
This commit is the first commit of the websocket integration in Odoo.
It focuses on the implementation of the websocket protocol as per RFC6455.

The implementation is tested thanks to the autobahn test suite.

A config parameter is available to customize the websocket connection:
   - websocket_keep_alive_timeout (default 600): Integer specifying how
     many seconds a websocket connection should be kept alive

Part-of: odoo/odoo#75510
2022-08-23 17:55:09 +02:00
Xavier Morel 9c0fa1efe3 [CHG] core: deprecate get_module_filetree
It's pretty much unused and fairly complicated.

Also deprecate `listdir` entirely since `get_module_filetree` is the
only extant user of the recursive listdir.

closes odoo/odoo#98034

Related: odoo/enterprise#30403
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2022-08-23 07:36:24 +02:00