The trigram index function jsonb_path_query_array("column_name", '$.*')::text
uses all translations' representations to build the indexed text. So the
original text needs to be JSON-escaped correctly to match it.
X-original-commit: 7547df664945dddcb839e4903068f7f25ecfc08c
Part-of: odoo/odoo#103031
Before: while upgrading to 16.0, the PO file is loaded before migrating
ir_translate. As a result, old translations from ir_translation
override the new translations imported by new PO files.
After: translations imported by the PO file take priority over the
translations migrated from the ir_translation table.
X-original-commit: 178270eff48221537a4efba1ad8fa0b53a1516f2
Part-of: odoo/odoo#103031
With PRs odoo/89145 and enterprise/26393 came new methods for
retrieving XSD files and using them for XML validation.
The new retrieval method expects modules to provide a 'prefix' that
is prepended to the XSD's filename. For example, l10n_cl_edi will
name its XSD files 'l10n_cl_edi.<filename>.xsd'.
However, this messes things up when one XSD file needs to import
another. For example, l10n_cl_edi.DTE_v10.xsd has the statement
'<xs:include schemaLocation="SiiTypes_v10.xsd"/>'
Currently, the filename resolver has no way of knowing that this
should resolve to 'l10n_cl_edi.SiiTypes_v10.xsd', not
'SiiTypes_v10.xsd'.
In addition, the new retrieval method saves the ZIP archives
received over the network under the '<filename.xsd>'. Thus
'SiiTypes_v10.xsd' might actually be a ZIP-encoded file.
So, we need to do something to fix the imports.
Here are two possible solutions:
1. We scrap this 'prefix' stuff and either save the ZIP files under
a different name, or we just don't save them.
2. Or, we provide a mechanism for indicating a prefix to the filename
resolver.
Personally, I don't see the point in saving the ZIP files, and this
'prefix' stuff seems pointless. So I prefer solution 1.
But, because I assume there must be a reason to all of that 'prefix'
stuff, here is an implementation of solution 2.
I'd be keen to know the reason, btw.
EDIT:
In addition to the first issue described above, we have the second
issue that some XSD files returned by the Chilean SII are encoded
using ISO-8859-1 encoding (e.g. SiiTypes_v10.xsd). If we leave them
in this encoding, then LXML isn't able to parse them when performing
imports.
closesodoo/odoo#102601
Solution: convert the files to UTF-8 before storing them.
X-original-commit: 75555df56475b457331938453657c1f73d231e33
Related: odoo/enterprise#32482
Signed-off-by: Josse Colpaert <jco@odoo.com>
Steps to reproduce:
- in any app log a note using full composer
- add styles to the text (underline, strike-through, italic)
Bug:
styles except bold are removed
Fix:
added missing styles to the whitlelist
opw-2956374
closesodoo/odoo#102056
X-original-commit: eb07ab104d5f47574e14db811108a7d0cbbbe4b2
Signed-off-by: David Monjoie (dmo) <dmo@odoo.com>
Purpose is to allow to have ``data-behavior-props`` and ``data-prop-name``
on HTML nodes and keep it even when HTML content is sanitized. Those attributes
will be used notably in Knowledge application to store some structured content
e.g. used to populate props.
Prepares Task-2796156
Prepares odoo/enterprise#29423
X-original-commit: 7ddb4685567c115ca4e14fa7e5f845b0cb33fd7c
Part-of: odoo/odoo#101694
Previous versions of spreadsheet files don't have a name
for list and pivot datasources.
The spreadsheet template "pipeline dashboard" in `document_spreadsheet_crm`
is such a file.
As a consequence, exporting the source terms from this module fails
with a `KeyError: 'name'`
Part-of: odoo/odoo#101659
This commit extract translatable content from spreadsheet data.
- chart titles and description
- pivot and list names
- argument of _t() functions
- labels of links
X-original-commit: d210a55f48b316fdf38fcbe5ae5a70cf0e2c3814
Part-of: odoo/odoo#101120
Nodes like
<attribute name="t-on-dragenter.stop.prevent">() => state.dropzoneVisible = true</attribute>
should not be exported in the translations files but
<attribute name="string">Hello World</attribute>
should be kept
It was already the case in the xml processing method for server side
templates.
Since 16.0, client side QWeb views also use the same inheritance
mechanism than the one on the server side so the exception needs to be
replicated.
X-original-commit: e05d1192481935b2cd7aae7e939f98e60e002899
Part-of: odoo/odoo#101053
Longpolling port is replaced with gevent port and is deprecated
This commit avoid saving the value.
Not really usefull but when saved the value was None leadind to an error
when casting to int. The default value should be an int.
closesodoo/odoo#100991
X-original-commit: adac9a9ceaa173c0181a4fc57d0ea83b7a37fc71
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Translated fields no longer use the model ir.translation. Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table. The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages). Empty text is allowed in translation values, but not NULL.
Here are examples for a field with translate=True:
NULL
{"en_US": "Foo"}
{"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
{"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}
Like before, writing False to the field makes it NULL, i.e., False in all
languages. However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.
Here are examples for a field with translate=xml_translate:
NULL
{"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}
Change for callable(translate) fields: one can now write any value in any
language on such a field. The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values. Basically the
structure of the value must remain the same in all languages, like before.
Reading a translated field is now both simpler and faster than the former
implementation. We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:
SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...
The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages). For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.
Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language. Before this change, only the source value of the
field ('en_US') could be indexed.
Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages. We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.
Code translations are no longer stored into the database. They become static,
and are extracted from the PO files when needed. The worker simply uses a cache
with extracted code translations for performance. This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker. Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).
Performance summary:
(+) reading 'model' translated fields is faster
(+) reading 'model_terms' translated fields is much faster (no need to inject
translations into the source value)
(+) searching translated fields with operator 'ilike' is much faster when the
field is indexed with 'trigram'
(+) updating translated fields requires less ORM flushing
(-) importing translations from PO files is 2x slower
Some extra fixes:
- make field 'name' of ir.actions.actions translated; because of the PG
inheritance, this is necessary to make the column definition consistent in
all models that inherit from ir.actions.actions.
- add some backend API for the web/website client for editing translations
- move methods get_field_string() to model ir.model.fields
- move _load_module_terms to model ir.module.module
- adapt tests in test_impex, test_new_api
- because env.lang is injected into SQL queries, its returned value is
now guaranteed to correspond to a valid active language or None
- remove wizard to insert missing translations (no longer makes sense)
task-id: 2081307
Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
We added trigram index for char fields since https://github.com/odoo/odoo/pull/83015.
But if `unaccent` is installed in the database (and isn't force to
`False` on field), these new trigram indexes are pointless and cost a
lot for nothing (almost nothing, it still can be used for equality
operator but in this case a btree will be far more efficient).
The simple way to fix it is to add `unaccent(<column>)` in the index
trigram definition, but unfortunately `unaccent` is not immutable and
may therefore not be indexed. In order to make `unaccent` indexable, we
must declare it as immutable (see
https://stackoverflow.com/questions/11005036/does-postgresql-support-accent-insensitive-collations/11007216#11007216
for more information and how to do that).
With this patch, trigram indexes are created with `unaccent(<column>)`
if the function `unaccent` is available in the database, and for the
fields that are not declared with `unaccent=False`. Moreover, we issue
a warning when `unaccent` is available but is not immutable, in which
case most trigram indexes will be useless.
odoo/upgrade#3736
task-2551518
closesodoo/odoo#95943
Signed-off-by: Rémy Voet <ryv@odoo.com>
Purpose
=======
Add a new field "Properties" to be able to light customization of workflows
based on a parent model. Those properties acts in some ways like Odoo fields
without requiring specific columns e.g. add new properties on tasks of a
specific project.
Usage
=====
Define properties on a parent model (e.g. project) with
```
attributes_definition = fields.PropertiesDefinition('Message Properties')
```
It defines properties available on children: types, default, value, model for
relational properties,...
Use it on children records (e.g. task) with
```
attributes = fields.Properties(
string='Properties',
definition='parent_id.attributes_definition',
)
```
Technical
=========
Parent | Properties definition
------------------------------
The properties definition is stored on the parent, on a JSON field.
This definition contains the type of the properties, the default value,
the model of the many2one,...
```
[
{
'name': 'name',
'string': 'Name',
'type': 'char',
'default': 'Default Name',
}, {
'name': 'partner_id',
'string': 'Partner',
'type': 'many2one',
'comodel': 'res.partner',
},
]
```
Child | Properties values
-------------------------
The value is stored on the child, using a Properties field.
```
{
'name': 'Mitchel',
'partner_id': 1337,
}
```
When we read this field, we will automatically read the definition on
the parent, and merge both JSON into one, so the web client has the
value of each property, and their definition.
```
[
{
'name': 'name',
'string': 'Name',
'type': 'char',
'default': 'Default Name',
'value': 'Mitchel',
}, {
'name': 'partner_id',
'string': 'Partner',
'type': 'many2one',
'comodel': 'res.partner',
'value': 1337,
},
]
```
Integrity
---------
If we remove a property on the parent, we won't update the child value.
Instead, when we read the child properties, we will filter them based
on the parent. So the removed properties will be removed the next time
we write on the field.
In the same logic, the many2one existence is checked when we read the
field. There's no foreign key between the integer stored in the JSON
in the SQL row corresponding to the record in database.
Write
-----
We can write on the Properties field with a list of field definition
+ value.
Some types are not JSONifiable (like the date, datetime), they are
stored as string in database and parsed when we read the value.
In order to update the parent definition by writing on the child,
you need to add the dict key `definition_changed` or
`definition_deleted`. This is because we need to be able to know
if the definition has been changed without doing extra SQL queries.
Access rights
-------------
A user can add a many2one / many2many property to a model only if he
has the access rights to it.
Many2one / Many2many
--------------------
The model choice of a many2one / many2many properties was subject to
changes.
First implementation stored models in both parent and children to easily
spot changes and avoid complex queries when fetching records, trying to
synchronize them, ...
As this leads to storing a lot of duplicated content we choose to instead
reset the value on the child if the model has been change. We generate a
new name for the property. So it behaves like if we removed the property
and created a new one.
To be able to restore the old value (e.g. if by mistake we changed the
model, and go back to the old model), we store the initial states.
Task-2852259
Part-of: odoo/odoo#95184
This patch discard the fields to delete from data structures like field
triggers, field inverses, and fields to compute.
This is quite useful when uninstalling modules. This commit fixes the
uninstallation of modules base_setup, bus, mail, web_tour.
closesodoo/odoo#98668
X-original-commit: 49398a769b2ec4965bf148f10b5e13d0a6c1acff
Signed-off-by: Raphael Collet <rco@odoo.com>
Currently, the 'similar lead detection' mechanism only considers email,
contact name, and partner name while finding duplicate leads in CRM.
After this commit, it will also consider the mobile/phone number for
the same. Note that the mobile numbers and phone numbers both will be
matched with each other while finding duplicate leads.
Task-2817884
closesodoo/odoo#88372
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
* = blog, forum, slides
For performance reasons, allow to increment multiple fields of the same record
within the same query.
With python tests.
Task-2663320
Part of odoo/odoo#79615
This commit is part of the websocket integration in Odoo.
The longpolling port does not make sense anymore: longpolling has been dropped.
This commit deprecate the `--longpolling-port` option and replace it by the
`--gevent-port` option.
closesodoo/odoo#75510
Related: odoo/enterprise#23184
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
This commit is part of the websocket integration in Odoo.
It focuses on implementing rate limiting to the incoming
websocket messages. When opening a connection, a burst of messages
is allowed. When requests are received too fast, an exception is
raised and the websocket is closed.
Two config parameters are added to customize the rate limtier
behavior:
- websocket_rate_limit_delay: Integer specifying the seconds that
should space out two requests.
- websocket_rate_limit_burst: Integer specifying how many websocket
frames can be accepted in excess of the specified rate.
Part-of: odoo/odoo#75510
This commit is the first commit of the websocket integration in Odoo.
It focuses on the implementation of the websocket protocol as per RFC6455.
The implementation is tested thanks to the autobahn test suite.
A config parameter is available to customize the websocket connection:
- websocket_keep_alive_timeout (default 600): Integer specifying how
many seconds a websocket connection should be kept alive
Part-of: odoo/odoo#75510
It's pretty much unused and fairly complicated.
Also deprecate `listdir` entirely since `get_module_filetree` is the
only extant user of the recursive listdir.
closesodoo/odoo#98034
Related: odoo/enterprise#30403
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
STEPS in v14:
1. Set user's timezone to Asia/Qatar
2. Inventory > Config > Setting > Enable Lots & Serial Numbers, Expiration Dates, Display Lots & Serial Numbers on Delivery Slips, Display Expiration Dates on Delivery Slips
3. Create a product (storable, tracking by lots, expiration date)
4. Manually update the on-hand quantity for the product
5. Create a lot for the product created in Step 3 > Set the expiration date to 3/30/2025 00:00:00
6. Create an SO with the product > Confirm it to Delivery
7. Validate the Delivery > Print the Delivery Slip
Before this commit the expiration date is rendered as 03/29/2025 instead of 03/30/2025
opw-2901367
closesodoo/odoo#98371
Signed-off-by: Julien Castiaux <juc@odoo.com>
It's really not a general purpose utility, it's just a way of forcing
a lazy collection's evaluation.
Alternatively, add a helper for this to / alongside `lazy`? Note that
the lazy object(s) may not be the top-level.
closesodoo/odoo#98023
Related: odoo/upgrade#3776
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
They're not used much and they don't seem much of an advantage
compared to just calling `subprocess` with the discovery utility
functions.
While at it, fix `dump_db` to not unnecessarily have an open stdin to
`pg_dump`.
Part-of: odoo/odoo#98023
Currently, there is no globally available way to load xsd files.
Modules that use xsd files to validate xml files all load the files with their own methods, but in the end they all do the same thing. It is redundant and hard to maintain. On top of that, new modules that need to load such xsd files need to implement it again.
This commit adds an easy way to load xsd files (either directly .xsd files or from .zip archives) and save them as ir.attachment to use with _check_with_xsd method for xml validation purposes.
It also adds a function to validate an XML file with an XSD. This function allows for a reloading method to be called if the XSD file was not found in database.
In order to avoid excessively downloading XSD files (during tests for instance), the 'skip_xsd' flag can be set to True in the context. This will skip the XSD validation (and thus download).
task id=2782053
closesodoo/odoo#89145
Related: odoo/enterprise#26393
Signed-off-by: Laurent Smet <las@odoo.com>
When a record is created through xml data, its HTML fields should
receive a `type="html"` attribute, not a `type="xml"` attribute.
When important XML data with XML type instead of HTML type will have 2
differences:
- The field value will be prefixed by `<?xml version="1.0"/>`
- If the HTML contains multiple root nodes, the value will be wrapped in
a `<data/>` tag.
See `_fix_multiple_roots()` and the `xml_import` class for more details.
closesodoo/odoo#98239
Related: odoo/enterprise#30491
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
In some case the frame may have a filename and corresponding file but
no lineno. This may lead to an error
File ".../odoo/tools/profiler.py", line 571, in _add_file_lines
line = filelines[lineno - 1]
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
This commit simply fixes this by skipping the logic if we have no lineno.
closesodoo/odoo#98217
X-original-commit: 44715af97413da3608e99f211c361a498f25d964
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
We deprecated all those in 14.0, so 16.0 seems like a good time to
remove them.
`getppid` was not deprecated but it doesn't seem useful to keep
closesodoo/odoo#97987
Around: windows support was added to getppid in Python 3.2.
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
To reproduce the issue:
(Need stock_barcode)
1. Create a product:
- Barcode: 1234567890123
2. Print the label
3. Try to scan the barcode and check the value read.
Error: The value is 1234567890128, the last digit is incorrect (8
instead of 3)
When printing a barcode, we use the library 'report-lab' to generate a
barcode image from a value and a barcode type. In case of EAN-13, if the
value contains a non-digit character, it will raise an error. We then
catch the error and retry to generate the barcode according to the
barcode type Code128:
https://github.com/odoo/odoo/blob/87698d90f02bfe93c6e643f4876a5ccd74788eff/odoo/addons/base/models/ir_actions_report.py#L569-L575
However, if the value contains only digits, the method will use the 12
first digits:
https://github.com/mattjmorrison/ReportLab/blob/dade0f303cb6fcdbe535c4cc92e6102c2417b699/src/reportlab/graphics/barcode/eanbc.py#L187-L188
and will then add the last one, the check digit, which is computed by
the library. This explains why, in the above use case, the barcode value
returned by the scanner is not the same than the expected one.
Note: Similar behavior with type EAN-8
OPW-2902150
closesodoo/odoo#97052
X-original-commit: a2c7470f8fd46b2520f7b8f0750c63216273ce96
Related: odoo/enterprise#30160
Signed-off-by: Steve Van Essche <svs@odoo.com>
Signed-off-by: Adrien Widart <awt@odoo.com>
The render API was confusing as mixing the access to the report and
the rendering env.
The ambiguity was present for code such as
`report.sudo()._render(record_ids)` where it was not clear if the
`sudo()` is needed to access to `report` or to `record_ids`. For low
priviledge users (such as portal or public), it was common to use
`report.with_user(SUPERUSER_ID)._render(record_ids)`.
This PR changes the render methods signature to be `api.model`. The
`report_ref` can be:
- ir.actions.report external id
- ir.actions.report id
- ir.actions.report recod
- `report_name` value
This will allow to call the report methods with any user and no longer
need to use `with_user(1)` to render reports as public user.
Task-id 2670865
closesodoo/odoo#91341
Related: odoo/upgrade#3650
Related: odoo/enterprise#27323
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Pillow 9.1 deprecates most if not all toplevel Image constants:
https://pillow.readthedocs.io/en/stable/releasenotes/9.1.0.html#constants
These constants have been moved to thematic enum classes (e.g. all the
resampling constants in `PIL.Image.Resampling`).
This triggers warnings in Odoo, and the removal delay is quite short
(slated for Pillow 10, release planned mid 2023). Pillow 9.1 is also
already in Debian Bookworm (current testing).
Fix by shimming at the import level: if the enums are available import
them into the local namespace, otherwise alias `PIL.Image` itself as
to the enum.
closesodoo/odoo#96799
X-original-commit: 7be04d31bad078681ef0a2919234c11840a2e8e2
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit fixes the profiler datetime usage by properly using the
'real_datetime_now' as a callable.
This was breaking the output json file when profiling.
closesodoo/odoo#96456
X-original-commit: 5fec27e42b033f79933afb3d0c85b142360981f9
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
When freezegun is used, the profiler and sql_db time are freezed,
Making the profile and sql perf counters invalids.
A possible solution would be to black list some modules in freezegun but
this doens't look possible in the pinned version (0.3.x).
Saving the builtin time.time is not enough, it looks like freezegun will
find all occurences and replace them.
We need to get the __call__ instead.
closesodoo/odoo#95100
X-original-commit: 9ac5fdf1e6d00e6e4b4ff4b6e94a3cd28f8cae11
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Move the definition of class Query to odoo.tools, in order to avoid
circular imports when importing Query in core Odoo modules.
Also reorganize imports in the impacted modules.
Part-of: odoo/odoo#66938
Leftovers of previous fixes of this issue.
closesodoo/odoo#95162
X-original-commit: dffc184862018670008ff4da839fc520a72bddca
Signed-off-by: Christophe Simonis <chs@odoo.com>
This commit fix 3 problem
Don't count studio field
------------------------
the commit https://github.com/odoo/odoo/commit/75b8c4eb1ed6e44f427886eea06ad216995e16c8
was supposed to stop counting count field generate by addins a smart
button with studio.
It works fine when base import module is not installed which is never
the case.
When base_import_module the field is counted as a field that belong
to a imported module installed : studio_customization
studio_customization should not be counted as an imported module
installed
Count Field with standard xml_id
--------------------------------
With https://github.com/odoo/odoo/commit/9afce4805fc8bac45fdba817488aa867fddff69b
Each manual can end up with an xml_id from a standard module:
the original module of the model
A standard module should never create a manual field so we can consider
they should be counted unless they match the criteria of the first
problem
If a field has a standard module xml_id and no other the module
should be odoo/studio and not the name of the standard module
Make possible to exclude some db record from cloc
-------------------------------------------------
It's possible to exclude some file in python module
but it's not possible to exclude some field or SA in
the database from the count
Make it possible if they are link to an xml_id
from the module __cloc_exclude__
The exclude record will be shown in cloc report with
the verbose mode
closesodoo/odoo#95028
X-original-commit: db52fb174461107e26b066147801f54a89d0dbf9
Related: odoo/enterprise#29007
Signed-off-by: Christophe Simonis <chs@odoo.com>
Previously, when importing both the default export and named imports
with a trailing comma, we would transpile into an object where the first
keys were the named imports, including any trailing comma, and
concatenating with the default import with a leading comma, this means
that the trailing comma would transpile to two successive commas which
is not valid syntax.
This commit fixes that by putting the default import first, and adding
the named imports after, preserving the trailing comma if any, but
precluding the possibility that we get two commas back to back.
closesodoo/odoo#93693
X-original-commit: 798d2f2c6e8a496f4ce0cd3c4f1609693e692254
Signed-off-by: Géry Debongnie <ged@odoo.com>
Signed-off-by: Samuel Degueldre <sad@odoo.com>
The methods to generate the attributes are called all the time in order
to reset the attribute dictionary. The profiler is modified so as not to
display directives in the log that do not exist on the tag.
Issue: attributes could be generated by directives and not be used (for
example on <t>). These attributes could end up unwittingly on the next
node.
closesodoo/odoo#93216
Issue: opw-2859447
X-original-commit: 38724f106c0747aef475991c894e9ee93c1db967
Signed-off-by: Vincent Schippefilt (vsc) <vsc@odoo.com>
Signed-off-by: Christophe Matthieu (chm) <chm@odoo.com>
The `t-cache` directive allows you to keep the rendered result
of a template part. The supplied key must be a tuple. This tuple
can contain recordset in this case the zone will be invalidated
each time the write_date of these records changes.
The `t-nocache` directive makes it possible to force rendering
of a part even if it is in a `t-cache`. The values available in
the `t-nocache` are the one provided when calling the template
(and therefore ignores any t-set that could have been done).
Part-of: odoo/odoo#88276
Before this commit, two lazy values could not always be compared. Indeed,
the comparison did indeed use the value for the self, but not the other.
A normally true comparison was returned as false. Inverting the values
causes the code to pass through the lazy functions of the objects. For
example, if we go through `__lt__` the fact of having reversed the
values means that we will necessarily go through the `__gt__` of the
other.
Part-of: odoo/odoo#88276
Since [1] which tried to fix the data-oe-xpath branding on nodes in some
cases, the branding actually became potentially incorrect on siblings of
a node which is replaced multiple times. E.g.
Parent view:
```xml
<hello>
<world class="a"></world>
<world class="b"></world>
<world class="c"></world>
</hello>
```
Child view 1:
```xml
<xpath position="//world[hasclass('a')]" position="replace">
<world class="new_a"></world>
</xpath>
```
Child view 2:
```xml
<xpath position="//world[hasclass('b')]" position="replace">
<world class="new_b"></world>
</xpath>
```
No problem, two distincts elements are replaced, the system understands
that the `data-oe-xpath` of the third world of the parent view should be
`/hello[1]/world[3]`.
But in this other case:
Parent view:
```xml
<hello>
<world class="a"></world>
<world class="b"></world>
<world class="c"></world>
</hello>
```
Child view:
```xml
<xpath position="//world[hasclass('a')]" position="replace">
<world class="new_a"></world>
</xpath>
```
Child view of the child view:
```xml
<xpath position="//world[hasclass('new_a')]" position="replace">
<world class="another_new_a"></world>
</xpath>
```
The `data-oe-xpath` of the third world of the parent view (in the
resulting view) was wrong: `/hello[1]/world[4]` -> because the system
saw two replacements + the unreplaced second `<world>`, so the index "4"
was computed.
Now the system will understand that the double replacement in fact acts
as a single replacement.
Note: this was also the same with "cross inheriting" (if the "new_a"
`<world>` of the child view was replaced by another child view of the
parent view).
At last, another 4th case was found and worth mentioning because it is
in fact the root cause of the problem. The problem is not actually the
double replacement as mentioned above but simply the replacement of a
root level element of a child view (which is what is basically done in
the last two mentioned cases). In that case, the root level nodes added
by the first child view have already their `data-oe-xpath` branding
computed before they are potentially replaced. Indicating the location
of the replacement in that case was thus only leading to bugs. E.g.
Parent view:
```xml
<hello>
<world class="a"></world>
<world class="b"></world>
</hello>
```
Child view:
```xml
<xpath expr="//world[hasclass('a')]" position="after">
<world class="x"></world>
<world class="y"></world>
</xpath>
```
Child view of the child view:
```xml
<xpath expr="//world[hasclass('x')]" position="replace"/>
```
Before this commit, before the branding is distributed, the result is:
```xml
<hello data-oe-model="ir.ui.view" data-oe-id="1439" data-oe-field="arch">
<world class="a"/>
<?apply-inheritance-specs-node-removal world?>
<world class="y" data-oe-id="1440" data-oe-xpath="/data/xpath/world[2]" data-oe-model="ir.ui.view" data-oe-field="arch"/>
<world class="b"/>
</hello>
```
=> Hence the `data-oe-xpath` of the last `<world>` was computed to
`/hello[1]/world[3]` instead of `/hello[1]/world[2]` after branding
distribution because the ProcessingInstruction marking the node
removal location should not have been added: it could only be useful
to following siblings which are not branded, which is not possible as
the branding added on the second `<world>` of the child view
(`/data/xpath/world[2]`) was computed before any removal.
Tests are added in this commit for the 3 last mentioned cases. As
explained, the last case is actually the same of the 2nd and 3rd ones
but it was decided to keep the 3 tests as it helps to understand the
problems better and, if the code evolves, it could become different
cases (= this is 3 cases which are currently technically equivalent but
these are different functionnal use cases). A test was written for the
first case then removed as it is basically a pure copy of other existing
tests written in [2] (trying to be improved by [1]).
[1]: https://github.com/odoo/odoo/commit/f67832a3ae0d9a3b5b53129132762e6bc1aed874
[2]: https://github.com/odoo/odoo/commit/c077ef05575d9677bce284195683f96c68386788closesodoo/odoo#92589
X-original-commit: d6e0b3d570a4b27f72852eb261660ad09de12eeb
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
We introduce a new decorator/context-manager to catch some exceptions
and re-raise them as another error. This utility main's purpose is to
hide a route that a user has no access to behind a fake HTTP 404 Page
not Found error.
The utility is at `odoo.tools.misc.replace_exceptions(*exceptions, by)`.
Its usage is as follow:
@route('/some/route', auth='public')
@replace_exceptions(AccessError, AccessDenied, by=NotFound())
def some_route(self):
if not request.session.uid:
raise AccessError("Must be connected to see this route")
...
Or as a context-manager if you don't want to except an entire function:
@route('/some/route', auth='public')
def some_route(self):
with replace_exceptions(AccessError, AccessDenied, by=NotFound()):
if not request.session.uid:
raise AccessError("Must be connected to see this route")
...
Task: 2800772
Close: #90433
Part-of: odoo/odoo#88134
Prepare an informative and nice-looking preview from the main content of the
email body, avoiding buttons/images alt etc.
Links, images, tables are removed.
Whitespace is added after the preview to avoid including the full message in
the preview (with markup).
Tags and attributes are also added to increase compliance with HTML5 standard,
including accessibility.
One additional SQL request is required to fetch the preview sub-template.
Task-2413355
closesodoo/odoo#86266
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
xmo-odoo