This commit moves and adapt the res.users.settings model from mail to base and
and allows to access and modify its content with web's user service.
This allows to access user settings without needing the mail module.
closesodoo/odoo#116005
Related: odoo/enterprise#38360
Signed-off-by: Michaël Mattiello (mcm) <mcm@odoo.com>
Translated fields no longer use the model ir.translation. Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table. The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages). Empty text is allowed in translation values, but not NULL.
Here are examples for a field with translate=True:
NULL
{"en_US": "Foo"}
{"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
{"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}
Like before, writing False to the field makes it NULL, i.e., False in all
languages. However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.
Here are examples for a field with translate=xml_translate:
NULL
{"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}
Change for callable(translate) fields: one can now write any value in any
language on such a field. The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values. Basically the
structure of the value must remain the same in all languages, like before.
Reading a translated field is now both simpler and faster than the former
implementation. We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:
SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...
The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages). For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.
Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language. Before this change, only the source value of the
field ('en_US') could be indexed.
Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages. We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.
Code translations are no longer stored into the database. They become static,
and are extracted from the PO files when needed. The worker simply uses a cache
with extracted code translations for performance. This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker. Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).
Performance summary:
(+) reading 'model' translated fields is faster
(+) reading 'model_terms' translated fields is much faster (no need to inject
translations into the source value)
(+) searching translated fields with operator 'ilike' is much faster when the
field is indexed with 'trigram'
(+) updating translated fields requires less ORM flushing
(-) importing translations from PO files is 2x slower
Some extra fixes:
- make field 'name' of ir.actions.actions translated; because of the PG
inheritance, this is necessary to make the column definition consistent in
all models that inherit from ir.actions.actions.
- add some backend API for the web/website client for editing translations
- move methods get_field_string() to model ir.model.fields
- move _load_module_terms to model ir.module.module
- adapt tests in test_impex, test_new_api
- because env.lang is injected into SQL queries, its returned value is
now guaranteed to correspond to a valid active language or None
- remove wizard to insert missing translations (no longer makes sense)
task-id: 2081307
Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Purpose
=======
Allow portal users to deactivate their accounts from the portal view.
After the deactivation, they are redirected to the login page, so
they can verify that they can not longer login with their credentials.
We first archived the record and remove sensitive information (password,
login, so he can not log in again with the same credentials).
After the deactivation, we blacklist the email and the phone of the
user, so we are sure that we never send him again email / SMS.
Then, we create a <res.users.deletion> to delete the user and the
partner in a CRON because this operation can be heavy (write_uid,
create_uid field on all models).
Task-2629544
Part-of: odoo/odoo#78298
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
QWeb is the primary templating engine used by Odoo. It is an XML
templating engine and used mostly to generate XML, HTML fragments and
pages.
To create new XML template, please see :doc:`QWeb Templates documentation
<https://www.odoo.com/documentation/15.0/developer/reference/frontend/qweb.html>`
In **input** you have an XML template giving the corresponding input
etree. Each etree input nodes are used to generate a python function.
This fonction is called and will give the XML **output**.
The ``_compile`` method is responsible to generate the function from the
etree, that function is a python generator that yield one output line at a
time. This generator is consumed by ``_render``. The generated function is
orm cached.
In the graphic below you can see theresume of the call of the methods
performed in the IrQweb class.
Odoo
┗━► _render (returns MarkupSafe)
┗━► _compile (returns function) ◄━━━━━━━━━┓
┗━► _compile_node (returns code string array) ◄━━━━━━━┓ ┃
┃ (add technical directives: t-inner-content, t-tag) ┃ ┃
┣━► _directives_eval_order (defined directive order) ┃ ┃
┃ ┃ ┃
┣━► _compile_directives (recursive) ◄━━━━┓ ┃ ┃
┃ ┣━► _compile_directive ┃ ┃ ┃
┃ ┃ ┗━► t-if ━━► _compile_directive_if ━┫ ┃ ┃
┃ ┃ ┗━► t-foreach ━━► _compile_directive_foreach ━┫ ┃ ┃
┃ ┃ ┗━► t-* ━━► ... ━┛ ┃ ┃
┃ ┃ ┗━► t-inner-content ━━► _compile_directive_inner_content ◄━━━━┓ ━┛ ┃
┃ ┃ ┗━► t-tag ━━► _compile_directive_tag ━┫ ┃
┃ ┃ ┗━► t-call ━━► _compile_directive_call ━┫ ━━━┛
┃ ┃ ┗━► t-out ━━► _compile_directive_out ◄━┓ ━┫
┃ ┃ ┗━► t-field ━━► _compile_directive_field ━┛ ┃
┃ ┃ ┃
┗━━┻━► _compile_static_node ━┛
Part-of: odoo/odoo#81024
This commit adds tooling to profile performance and save execution by
saving stack traces and queries to a file/database in specific format.
----------
Collectors
----------
For now, three different profiling modes (aka Collectors) are available
even if a last once should be introduced by @Gorash to profile qweb
execution.
- SQLCollector (or 'sql'): Saves the current stack trace and the query
every time Cursor.execute() is called. Any query executed on the thread
will be collected, no matter the cursor.
- PeriodicCollector (or 'traces_async'): Saves the stack trace every
'interval' seconds using a parallel thread to profile the caller thread.
The python implementation was optimized to minimize impact on
performance while remaining portable and easy to enable/disable
inside a odoo execution. Higher the frequency (lower the interval),
more impactful the profiling will become on the execution and increase
memory usage. From last experiments, 1ms looks to be a good minimum for
short executions.
- SyncCollector (or 'traces_sync'): Saves the stack trace every function
call/return. This collector is obviously quite impactful on performance
and can quickly overload the memory for long executions, but this is
quite useful to understand the precise path followed by some short
executions. Any time related information will be almost irrelevant with this
collector.
A base Collector defining minimal collectors features can easily be
extended to create custom collectors if needed.
----------------
Profiler & Usage
----------------
Collectors are not supposed to be used by themselves, but should be
given to a Profiler. The Profiler will synchronize collectors starts and
stop, and manage saving them to a file of in a ir_profile in the
database.
Exemple of usage:
```
with Profiler():
do_stuff()
```
This simple example will use the default collectors (sql and
traces_async) and save them to the database. The database is defined
automatically from current_thread 'dbname' if available.
Example of usage:
```
with Profiler(collectors=['sql'], db=False, path=/home/user/logs/do_stuff_profile/{time}):
do_stuff()
```
This more complex example disable the default behavior consisting
to save to the database, gives a path where the profile will be saved
and specify to only use the 'sql' collector. Note that
collectors=[SQLCollector()] would have the same behavior since
Collectors can be either a Collector instance or a string describing the
desired collector. This allows to define custom params for the
collectors and use custom collectors if needed.
Note that it is always possible to get results after execution without
saving it since they are available on the profiler.
```
with Profiler(collectors=['sql'], db=False) as p:
do_stuff()
print(len([None for entry in p.collectors[0].entries if ...]))
```
Profiler will also save the stack below the profiler start point, and
collectors will only collect the part of the stack over this stack.
This is a good way to reduce collectors CPU and memory usage.
Collected entries will be saved as follows:
```
[{
'start': 2.0,
'context': {},
'stack': [
['path_to_file', lno, 'func_name', 'line_content'],
...
],
},
...
]
```
SQLCollector will add three additional keys on each entry:
- query (query without parameters)
- full_query (mogrified query with parameters)
- time (the 'exact' execution time of the query)
----------------
ExecutionContext
----------------
A last tool, ExecutionContext, allows to define some context on some block of code:
Example of usage:
```
def process_modules(modules)
for module in modules:
with ExecutionContext(module=module): # note the 'not linter frienldy but still convenient' 2 spaces indentation
do_stuff(module):
```
This context will automatically be added in the stack as a virtual frame between
process_modules and do_stuff in order to split do_stuff from one single frame to
one frame per module.
----------
Speedscope
----------
The saved data are in a simple json format easy to analyze, but can't be visualized in
speedscope as they are. A utility class `Speedscope` can be used to generate a format
readable by speedscope. The used format is actually the format defined by speedscope,
meaning that all features should be available using it.
The output format is evented, meaning that we need to transform a list of samples
(a list of stack) to a list of event (going in/out a frame).
This is the main task of the Speedscope, as well as combining samples from different
sources, to display SQLCollector and PeriodicCollector results mixed together.
When stored on an ir_profile, the default speedscope generation can easily be generated
with the speedscope computed field.
This class can be used as it is but will mainly be useful for the next commit.
Special thanks to @rco-odoo for the in depth review and @Gorash for support.
Description of the issue/feature this PR addresses:
It is currently quite difficult to differentiate users. Most of the time, people
don't take the time to upload an actual avatar so everybody looks the same. This
PR generates a custom avatar with the users initials and random color to
differentiate them. For res.users, res.partner and hr.employee, image fields now
hold the binary image and avatar are used to show the image or svg.
Current behavior before PR:
Avatar had only random colors and was being saved in database, being inefficient
Desired behavior after PR is merged:
A new mixin defines image fields and in case no image is set, it generates an
SVG image with the user's initials and random color.
closesodoo/odoo#69819
Task: 2404630
Related: odoo/enterprise#18199
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
This commit changes the way assets are declared in Odoo modules.
Before: assets were declared in template files. Template bundles were
generated from primary templates, so technically any qweb template could
have been called as an asset bundle, with the 't-call-assets' directive.
Being standard qweb templates, they had access to standard HTML tags
(script, link, with or without raw scripts or style definition), qweb
directives (t-call, t-raw, etc.) and could be inherited by other
templates.
Now: assets are defined in the module's manifest and generated by the
't-call-assets' directive.
More information on the new system can be found on the updated user
documentation (see the "JavaScript Reference" section).
Task: 2352566
Co-authored-by: Bruno Boi <boi@odoo.com>
Co-authored-by: Julien Mougenot <jum@odoo.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
Co-authored-by: Mathieu Duckerts-Antoine <dam@odoo.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Simon Genin <ges@odoo.com>
The onboarding modal for setting up the few base fields of a company
has now been moved to a wizard
It is accessible from the general settings, but also in the onboarding
section of sale and account modules.
The following company settings are editable with that wizard:
- Set report **layout**:
The user can chose the overall look of the report. The current choices
are : *Standard* (default), *Background*, *Boxed* and *Clean*.
- Set company **logo**:
Changes the company logo.
- Set report **colors**:
The user can set the primary and secondary colors of the report through
a newly added widget allowing to pick a custom color.
When changing the **logo**, colors are automatically set to its most dominant
colors.
> A "Reset colors" button also triggers the color calculation.
- Set report **font**:
Changes the overall font of the report. Only Google Fonts are used
for enhanced compatibility.
- Company **tagline**, also called "header"
- **Footer**
- **Paper format**
- Report **preview**:
A mockup of a final report
Automatically updates when changing **layout**, **logo**, **colors** or **font**
Co-authored by: Julien Mougenot <jum@odoo.com>
closesodoo/odoo#33863
Signed-off-by: VincentSchippefilt <VincentSchippefilt@users.noreply.github.com>
Co-authored-by: Lucas Perais <lpe@odoo.com>
As the decimal_precision module was removed, allow code like
from odoo.addons import decimal_precision as dp
fields.Float(digits=dp.get_precision('Foo')
to keep working
Co-authored-by: Olivier Dony <odo@odoo.com>
The decimal precision feature makes sense to be an ORM feature, no need to be
in a specific module
Previous syntax was
from odoo.addons import decimal_precision as dp
fields.Float(digits=dp.get_precision('Foo'))
and now is:
fields.Float(digits='Foo')
Remove the possibility to have a callable method on the digits attribute (it
was only used for precision anyway) and directly retrieve the digits on the
decimal.precision model
Rename the method digits to get_digits to avoid confusion between the field
attribute when declaring a field and the method to retrieve the precision
Task id: 48198
* = account, purchase, website_sale_comparison, website_sale_wishlist
Raw Image & Mixin
=================
Previously, the main image from a product was stored resized. This caused
inconsistencies with the size of the images on ecommerce: the main image was
resized, but the extra images were not.
Now, we always store all raw images, thanks to the new mixin. This way, product
images that were created before the installation of ecommerce will be ready to
be used in ecommerce without any other change.
We also store the resized versions instead of computing them on the fly: this
will take more disk space, but significantly increase the speed of the
pages when displaying images, as well as reduce server CPU load.
Images on variants
==================
Previously, it was only possible to set the extra product images on a template,
and not on a variant.
Now, we allow extra images also on the variants, this gives more flexibility to
the user.
Both extra images on template and variant have been given a sequence field to be
able to sort them easily.
On /shop we always display the image of the template for performance reasons.
On /product we never display the image of the template, unless the variant has
no image, then the template image is the fallback.
Carousel
========
The carousel view has been cleaned and improved:
- remove the duplicate XML that was created with the product configurator fix
- use a single loop from a single source to get all the images, instead of
merging the different sources in the view with complex conditions
- improve the HTML structure, reduce the CSS with appropriate classes
Preview Image
=============
Both in website and backend: make better use of `preview_image` to display
a small image whenever possible, to reduce download size for the end user.
task-34045
PR: #30656
Co-authored-by: Parth Gajjar <pga@odoo.com>
Co-authored-by: Sébastien Theys <seb@odoo.com>
Task 1856935
Currently, if installing a module's demo data fails (because the
module was uninstalled and some were left over and can't be
reinstalled, or because they're broken, or…) the installation of the
module fails and further installations are skipped (?).
Since demo data are non-essential (though required to run tests),
rather than fail everything if they fail just roll them back and
notify the user.
This change generates a warning log *and* shows a notification popup
to the end user.
Julien Carion (juca)