One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.
Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.
The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.
Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.
The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.
We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.
Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.
Note that this poc is inspired from the long term cache but not all
use case where applie yet.
Part-of: odoo/odoo#119813
This test will help making stats on routing_map generation performances
This will help to mesure the time for the main `None` routing map as
well as for website1, the idea being that it would be possible to
mutualize a part of this computation between routing maps.
closesodoo/odoo#120591
X-original-commit: 6646fa7e2ef64e5f060bf38b0d85bf1188d79e4f
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
The file extension detection previously used to determine whether a
filename required an extension was not very smart and in fact only
checked for a dot in the filename.
`mimetypes.guess_type` is now used on the filename to better determine
whether the filename still needs a file extension or not.
This commit is a follow-up to: https://github.com/odoo/odoo/pull/90614
Which aimed to fix the same issue.
TaskId-2826061
closesodoo/odoo#91254
X-original-commit: bf904a7f98ba5bcc0a8909cca3241e2965c4464c
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Signed-off-by: William Braeckman (wbr) <wbr@odoo.com>
Avoid to base64 encode, then decode to process assets and images for a ~25% speed improvement.
Change image processing tool to work on images, rather than base64 encoded strings.
Performance is ~25% faster on assets & images:
/web/assets/...frontend.min.css: 13ms to 7ms, base64 enc/dec: 2 -> 0
/web/image/XML_ID: 10ms to 8ms, base64 enc/dec: 3 -> 0
/web/image/res.users/2/avatar_128: 40ms to 20ms, base64 enc/dec: 6 -> 2
closesodoo/odoo#82851
Related: odoo/enterprise#23537
Signed-off-by: Fabien Pinckaers <fp@odoo.com>
This branch is the combination of several optimizations in the ORM:
* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;
* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;
* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);
* make method `modified` take advantage of inverse fields to inverse
dependencies;
* filter records by evaluating a domain on records in Python;
* a computed field with `readonly=False` behaves like a normal field
with an onchange method;
* computed fields are computed in superuser mode by default.
Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.
closesodoo/odoo#35659
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
The following models are already using big images, or they might need big images
in the future:
- partner
- hr employee
- shop category
- lunch product
- gamification badge and karma rank
PR: #34925
image_original => image_1920 (now resized to 1920)
image_big => image_1024
image_large => image_256
image_medium => image_128
image_small => image_64
image replaced by image_1920 (when writing) or by image_1024 (when displaying
what was previously the big size)
+ add new intermediate format:
image_512
PR: #34925
This commit removes the field `datas_fname` from `ir.attachment` as
it was unnecessary and most of the time the duplicate of `name` or
`url`.
Task #1909865closesodoo/odoo#32976
Signed-off-by: Martin Geubelle (mge) <mge@openerp.com>
* = tools, base, web, website_profile, website_slides
Before this commit it was impossible to pass those parameters in the URL because
they were received as string but expected as int.
The opportunity is also taken to properly use named parameters when calling
`image_process`.
closesodoo/odoo#33506
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
* = hr, im_livechat, mail, payment, purchase, web, web_editor, web_unsplash,
website_profile, website_slides
Since the merge of all image tools into one function, the intermediary functions
are not needed anymore.
task-1958000
PR: #31811
Add support for a 256*256 image, to be used in the following commit.
Define sizes in named variables instead of being hard-coded in several places in
the code.
Allow parameters `preserve_aspect_ratio` and `upper_limit` to be passed from
the different helper methods.
Factorize the code inside `image_resize_images` to make it easier to read.
Add new function to compute whether the size of an image is above a given size.
task-34045
PR: #30656
*: tools, web, website, website_forum, mail, im_livechat
This commit refactors ir_http to make it more readable
and flexible.
Move the resize function of web/image to odoo.tools
Co-authored-by: XavierDo <xdo@odoo.com>
Co-authored-by: Antony Lesuisse <al@openerp.com>
closes: #28563
task: #1908896
When an attachment is requested with an access_token, a server error
would occur in some situation.
With this changeset when an access token is specified:
- returns a 404 error if the attachment does not exist
- returns a 403 error if the attachment has no access_token
Without this change, the added test would fail with:
```
status = test_access(access_token='Secret')
self.assertEqual(status, 403,
"no access if access token for attachment without access token")
| if not consteq(obj.access_token, access_token):
| TypeError: unsupported operand types(s) or combination of types: 'bool' and 'str'
status = test_access(access_token='Secret')
self.assertEqual(status, 404,
"no access with access token for deleted attachment")
| if not consteq(obj.access_token, access_token):
| odoo.exceptions.MissingError: ('Record does not exist or has been deleted.', None)
```
close#26647
opw-1884419
closes#27275
Co-authored-by: Wolfgang Taferner <wtaferner@users.noreply.github.com>
Xavier-Do