In werkzeug 2.2.2, the following characters "$!'()*+,;" are now
considered as safe by url_quote. This makes the filename_secure test
fail with the hard coded expected string containing a single quote as
'%27'.
This commit adapt the filename_secure test in order to work with all
versions of werkzeug.
closesodoo/odoo#112298
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Before this commit, if you have some special char like a return line \n,
or \r the get_stream_from method will crash with exception:
```
File "/home/odoo/src/odoo/odoo/addons/web/controllers/binary.py", line 163, in content_image
return stream.get_response(**send_file_kwargs)
File "/home/odoo/src/odoo/odoo/odoo/http.py", line 578, in get_response
res = _send_file(self.path, **send_file_kwargs)
File "/home/odoo/src/odoo/odoo/odoo/tools/_vendor/send_file.py", line 156, in send_file
headers.set("Content-Disposition", value, **names)
File "/usr/local/lib/python3.9/dist-packages/werkzeug/datastructures.py", line 1218, in set
self._validate_value(_value)
File "/usr/local/lib/python3.9/dist-packages/werkzeug/datastructures.py", line 1182, in _validate_value
raise ValueError(
ValueError: Detected newline in header value. This is a potential security problem
```
Now we replace `\n` `\r` by `_` before to serve the stream to avoid this
security exception from a safe way.
We decided to not use secure_filename from werzkeug because we want to
continue the support of non ascii char.
closesodoo/odoo#111851
X-original-commit: 95584e71a898017a92112f89e8a94315dd9235ac
Signed-off-by: Jérémy Kersten <jke@odoo.com>
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
When a filename is given to the function binary_content it was shadowed
by _binary_ir_attachment_redirect_content if it is an ir.attachment.
To reproduce:
1. Start a brand new database in V14, install any app on which you can add an attachment (like Project or CRM)
2. Add a file as an attachment
3. Try to call the route "/web/content/<string:model>/<int:id>/<string:field>/<string:filename>"
closesodoo/odoo#87879
Solution: Use another variable to store the return value of _binary_ir_attachment_redirect_content and use it if the filename is not provided.
X-original-commit: 55e6097502da6169b85056b4769f4bf7fd230968
Signed-off-by: Wanderscheid Mathieu (mawa) <wama@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
When loading a page on an existing starting database, registry is not
fully loaded causing potential error when trying to access model
existing in database (views, menitem, ...) since model added in last
loaded module does not exist in registry.
Thus, executing browser js test may lead to errors when executed during
an update on a database with other modules installed.
HTTPCase should be executed post_install to ensure that registry is
fully loaded to avoid this problem.
Since HTTPCase are slower than other test, it is also a good idea to
execute them at the end, in order to prioritize fast fail.
With this commit, a warning is isued if such a test class is tagged to
run at install time.
While at it, remove deprecated at_install and post_install helpers and
remove the deprecated phantom_js alias.
closesodoo/odoo#39462
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
There are too many image sizes. Since they are stored resized this takes time to
generate when saving a new image, it's more rows on the attachment table, more
files on the disk, ...
64px is close enough to 128px that it can be removed without a big impact on
download size.
It will even reduce download and number of requests when both images are
displayed because now only one has to be downloaded and then benefit from cache.
The difference between the two is typically around 1.5kB which is negligible
these days, especially when the request overhead is around 0.5kB already, not
even taking into account other factors such as latency.
If a 64px image must absolutely be returned, it is still possible to pass the
size parameters to the image route. But the current guideline is to handle
resizing in the views when necessary.
Views
=====
- remove width and height attributes when existing CSS rules are overriding them
(eg. `.oe_kanban_avatar` in the right context)
- add CSS rules instead of width and height attributes when possible
- use `object-fit: cover;` where width and height are forced to avoid distortion
of non-square images
- for products, use `object-fit: contain;` instead, keep ratio but without crop
- add new CSS rules where the expected size was max 64px*64px before due to the
image size itself
- remove `img-fluid` where using size classes to avoid conflicting rules
task-2060865
closesodoo/odoo#36147
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
This branch is the combination of several optimizations in the ORM:
* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;
* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;
* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);
* make method `modified` take advantage of inverse fields to inverse
dependencies;
* filter records by evaluating a domain on records in Python;
* a computed field with `readonly=False` behaves like a normal field
with an onchange method;
* computed fields are computed in superuser mode by default.
Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.
closesodoo/odoo#35659
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
image_original => image_1920 (now resized to 1920)
image_big => image_1024
image_large => image_256
image_medium => image_128
image_small => image_64
image replaced by image_1920 (when writing) or by image_1024 (when displaying
what was previously the big size)
+ add new intermediate format:
image_512
PR: #34925
Before this commit, the No Change status wasn't properly set because
the `filehash` wasn't passed to `_binary_set_headers` and because
the status was not changed to 304 if it was already set to 200 earlier.
This commit fixes this issue and also prevents images to be processed
in `content_image` if the status is 304.
opw-2008426
closesodoo/odoo#34032
Signed-off-by: Christophe Simonis <chs@odoo.com>
* = website_partner, website_profile, website_sale, website_slides
Before
======
Since commit: 2e3848b394
The images that are resized have additional borders if the target ratio is
different than the image ratio. Those borders are transparent if the image
format supports it, and are white otherwise.
With that current solution, if the background where the image is displayed is
another color than white, it is looking really bad.
Moreover, most images are stored resized like this, so it is not even possible
to decide if it should have borders or not depending on the context, the
original image and ratio is forever lost.
It is also inconsistent because if the image is already smaller than the target
size then it doesn't include borders. In that case it keeps the original ratio
instead of the target ratio. So it isn't even guaranteed that the target
ratio is going to be respected.
After
=====
This commit will solve all of those problems by always keeping the ratio of the
original image.
This implies the views should be taking care of adding borders when necessary.
task-1958000
PR: #31811
This commit will add tests for the PR #30656 which added the following commit:
7daa85dd12
The tests in the current commit achieve 100% code coverage for:
- the `image.mixin`
- the overridden `image.mixin` methods in `product.product`
- the `_get_images` methods of ecommerce
- the size logic in method `content_image` (route `/web/image`)
PR: #31181
In 3d2ca8104e some change were introduced to the graph view.
But part of it had no sense when displaying data over 2 group by.
eg. if the first level has 2 groups (2018, 2019) and each second level 2
groups (Done,Cancelled), we could have:
2018/Done: 0
2018/Cancelled: 5
2019/Done: 8
2019/Cancelled: 9
this gives us two main groups [2018: 5] / [2019: 17], the code
remove the 2018 because [2018/Done] is 0 so we have something odd
with columns not in order, and in stacked bar chart some part of
a bar at the wrong offset
opw-1932517
closes#30529
Christophe Monniez