Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closes odoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
248 lines
10 KiB
Python
248 lines
10 KiB
Python
import logging
|
|
import werkzeug.http
|
|
from datetime import datetime
|
|
from mimetypes import guess_extension
|
|
|
|
from odoo import models
|
|
from odoo.exceptions import MissingError, UserError
|
|
from odoo.http import Stream, request
|
|
from odoo.tools import file_open, replace_exceptions
|
|
from odoo.tools.image import image_process, image_guess_size_from_field_name
|
|
from odoo.tools.mimetypes import guess_mimetype, get_extension
|
|
|
|
|
|
DEFAULT_PLACEHOLDER_PATH = 'web/static/img/placeholder.png'
|
|
_logger = logging.getLogger(__name__)
|
|
|
|
|
|
class IrBinary(models.AbstractModel):
|
|
_name = 'ir.binary'
|
|
_description = "File streaming helper model for controllers"
|
|
|
|
def _find_record(
|
|
self, xmlid=None, res_model='ir.attachment', res_id=None,
|
|
access_token=None,
|
|
):
|
|
"""
|
|
Find and return a record either using an xmlid either a model+id
|
|
pair. This method is an helper for the ``/web/content`` and
|
|
``/web/image`` controllers and should not be used in other
|
|
contextes.
|
|
|
|
:param Optional[str] xmlid: xmlid of the record
|
|
:param Optional[str] res_model: model of the record,
|
|
ir.attachment by default.
|
|
:param Optional[id] res_id: id of the record
|
|
:param Optional[str] access_token: access token to use instead
|
|
of the access rights and access rules.
|
|
:returns: single record
|
|
:raises MissingError: when no record was found.
|
|
"""
|
|
record = None
|
|
if xmlid:
|
|
record = self.env.ref(xmlid, False)
|
|
elif res_id is not None and res_model in self.env:
|
|
record = self.env[res_model].browse(res_id).exists()
|
|
if not record:
|
|
raise MissingError(f"No record found for xmlid={xmlid}, res_model={res_model}, id={res_id}")
|
|
|
|
if record._name == 'ir.attachment':
|
|
record = record.validate_access(access_token)
|
|
|
|
return record
|
|
|
|
def _record_to_stream(self, record, field_name):
|
|
"""
|
|
Low level method responsible for the actual conversion from a
|
|
model record to a stream. This method is an extensible hook for
|
|
other modules. It is not meant to be directly called from
|
|
outside or the ir.binary model.
|
|
|
|
:param record: the record where to load the data from.
|
|
:param str field_name: the binary field where to load the data
|
|
from.
|
|
:rtype: odoo.http.Stream
|
|
"""
|
|
if record._name == 'ir.attachment' and field_name in ('raw', 'datas', 'db_datas'):
|
|
return Stream.from_attachment(record)
|
|
|
|
field_def = record._fields[field_name]
|
|
|
|
# fields.Binary(attachment=False) or compute/related
|
|
if not field_def.attachment or field_def.compute or field_def.related:
|
|
return Stream.from_binary_field(record, field_name)
|
|
|
|
# fields.Binary(attachment=True)
|
|
field_attachment = self.env['ir.attachment'].sudo().search(
|
|
domain=[('res_model', '=', record._name),
|
|
('res_id', '=', record.id),
|
|
('res_field', '=', field_name)],
|
|
limit=1)
|
|
if not field_attachment:
|
|
raise MissingError("The related attachment does not exist.")
|
|
return Stream.from_attachment(field_attachment)
|
|
|
|
def _get_stream_from(
|
|
self, record, field_name='raw', filename=None, filename_field='name',
|
|
mimetype=None, default_mimetype='application/octet-stream',
|
|
):
|
|
"""
|
|
Create a :class:odoo.http.Stream: from a record's binary field.
|
|
|
|
:param record: the record where to load the data from.
|
|
:param str field_name: the binary field where to load the data
|
|
from.
|
|
:param Optional[str] filename: when the stream is downloaded by
|
|
a browser, what filename it should have on disk. By default
|
|
it is ``{model}-{id}-{field}.{extension}``, the extension is
|
|
determined thanks to mimetype.
|
|
:param Optional[str] filename_field: like ``filename`` but use
|
|
one of the record's char field as filename.
|
|
:param Optional[str] mimetype: the data mimetype to use instead
|
|
of the stored one (attachment) or the one determined by
|
|
magic.
|
|
:param str default_mimetype: the mimetype to use when the
|
|
mimetype couldn't be determined. By default it is
|
|
``application/octet-stream``.
|
|
:rtype: odoo.http.Stream
|
|
"""
|
|
with replace_exceptions(ValueError, by=UserError(f'Expected singleton: {record}')):
|
|
record.ensure_one()
|
|
|
|
try:
|
|
field_def = record._fields[field_name]
|
|
except KeyError:
|
|
raise UserError(f"Record has no field {field_name!r}.")
|
|
if field_def.type != 'binary':
|
|
raise UserError(
|
|
f"Field {field_def!r} is type {field_def.type!r} but "
|
|
f"it is only possible to stream Binary or Image fields."
|
|
)
|
|
|
|
stream = self._record_to_stream(record, field_name)
|
|
|
|
if stream.type in ('data', 'path'):
|
|
if mimetype:
|
|
stream.mimetype = mimetype
|
|
elif not stream.mimetype:
|
|
if stream.type == 'data':
|
|
head = stream.data[:1024]
|
|
else:
|
|
with open(stream.path, 'rb') as file:
|
|
head = file.read(1024)
|
|
stream.mimetype = guess_mimetype(head, default=default_mimetype)
|
|
|
|
if filename:
|
|
stream.download_name = filename
|
|
elif filename_field in record:
|
|
stream.download_name = record[filename_field]
|
|
if not stream.download_name:
|
|
stream.download_name = f'{record._table}-{record.id}-{field_name}'
|
|
|
|
if (not get_extension(stream.download_name)
|
|
and stream.mimetype != 'application/octet-stream'):
|
|
stream.download_name += guess_extension(stream.mimetype) or ''
|
|
|
|
return stream
|
|
|
|
def _get_image_stream_from(
|
|
self, record, field_name='raw', filename=None, filename_field='name',
|
|
mimetype=None, default_mimetype='image/png', placeholder=None,
|
|
width=0, height=0, crop=False, quality=0,
|
|
):
|
|
"""
|
|
Create a :class:odoo.http.Stream: from a record's binary field,
|
|
equivalent of :meth:`~get_stream_from` but for images.
|
|
|
|
In case the record does not exist or is not accessible, the
|
|
alternative ``placeholder`` path is used instead. If not set,
|
|
a path is determined via
|
|
:meth:`~odoo.models.BaseModel._get_placeholder_filename` which
|
|
ultimately fallbacks on ``web/static/img/placeholder.png``.
|
|
|
|
In case the arguments ``width``, ``height``, ``crop`` or
|
|
``quality`` are given, the image will be post-processed and the
|
|
ETags (the unique cache http header) will be updated
|
|
accordingly. See also :func:`odoo.tools.image.image_process`.
|
|
|
|
:param record: the record where to load the data from.
|
|
:param str field_name: the binary field where to load the data
|
|
from.
|
|
:param Optional[str] filename: when the stream is downloaded by
|
|
a browser, what filename it should have on disk. By default
|
|
it is ``{table}-{id}-{field}.{extension}``, the extension is
|
|
determined thanks to mimetype.
|
|
:param Optional[str] filename_field: like ``filename`` but use
|
|
one of the record's char field as filename.
|
|
:param Optional[str] mimetype: the data mimetype to use instead
|
|
of the stored one (attachment) or the one determined by
|
|
magic.
|
|
:param str default_mimetype: the mimetype to use when the
|
|
mimetype couldn't be determined. By default it is
|
|
``image/png``.
|
|
:param Optional[pathlike] placeholder: in case the image is not
|
|
found or unaccessible, the path of an image to use instead.
|
|
By default the record ``_get_placeholder_filename`` on the
|
|
requested field or ``web/static/img/placeholder.png``.
|
|
:param int width: if not zero, the width of the resized image.
|
|
:param int height: if not zero, the height of the resized image.
|
|
:param bool crop: if true, crop the image instead of rezising
|
|
it.
|
|
:param int quality: if not zero, the quality of the resized
|
|
image.
|
|
|
|
"""
|
|
try:
|
|
stream = self._get_stream_from(
|
|
record, field_name, filename, filename_field, mimetype,
|
|
default_mimetype
|
|
)
|
|
except UserError:
|
|
if request.params.get('download'):
|
|
raise
|
|
if not placeholder:
|
|
placeholder = record._get_placeholder_filename(field_name)
|
|
stream = self._get_placeholder_stream(placeholder)
|
|
|
|
if stream.type == 'url':
|
|
return stream # Rezising an external URL is not supported
|
|
|
|
if (width, height) == (0, 0):
|
|
width, height = image_guess_size_from_field_name(field_name)
|
|
stream.etag += f'-{width}x{height}-crop={crop}-quality={quality}'
|
|
|
|
if isinstance(stream.last_modified, (int, float)):
|
|
stream.last_modified = datetime.utcfromtimestamp(stream.last_modified)
|
|
modified = werkzeug.http.is_resource_modified(
|
|
request.httprequest.environ,
|
|
etag=stream.etag,
|
|
last_modified=stream.last_modified
|
|
)
|
|
|
|
if modified and (width or height or crop):
|
|
if stream.type == 'path':
|
|
with open(stream.path, 'rb') as file:
|
|
stream.type = 'data'
|
|
stream.path = None
|
|
stream.data = file.read()
|
|
stream.data = image_process(
|
|
stream.data,
|
|
size=(width, height),
|
|
crop=crop,
|
|
quality=quality,
|
|
)
|
|
stream.size = len(stream.data)
|
|
|
|
return stream
|
|
|
|
def _get_placeholder_stream(self, path=None):
|
|
if not path:
|
|
path = DEFAULT_PLACEHOLDER_PATH
|
|
return Stream.from_path(path, filter_ext=('.png', '.jpg'))
|
|
|
|
def _placeholder(self, path=False):
|
|
if not path:
|
|
path = DEFAULT_PLACEHOLDER_PATH
|
|
with file_open(path, 'rb', filter_ext=('.png', '.jpg')) as file:
|
|
return file.read()
|