* clean and improve docstrings in orm
* fix typos found with codespell
* rely on the Environment class docstring instead of doc content (and
therefore move part of the doc inside the class docstring)
closesodoo/odoo#102969
X-original-commit: 8250cd4b210005d223a4cdb8afa4014425ca6fa3
Related: odoo/documentation#2803
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Both implementations of `set_cookie` in `http.py` and the one of
`setCookie` in `cookie_utils.js` are slightly different.
This commit unifies both Python implementations and replicates the same
behavior in JS:
- default the cookie type to "required"
- if setting that type is not allowed, delete that cookie if it was
previously set
task-2800976
closesodoo/odoo#102380
X-original-commit: fa8fdab2894607f1f89cdd195f62ec37af7b0342
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
*: im_livechat, survey, utm, website_crm_iap_reveal, website_forum,
website_livechat, website_sale, website_sale_comparison
Before this commit all cookies were considered essential.
This commit makes some of them optional. It also makes it possible for
the website visitor to only accept the essential cookies.
task-2800976
X-original-commit: 9a8a9463289a7446e9be0ef62ff895feb37a4de4
Part-of: odoo/odoo#101845
Co-authored-by: Benoit Socias <bso@odoo.com>
This commit avoid to have a dict by reference that will be global.
Now get_default_session return a new dict each time for the context key.
From this way the session.context['lang'] is not shared between several
users on the same worker.
To reproduce the bug, restart the server with 2 workers, make request in
lang A on these 2 workers. DEFAULT_SESSION['context']['lang'] now is set
to this lang A.
Now, make request to an url without lang in path and without cookies and
withtout session, you should be redirected to lang B (preferred lang
from the request header) but you will be redirect to lang A due to the
dict session.context that is shared for the worker...
When we initialize the new Session, we get the wrong lang A as value for
context.lang, so we don't recompute the expected lang for the end user.
X-original-commit: 62179de74862210fe2a055d15b367b1850c24263
fwd-port of #100102closesodoo/odoo#100910
X-original-commit: 42e46b2d89dde276f796b980f29e33cc216e7cb2
Signed-off-by: Jérémy Kersten <jke@odoo.com>
XML files are now declared in python module manifests. During the qweb
't-call-asset' directive, assetbundle will fetch the declared xml files,
apply the inheritance (t-inherit) and create a javascript service (for
eg: 'web.assets_backend.bundle.xml') which is added at the end of the
*.js mimifier file.
When the debug mode is activated, comments are added in the template
indicating which file the template comes from as well as the
inheritances applied to it.
****
JavaScript:
assets.js (module @web/core/assets) takes care of loading libraries,
javascripts and styles.
`loadJS(url)` (loads the javascript and returns a resolved promise when
the templates are also loaded via the '*.bundle.xml' service)
`loadCSS(url)` (loads the style a resolved promise when the file is
loaded)
`loadXML(xml, app=assets.defaultApp)` (load template into
application/owl, used by the `*.bundle.xml` services)
`getBundle(bundleName)` (get the bundle descriptor)
`loadBundle(desc)` (load the files and bundle from a descriptor)
templates (XML element content all owl templates)
A new `ready(serviceName)` method on boot.js lets you know when a
service is loaded are the require.
The xmlDependencies attribute no longer exists.
Python:
The xmls taken into account by assetbundle.py, applying `t-inherit`
inheritances and adding an `name_of_the_bundle.bundle.xml` service in
the generated JavaScript file.
****
Every manifest changes is into the next commit, except 'web_tour' in
this current commit as example.
Part-of: odoo/odoo#95500
When a file is immutable the different network layers can cache it.
Services receiving this header should never invalidate these files.
Part-of: odoo/odoo#95500
*: hr_presence, web_editor.
This commit is part of the websocket integration in Odoo.
It focuses on adapting the bus to support websockets:
- last notification id is now kept on the server
- channel list is built by overriding the `_build_bus_channel_list`
method of the `ir_websocket` model instead of overriding the `_poll`
method of the bus controller.
- The bus presence was updated during polls, since there is no more poll,
bus presence update will be the responsability of the client.
- The `/websocket/peek_notifications`, `/websocket/update_bus_presence`
routes will be available so that odoo sh can access notifications/update presence
from http requests.
- /longpolling routes are now prefixed with /bus thus won't be redirected to the
gevent worker anymore except for `/longpolling/health` which is the
health check route of the gevent server.
Since websocket now handle incoming messages, a way to manage authentication
have been introduced :
- The session is retrieved from the HTTP handshake.
- When a websocket message comes/leaves the session is retrieved
on the file system so that we're sure it still exists and that
it is up to date.
- The session is checked
- If no session is found on the file system or `check_session`
fails, the websocket connection is closed with the `SESSION_EXPIRED`
close code (which is a custom close code: 4001).
- Note that websocket connections are closed every `KEEP_ALIVE_TIMEOUT`
seconds to ensure no websocket connection will stay open if the user
clears its cookies.
- Note that a wsrequest object is available when processing incoming
messages. It is similar to the http request and contains various
useful informations (session, env, ...).
Part-of: odoo/odoo#75510
This commit is the first commit of the websocket integration in Odoo.
It focuses on the implementation of the websocket protocol as per RFC6455.
The implementation is tested thanks to the autobahn test suite.
A config parameter is available to customize the websocket connection:
- websocket_keep_alive_timeout (default 600): Integer specifying how
many seconds a websocket connection should be kept alive
Part-of: odoo/odoo#75510
It allows to always have a OdooResponse Object and don't allow redirect
to external except when you allow it explicitly with local=False.
Always return to a local url:
/website/add
/slides/slide/<model("slide.slide"):slide>
/microsoft_outlook/confirm
Allow previously external redirect without reason, now blocked
/website/lang/<lang> -> open redirect
Allow external redirect for good reason and url is controlled by code.
/social_facebook/redirect_to_profile/
PS: HTTP Code 303 is a better default for generic redirects. It's not
historically the default for werkzeug.utils, but it is what we want in
general. Contrary to 302, there is no browser-dependent behavior, and
no risk of asking the user whether they want to accept the redirect if
the original method wasn't GET. It's always a non-permanent GET on the
target location.
closesodoo/odoo#95019
X-original-commit: 77f8d9c5d96a9274785ffc2ad83b95ea157d26ad
Related: odoo/enterprise#29000
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Jérémy Kersten <jke@odoo.com>
Leftovers of previous fixes of this issue.
closesodoo/odoo#95162
X-original-commit: dffc184862018670008ff4da839fc520a72bddca
Signed-off-by: Christophe Simonis <chs@odoo.com>
Create an empty database and start 2 http workers, go on the web app
menu and install website (don't install website via -i). Once website is
installed, you are redirected on `/website/configurator` but the route
does not exist and it fails with a 500 internal server error.
The problem is due to an invalid registry manipulation introduced in the
saas-15.3's httpocalypse. When new modules are installed the registry
must be reloaded in all workers. The function that determine if the
registry must be reloaded and reloads it is `check_signaling`.
When the current registry is up-to-date, it is returned as-is by
`check_signaling`. When it is outdated, `check_signaling` creates and
returns a new fresh registry; it does not nor discard nor change
in-place the previous (outdated) registry, it is up to the callee to
discard the previous registry itself.
closesodoo/odoo#93579
X-original-commit: 23bdcc3fd31e6908ee6737cb74787aefa5d057a5
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
This commit tries to avoid a traceback when no postgresql connexion is available.
closesodoo/odoo#92195
X-original-commit: 8199daf4bcba76277fed39554863efa3f1585b20
Signed-off-by: Josse Colpaert <jco@odoo.com>
When started with the CLI option --dev=werkzeug, errors in controllers
are caught by a friendly web debugger. This debugger should not be
started in case of error in a JSON-RPC controller.
closesodoo/odoo#90420
Task: 2837457
X-original-commit: 711a29e76f55e87d73dad89ddbef889dd5c7af10
Signed-off-by: Julien Castiaux <juc@odoo.com>
With this work we relax the http controller so that it accepts all
requests, including `Content-Type: application/json`. We also enrich
the framework with two new helper methods dedicated to serialaze json
requests and responses:
- `request.get_json_body()`, loads the json content from the request's
body and returns the corresponding python object (usually a dict).
- `request.make_json_response(data)`, dumps `data` to json and makes an
http response out of it.
closesodoo/odoo#86300
Task: 2779837
Signed-off-by: Julien Castiaux <juc@odoo.com>
Every request comes with a session, a dictionary that is persisted on
the filesystem and that saves various information such as the user
cart on the ecommerce.
When a user simply visits the website, a default session is created and
saved on disk, this bloats the filestore with many sessions. Creating
the session on-the-fly is cheaper than loading it from the filesystem.
With this work the default session is not saved on disk anymore unless
explicitly asked via `session.touch()`.
An exception to the statement "creating the session on-the-fly is
cheaper" is geoip, the ip geolocalization is not cheap. In this work,
geoip have been moved from http_routing/request.session.geoip to a
lazy property core/request.geoip. When requested the info is persisted
on the session. Like other keys from the default session, geoip will not
be persisted unless there is non-default stuff in the session.
Because the CSRF-TOKEN is based on the session-id, it is important the
session-id stays the same across multiples requests even when the
session is not persisted on disk. Even when a session is not persisted
on disk, the session-id cookie is still set so that the next session
created on-the-fly uses the same session-id.
Technical note regarding the session, it has been decided to drop the
session-snapshot protocol and to reintroduce a "modified" flag. It has
been decided not to use werkzeug's session (which natively comes with a
"modified" flag) and to keep our own session object. We decided to
extend MutableMapping instead of dict; using MutableMapping we only
have to override __setitem__ and __detitem__; using dict we would had to
override update()/pop()/... too.
Task: 2789035
Part-of: odoo/odoo#86015
Steps to reproduce:
1) start from a clean database (http_routing should not be installed).
2) go to the app menu and install project (don't install via `-i`!!).
3) traceback: `request` has not attribute `is_frontend` while rendering
a template.
The `is_frontend` attribute on request is set by the http_routing
module. At the moment the "install now" button is clicked, http_routing
was not installed so the `is_frontend` attribute was not set.
FF to the end of the installation: the registry is reloaded to include
the modules that have been installed, http_routing among them.
We are in a tricky situation: (1) there is a request, (2) http_routing
is installed and (3) the `is_frontend` attribute is missing from the
request.
This situation is illegal, when http_routing is installed, the
`is_frontend` attribut should always be set. In this work we reset the
missing attributes using sensitive default values via post-init hooks.
closesodoo/odoo#87684
Signed-off-by: Julien Castiaux <juc@odoo.com>
With the iot-box it is possible to download modules from an odoo server
into the iot-box. Those odoo modules are `exec`-like and exposed on the
box. With those modules come some controllers. Because they were
`exec`-like, they lack a python module name which makes
`_generate_routing_rules` to reject them.
With this fix, we authorise to declare controllers outside of odoo
addons. Those controllers cannot be extended, nor can they override
other controllers.
closesodoo/odoo#86275
Signed-off-by: Julien Castiaux <juc@odoo.com>
* http.py:docstring of odoo.http.route:19: WARNING: Unexpected
indentation.
* http.py:docstring of odoo.http.route:20: WARNING: Block quote ends
without a blank line; unexpected unindent.
* http.py:docstring of odoo.http.route:30: WARNING: undefined label:
csrf
-> No section in the doc exists with reference csrf
* Add docstring for the default_lang method.
closesodoo/odoo#85505
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
This commit is the 14th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
* `request.uid = x` => `request.update_env(user=x)`.
* `request.context = x` => `request.update_env(context=x)`.
* `request.context = dict(request.context, x=y)`
=> `request.update_context(x=y)`.
* `request.cr = None` => `request.cr.close()`.
* `http.mono_db()` => `request.db`.
* `http.dispatch_rpc()` => `service.dispatch_rpc()`.
* `@service.model.check` => `service.model.retrying()`.
* `request.endpoint`
=> `env['ir.http']._match(request.httprequest.path)[0].endpoint`.
* `request.routing_iteration `=> `removed`.
* `request.jsonrequest` => `request.dispatcher.jsonrequest`.
Note that `request.params` is now set much later in the process. If you
are in a situation where you values from the query string or the
http body you can use `request.get_http_params()`.
Note that using the new `request.future_response`, it is possible to
add headers and cookies on the response object before the response
object is initialized. Please note that headers/cookies saved on
the future response will NOT be injected in case of error.
PR: odoo#78857
Task: 2571224
This commit is the 12th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
The web module is twofold, on one side there are many controllers: /,
/web, /web/login, /web/database/selector, /web/dataset/call_kw, etc, on
the other side there is `session_info`: the method responsible to create
the web client's environ.
This module is kinda an exception as it is (with base) a server wide
module. In the case of the HTTP framework, it means that the controllers
of web are always accessible, i.e. going to / or /web/login will never
return a 404 Not Found even if the user is not connected to a database.
This is both a blessing and a curse. It is a blessing because the
controllers are always accessible it means that a new users can freely
access those routes. It is a curse because *any* user can access them,
even user who don't have a session yet thus who are not connected to a
database yet. From a developer standpoint, we have to put extra care to
correct serve users with and without a database. An example is the
/web/login route, the login/password pair is stored in a database,
without database it is impossible to validate a user login but users can
still access this route without db.
To solve this problem, there is the `ensure_db` function. This function
attempts to find a database using various sources (?db= query-string,
session db, mono db) and to save it on the user session. In case no db
is found, the user is redirected to the database selector. In a way,
this function grants a database to the user in a seamingly experience.
In a way, this function brings a welcome differentiation between
`auth='none'` with a database and `auth='none'` without a database. Such
differentiation only matters for the server wide modules as "regular"
module controllers are only accessible via the ir.http routing map, i.e.
it is not possible to declare a nodb controller outside of server wide
modules.
An important changement is the `session.authenticate` method, before it
was possible to call the method when the cursor was not yet initialized,
authenticate would open a cursor against the given database, setup a
registry and an environment and ultimately save everything on the
current request. Because the cursor is now greedily created, it is no
more possible to update the request environment when authenticating on
another database.
PR: odoo#78857
Task: 2571224
This commit is the 11th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals. See also [REF] core: HTTPocalypse (9) ORM initialization.
Where `request._serve_db` is reponsible for initializing the ORM,
`request._serve_ir_http` is responsible to do the match -> authenticate
-> pre dispatch -> dispatch -> post dispatch sequence. This sequence is
very much similar to `_serve_nodb` with the notable addition of the
authenticate step.
Where `_serve_nodb` was working in a db-free environment, calling the
methods of (Http|Json)Dispatcher right away, `_serve_ir_http` delegates
to the `ir.http` abstract model which in turn use the dispatchers.
The ir.http model is crucial for several modules, web, portal and
website on top. They override the various methods to add advenced
capabilities to the http framework.
An important changement to the ir.http model in this work is regarding
the `_dispatch` method. Before this work it was that method that was
responsible of doing the sequence match -> pre dispatch -> ... . It was
decided to move the responsibility to http.py in order to prevent
modules from taking over the entire framework. One such take-over was
done in the `http_routing` addon and resulted in arguably very poor and
fragile code.
All modules must now carefully override the correct method. You need to
do smart stuff on the URL prior to matching an endpoint? You should
override `_match`. You need to save a special option from the query-
strings to the session before calling the controller? Override
`_pre_dispatch`.
PR: odoo#78857
Task: 2571224
This commit is the 10th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
One of the objectives of this comprehensive refactor was to simplify the
error reporting, to show developers shorter and more informative
traceback upon errors.
We argue that the current error reporting is too much bloated:
1. There is a "dual" traceback, one traceback for the frames more recent
than `/base:ir.http_dispatch` (`@route` wrappers + controller) and a
second traceback for the former frames (http framework + web server).
2. We trace too far back, we argue that it is useless to trace back to
the thread/worker creation, that we should only trace back until the
WSGI entry point.
3. There is too much noise, many frames are either implementation
details or useless information. e.g. middlewares, `_dispatch`
overrides, wrappers, "checks".
To solve the above problem we did the following:
1. All errors bubble up to `request.__call__` where it is logged, any
wrapper that catch an error must log it or re-raise it. By logging
exceptions in `__call__`, we ensure the traceback at that function.
2. When one raises an HTTP error, e.g. 404/NotFound, the object is at a
same time an exception and a response. We don't log it and use the
response right way.
3. When one raises an exception, e.g. ValueError, the exception is sent
to `handle_error`, its job is to return the HTTP error corresponding
to the given exception, e.g. 500/InternalError in case of ValueError.
The given exception's `__traceback__` and `__clause__` are not
modified, we log the real exception but use the http error as
response.
Minimalistic example showing a traceback before and after this work:
https://gist.github.com/Julien00859/d6a48d523cac42d0df2e3136f5ba3e54
PR: odoo#78857
Task: 2571224
This commit is the 9th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals. See also [REF] core: HTTPocalypse (11) ir.http base model.
This is a two-part commit with "(11) ir.http base model". In this commit
we focus on the initialization of the various ORM objects, namely: the
registry, the cursor, the environment, the user and the context. In the
other n°11 commit we focus on the ir.http model and its relation to the
current http.py module.
One of the objectives of this comprehensive refactor was to ease the
cognitive complexity of the http framework, in other words to make it
simplier. One of the problem identified quite early during the
preparation of this work is the way the various ORM objects are
initialized, modified and cleaned during the request lifetime.
Before this work, all the ORM internals were lazily initialized via
properties. It is the first time one uses `request.cr` that a cursor is
opened to `request.db` and stored on `request._cr`. It is the first time
one uses `request.env` that an environment is create with the current
`request.user` and `request.context`. Upon user or context modification,
the current environment is discarded, the next usage of `request.env`
will create yet another environment on the fly using the modified user
and/or context.
Using this model, no ressource is initialized if not necessary. It is
possible for nodb-compatible endpoint to be served via the db-compatible
router and ir.http without ever opening a cursor to the database.
But this model is harder to reason about and ultimately to maintain.
In this work we propose to drop the lazy approach for a greedy one. In
this work the first steps of `_serve_db`, the db-compatible counter-part
of `_serve_nodb`, are dedicated to setup a registry, open a cursor to
the database and create an environment using the session's user and
context. In this work, when one wants to change the environ's user or
context, he must call `request.update_env` or `request.update_context`,
both method will recreate the environment *now* with the given values.
The downside of this approach is that resources are always allocated
even when it is not necessary. We argue that, in general, the
controllers that do not use the ORM are rare thus it is rare we allocate
unecessary resources. We also argue that the cognitive benefits are more
than welcome and that the new APIs will help at writing more robust
applications.
PR: odoo#78857
Task: 2571224
This commit is the 8th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
PR: odoo#78857
Task: 2571224
See also: [REF] core: HTTPocalypse (6) serve db-free routes
This commit is the 7th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
PR: odoo#78857
Task: 2571224
See also: [REF] core: HTTPocalypse (6) serve db-free routes
This commit is the 6th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
There are some routes that must work even when the user is not connected
to a database yet. Two obvious examples are the index (/) and the
database selector (/web/database/selector). Accessing those routes
should be possible even when the user is not connected to a database yet,
the `nodb_routing_map` and the `_serve_nodb` method fulfill this need.
Along with the introduction of the `_serve_nodb` method, we also
introduce the match -> pre dispatch -> dispatch -> post dispatch
sequence. `match` matches an endpoint using the http path,
`pre_dispatch` prepares the system, `dispatch` is the actual endpoint
call, `post_dispatch` cleanups the system and add some http headers to
the http response.
At the beginning, when an endpoint is matched, we retrieve the `routing`
dictionnary that is attached to the endpoint method. This dictionnary
contains the information set by the `@route` decorator, among other
values is the routing type `http` or `json`. This routing type is used
to specialize the request via one of the dispatchers: HttpDispatcher for
http and JsonDispatcher for json. Via the dispatchers, it is possible to
add custom behavior in the form of (pre_|post_)dispatch override. One
example of such custom behavior is the way the request body will be
loaded: urllib.parse.parse_qs (http) vs json.loads (json).
Before this work, it was not possible for multiple dispatchers to be
compatible with a same request mimetype. Because `JsonRequest` was
implementing the support for `application/json` (and like) mimetypes,
all requests having a body's mimetype `application/json` were considered
JSON-RPC2 (the protocol implemented by `JsonRequest`) even if the
request was actually bare json data. With the introduction of the
`is_compatible_with` classmethod, multiple dispatchers can be compatible
with a same mimetype and it is up to the `@route(type=...)` param to
define which dispatcher should be used with this route.
PR: odoo#78857
Task: 2571224
This commit is the 5th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
A session is a file that contains user data that must be preserved
across several requests. It is a small (=at most a few kb) file that is
stored on the filesystem in the session store. Each filename is a random
sha-1 string, the same sha-1 is saved in the request's cookies. The
browser has no access to the session, it only possesses the random sha-1
cookie which the server use to retrieve the session upon each request.
It is in the session that the database that the user is connected to is
stored. The first time the user accesses the server the database is
determined using the HTTP Host header of the request plus the list of
available databases and is saved into the user's session. Later requests
merely ensure the database stored in the session is still enabled.
The session "feeling" is the same as before the httpocalypse. The
session is still a DotDict-like structure with methods such as
`authenticate` and `logout`. The difference is the implementation, in
master we were using the werkzeug session, with this work we decided to
get rid of those off-the-shelf sessions to implement our own object.
The major difference between the two implementations is the way we
determine when a session is "dirty", when the session should be written
back on the filesystem. In werkzeug it is a `modified` flag attribute
that is set as soon as `__setitem__` is called, in our implemention we
keep an immutable copy of the original session, at the end the of
request lifecycle we compare the live session to the copy to determine
if it was changed. The former was setting the `modified` flag even when
one was setting the same value as existing in the session. The later
approach is not affected by this limitation.
PR: odoo#78857
Task: 2571224
This commit is the 4th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
Complete refactor of the "registry" of controllers. The `ControllerType`
metaclass have been replaced by an abstract class with a py3.7
`__init_subclass__`. The `Endpoint` class is gone too, replaced by a
clever usage of `functools.partial`. This refactor is "pure", it doesn't
add any new feature, it only merely adds a few warnings.
The four `@route`, `Controller`, `_generate_routing_rule`, `routing_map`
work as follow:
1. A reference to each immediate child class of `Controller` (not grand-
children) is registered in a global list indexed by module (thus a
dictionnary) everytime the server starts. Remember that every first-
child (not grand-children) of `Controller` is the primary controller,
the one that can later be extended by other controllers (the grand-
children) in other modules. Remember that it is possible to get each
class's children via the `__subclasses__` dunder method.
2. Every controller method that is decorated with `@route` is granted an
attribute: `original_routing`, a dictionnary containing the `@route`
arguments. When a controller method has the `original_routing`
attribute, this method is called an `endpoint`.
3. `_generate_routing_rules` receives the list of installed module
names, this list is topologicaly sorted according to the modules
dependencies. That is `base` comes before `web` in this list. The
objectif of this function is to pair each route to an endpoint whoose
class's MRO respect the above-mentioned order. Whe achieve this by
carefully crafting classes at runtime, classes inheritating from the
correct "source code" controllers in accordance to the topology. This
method is also responsible of merging each method's `original_routing`
into one `routing` dictionnary, the very `rule.endpoint.routing` dict
that is used through the rest of the http framework.
4. Each route-endpoint pair is saved into a `routing_map`, an object
that bind each route to its endpoint. This object exposes a `match`
method used to find back the endpoint given its route (=http path).
In addition to this refactor, we added some new helpers in our tools,
among them `submap` that implement a kind of `dict() - set() -> dict()`
operator. Filtering a dict on a set of keys is a common operation but
the python standard library lacks a dedicated operator.
PR: odoo#78857
Task: 2571224
This commit is the 3rd commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
Static files were historically served thanks to a werkzeug middleware,
the usage of that middleware prevented easy modification of the response
headers. Using `http.send_file` and `request._serve_static` we achieve
the same result as the middleware with less code and less frames in the
callstack.
The historic `http.send_file` have been splitted in two. One to send
a file via its path on the filesystem (prefered function) and one to
send a file already open. It is preferred to use `http.send_filepath`.
During development we tried to remove the fallback from the
`_serve_static` function into the more capables (and resource hungry)
db-compatible router. The purpose was to quickly respond to all requests
requesting static files at the cost of reduced customization. We have
been prooved wrong, e.g. the website themes all depends on the fallback
mechanism to deliver static files.
PR: odoo#78857
Task: 2571224
This commit is the 2nd commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
This commit poses the global structure of the new HTTP framework. Major
classes are present (yet empty) with a minimal docstring explaining
their general purpose.
PR: odoo#78857
Task: 2571224
This commit is the 1st commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
Our application framework is quite complicated. There are a few explicit
middlewares: `ProxyFix`, `DisableCacheMiddleware`, `SharedDataMiddleware`.
There are many (13, in total) implicit ones in the form of
`ir.http._dispatch` overrides. There are a few frames that don't have
much values by themselves and are very much implementation details
there are: `application_unproxied`, `_call_function`, `checked_call` and
`EndPoint.__call__`.
The ORM initialisation is quite magical, the cursor, registry and
environment are all setup lazily thanks to class properties. The
context and uid of the environment are aliased on the request. The
abuse of properties makes it impossible to reason about where and when
the environment is actually instantiated and modified.
The dispatch of a request follow the next scheme:
`odoo.http/Root.dispatch` -> `odoo.addons.base.models/ir.http._dispatch`
-> `odoo.http/(Http|Json)Request.dispatch` -> `odoo.http/@route` ->
controller. Because the request becomes http or json specialized quite
late, the many ir.http override all need to parsimony try/except their
code in order to manually call `_handle_exception()` uppon error, they
cannot let the error bubble-up. This back-and-forth between odoo.http
and ir.http is source of some headaches.
Speaking about error, the `odoo.http/WebRequest._handle_exception`
implementation is quite complicated, it basically craft a new exception
out of the passed exception object in order to correct its traceback.
If the error had been bubbled-up instead of handled by
`_handle_exception()` such python hack would not have been necessary.
The objectives of this refactor are about refounding the technical dept:
* We want shorter error reporting;
* We want simpler request and ir.http APIs;
* We want better integration of all the ir.http extensions.
PR: odoo#78857
Task: 2571224
Core client remains incompatible with CSP, however it can't hurt to
CSP the sub-resources.
Current scheme is simplistic, however if useful of necessary it could
be made more flexible e.g. there could be a map of mimetypes to CSP
configuration, that sort of things.
Access /base/static/img/country_flags/fr.png, 404 file not found but it
exists.
The conditional was wrong. We should load static files when the addon is
installable OR that it contains assets.
Code before 2e29a93:
expr = not manifest or (
not manifest.get('installable', True)
and 'asset' not in manifest
)
if expr:
continue
...
Code after 2e29a93:
expr = manifest and (manifest['installable'] or manifest['assets']
if not expr:
...
Proof:
a is manifest
b is manifest.get('installable', True)
c is 'asset' in manifest
expr = ~a | (~b & ~c)
~expr = ~(~a | (~b & ~c)
= a & ~(~b & ~c)
= a & (b | c)
Fine-tuning of 2e29a93closesodoo/odoo#81509
Signed-off-by: Raphael Collet <rco@odoo.com>
The http.addons_manifest is a map {module: manifest_dict} that is
populated upon the first http request. This map is basically a module
manifest cache with an extra `addons_path` key, the path of the module
on the file-system. This cache is eagerly populated upon the first http
request, the map is empty in non-http contextes (e.g. cron) which have
been a source of bugs (e.g. 50c8eb1).
A manifest cache is necessary because reading and parsing python files
from the file-system is not that cheap but there is no reason that cache
is located in `odoo.http`. A thin cache layer now wraps
`load_information_from_description_file()`/`load_manifest()` and is
lazily populated.
The `http.addons_manifest` have been removed. The extra `addons_path`
key is now present in the "normal" manifest. The `read_manifest()` was
hardly used so it has been deprecated. The only way to retrieve a
manifest is now `load_information_from_description_file()` which was
renamed `load_manifest()` (no cache) and `get_manifest()` (cache).
Side note about performances, the cache is necessary. Addons manifest
are read-only and reading + parsing python files from the file system is
not a cheap operation. Running the e-commerce tour
`@website_sale.test_04_admin_website_sale_tour` without cache on
`load_manifest()` requires 68,29 secs to complete on my laptop,
exceeding the default 1-minute time frame allowed in tests. Using a
cache the time is down to 36,53 secs. The performance impact is huge.
Part-of: odoo/odoo#79977
* add configuration for `flake8[flake8-rst-docstring]`
* enable docstring-related checks
* fix invalid docstrings in odoo's core & `base`
* fix a few more bits (mostly missing or incorrect `:param:` info
fields) are out of scope for the lint but my editor catches
closesodoo/odoo#74604
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
With the introduction of the guest feature, a lot of fields are now
accessed through a controller so that they can be available to guests,
with additional checking in the controllers.
Avatar is one such field, however the extra checks were too restrictive:
we only allowed people to see the avatar if the user whose avatar was
requested was also a member of the channel. This breaks the avatar on
previous messages if a user leaves a channel.
This commit relaxes this restriction for internal users (they can now
see all avatars through this route), and adds a fallback to the avatar
placeholder for people who do not have acces (ie: guests) so that the
UI doesn't look broken.
closesodoo/odoo#76341
X-original-commit: b849fd426806332c1ecc1bbba7b568384a021c65
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Despite that Werkzeug documentation specify:
'location (str) – the location the response should redirect to.'
Werkzeug support URL as location.
So now Odoo will support URL for request.redirect as argument too.
This commit closes#73729
Introduce the env variable ODOO_DISABLE_SESSION_GC to disable session_gc
on high-volume systems and avoid random slow calls as discussed in
https://github.com/odoo/odoo/pull/70063closesodoo/odoo#73376
X-original-commit: a61d459a200a56fba74f655f73b10d1ddd8e3a1e
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
This branch adds request.redirect on all requests.
In case of a front end request, we do an url_for to the location.
We removed redirect_with_hash that was only for retro compatibility
local_redirect has been renamed to redirect_query, and param keep_hash has been
removed and moved.
Default code for redirect is 303 now instead of 302.
Now redirect and redirect_query make local redirect by default, you need to
pass local=False to make external redirect.
All werkeug.utils.redirect has been replaced by request.redirect.
Http.redirect now use an http.Response type, and it become easy to add an
override like 'set_cookies' e.g.
Dispatch of a website.page return an http.response too, so we first need to
check if it is a cached version before to check if it is an Odoo Response.
Migrate your code:
http.redirect -> request.redirect(location, code, local)
http.local_redirect -> request.redirect_query(location, query, code, local)
http.redirect_with_hash -> request.redirect
Courtesy of odony for help and review ;)
closesodoo/odoo#72599
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
The usage of an additional optional context manager leads to the usage
of ExitStack. Unfortunately, this changes the initial format quite a
lot, decreasing readability and adding a loop on context managers, even
if there should be only one most of the time (request).
This commit proposes to a nesting utility for the profiler, allowing to
nest another context manager inside a profiler.
This solution allows an easier integration into http.py, avoids to
manage the "enter_context" case when recovering the init stack and can
be useful in other cases.
Improvement of get_dispatch_context_managers conditions and styling:
- Check profile expiration very first.
- Add an additional check for odoo.evented. Profiling will crash on an
evented server. Only longpolling should use evented servers, it but
this is more future proof.
Victor Feyens