*: base, http_routing, mass_mailing, web, web_editor, website_slides
In some situations `werkzeug.wrappers.Response` are used instead of
`odoo.http.Reponse` that extends it.
This is a problem because since [1] the calls to `set_cookie` expect it
to accept the `cookie_type` parameter, which is not the case in the base
werkzeug implementation.
This commit replaces the `werkzeug.wrappers.Response` by
`odoo.http.Response`.
[1]: https://github.com/odoo/odoo/commit/2cbda6c98ee947cea1d06c09880eee8c758304a8closesodoo/odoo#112827
X-original-commit: 28da08292b7028575e628c5ad846fc05d30498f2
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
This commit fixes a change in behavior between 15.2 and 15.3.
Previously, if an unidentified user tried to reach a route that had auth='user', it would simply redirect to the login page.
Currently, it redirects and invalidates the session_id.
This is an issue in the latest version of master after this PR https://github.com/odoo/enterprise/pull/36521
This commit changes the route of service-worker.js to auth='user'.
This route is called on the login page, which rotates the sid and therefore invalidates the csrf token. Making it impossible for a user to log in.
This is a race condition, meaning it would only appear if the user stayed on the login page for a few seconds, hence why the automated testing did not block the commit.
closesodoo/odoo#112239
X-original-commit: d5d80d172616afe02bd41934930ea18dc273c739
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
Co-authored-by: Julien Castiaux <juc@odoo.com>
Doesn't seem like it could hurt, and should only make CORB more
reliable by avoiding sniffing. Worst case scenario requires fixing a
few mimetypes, but aside from CORB it looks like modern (non-IE)
browsers only try to guess document mimetypes in very limited
contexts.
closesodoo/odoo#107957
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When overriding an existing controller route, developers can
easily c/p the route definition and call super() in the overridden method
when the route attributes are automatically deducted by odoo from the parent route.
Removing those redefined attributes simplifies the routes definition,
clearly highlighting what's changed by the override.
Also reduces unexpected behavior when modifying the base route without
noticing/considering the redefined attributes in a overridden route,
which overrides the changes made to the base route when the sub-module is installed.
This commit adds a test to catch routes attributes redefinition, and clean existing routes.
closesodoo/odoo#108512
Related: odoo/enterprise#35176
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Some cookies were left around even when the user logged out
The next user should log into the default company instead of the company
of the last user.
task-3077421
closesodoo/odoo#108998
Related: odoo/enterprise#35685
Signed-off-by: Julien Castiaux <juc@odoo.com>
Start odoo-bin shell with `--geoip_country_db='/i/dont/exist'` and type
the following code:
from odoo.http import GeoIP
GeoIP('8.8.8.8').country_code
TypeError: The country method cannot be used with the GeoLite2-City
database
There are two "things" here: the raw country and city entries in the
databases and the rich python country and city objects returned when an
entry was found and parsed.
Then there are two "readers", a reader capable of reading and parsing
entries from the city database and **another** reader for the country.
The two readers are **only** compatible with their specific database:
the city reader cannot read or parse information from the country
database and vis-versa. That's the bug reported by the TypeError that
this commit fixes.
But, the two python instances created from parsing the entries of
respecting databases with respecting parsers **do** share a same API:
the city record class inherits from the country record class.
# Valid
city_db = geoip2.database.Reader(config['geoip_city_db'])
city_record = city_db.city('8.8.8.8')
city_record.country.name # works, a city record also
# holds country attributes
# Invalid
city_db = geoip2.database.Reader(config['geoip_city_db'])
country_record = city_db.country('8.8.8.8') # TypeError
closesodoo/odoo#109095
Signed-off-by: Julien Castiaux <juc@odoo.com>
request.geoip is no more a dictionnary cached in the session. It is now
a full blown object with lazy and smart geolocalisation capabilities.
Among other things, the previous dictionnary API is now deprecated. The
changes are:
* `request.geoip['country_name']` -> `request.geoip.country_name`
* `request.geoip['country_code']` -> `request.geoip.country_code`
* `request.geoip['city']` -> `request.geoip.city.name`
* `request.geoip['latitude']` -> `request.geoip.location.latitude`
* `request.geoip['longitude']` -> `request.geoip.location.longitude`
* `request.geoip['region']` -> `(request.geoip.subdivisions[0].iso_code if request.geoip.subdivisions else None)`
* `request.geoip['time_zone']` -> `request.geoip.location.time_zone`
It is safe to access all the attributes. Doing `request.geoip.city.name`
when the geolocalization failed (missing db, invalid address, ...)
evaluates to None. It does not raise an AttributeError.
Task: 2848206
Part-of: odoo/odoo#91337
Maxmind offers multiple ip-geolocalization databases, historically we
have been using the City database which contains records on a
city-basis. Many years later it turns out we are primary using geoip to
know the country of the user. Geolocalization in the City database is
considered slow by our standard and we have been clever in order not to
geolocate each request by saving the info in the session.
On the other hand, the Country database that is offered by Maxmind is
much more lightweight and geoip using that country is considered a fast
operation by our standard.
In this work we make Odoo compatible with both the City and the Country
databases. Using multiple database at the same time, we can be smart and
only query each of the two on-demand. If a user ask for its country,
we'll use the fast Country db. If a user ask for its city/timezone we'll
use the slower City db.
By default it loads both database from the `/usr/share/GeoIP/` folder,
respectively the files `GeoLite2-City.mmdb` and `GeoLite2-Country.mmdb`,
you can provide alternative paths using the `--geoip-city-db` and
`--geoip-country-db` CLI options.
In the same mindset as #86015, geoip is still lazy. It is done on-demand
and the result is cached on the current request. The different with the
related PR is that as we know consider geoip to be fast, we no longer
cache the result in the session.
Task: 2848206
Part-of: odoo/odoo#91337
*: base_setup, hr_timesheet, mail, partner_autocomplete, web_tour
Start odoo without -d and with a --dbfilter that allows multiple
databases. Via JSON-RPC access the /web/session/authenticate route
providing a non-filtered database and valid credentials. Traceback,
`request.env` is None.
Since httpocalypse the initialization of the ORM (cursor, registry,
environment) is greedy. It means that the connection to the database is
established very early during the request routing or skip altogether in
case no dbname was known at that time. This contrast with prepocalypse
where the various ORM thingies were lazily setup the first time they
were accessed.
This changement has an important implication regarding authentication.
In prepocalypse, thanks to the lazy approache, a cursor/registry/env
would be setup on the database you just login upon using the
`request.env` for the first time. This was very nice in this regard but
had other problems.
Since httpocalypse such operation is no more possible. Devs must
initialize and use their own cursor/registry/env in case they
authenticate on another database than the one `request.cr` is (maybe)
connected to.
The `/web/session/authenticate` controller is an example of such case.
It crates its own cr/registry/environment after authentication. The
problem the controller uses `ir.http.session_info` and that not all
overrides were updated to use `self.env` (=the env created in the web
controller) instead of `request.env` (=the missing env of the request).
closesodoo/odoo#108063
X-original-commit: 7b9bd9d37731fae724dc5d91da656dab70aa9ad4
Related: odoo/enterprise#35012
Signed-off-by: Julien Castiaux <juc@odoo.com>
The main goal of this commit is to reduce the size of the registry by
removing the (almost) useless __last_update field.
Statistics # of fields with all modules installed:
before 30184 fields, 1299x last_update (4.30%)
Before this commit, the computed field __last_update was added on every model.
The idea behind this field was to have a computed field that had either
the write_date or the create_date if the write_date was empty. However,
the write_date is always written, even on creation, making it useless
to have the computed field __last_update
After this update, we completely remove from BaseModel:
* __last_update
* CONCURRENCY_CHECK_FIELD that was always defined as "__last_update"
* _compute_concurrency_field that was the compute function for __last_update
closesodoo/odoo#105739
Task-id: 3062140 (part of 3062137 improve registry load time)
Related: odoo/upgrade#4038
Related: odoo/enterprise#33939
Signed-off-by: Raphael Collet <rco@odoo.com>
The routing-map is a mapping that maps HTTP verbs and paths to python
controller methods (endpoints), e.g. it maps `GET /web/health/` to
`/web:Home.health`. The `_generate_routing_map` function is the function
responsible to generating the werkzeug routing-map in regard to the
controller inheritance mechanism. The mechanism makes it possible to
override an endpoint is various odoo modules to enrich it with new
features, e.g. `/web/login` is overriden in website to change the visual
of the page.
Implementation-wise, the informations from each `@route` decorator must
be merged with the other `@route` info for each endpoint override. When
a method is not overriden in a controller, there is not new info and
that controller should be skipped for that method.
The previous implementation attempted to skip such method using the
following idiom:
if not hasattr(controller, method_name):
continue
That idiom doesn't work as `hasattr` will perform a lookup on the full
controller's MRO instead of a lookup only on controller's own methods
and attributes. The controller's own methods and attributes are
actually found in `controller.__dict__`.
closesodoo/odoo#107064
X-original-commit: 42f52d26b6a81a68f1fe28e28971f0e7f4c97d11
Signed-off-by: Julien Castiaux <juc@odoo.com>
Define the following controller:
@route('/get-context', auth='public', type='json', website=True):
return request.env.context
Access that route via JSON-RPC providing a context:
this._rpc({
route: '/get-context',
params: {
context: {"a": 1}
}
})
Since httpocalypse, it replaces the context entirely instead of updating
it. i.e. before you would get along those lines:
{"a": 1, "lang": "en_US", "tz": "Europe/Brussels", "uid": 2}
but since you get instead (and that's wrong):
{"a": 1}
Note that after this commit, this merging-context behavior will be the
same whether or not website=True is defined (while it was only there for
website=True before).
Discovered via opw-2885948
X-original-commit: 7887cfc3f177d9e64f008aabb8bd710f54024dea
Part-of: odoo/odoo#107179
Move code to support only the redirect from url containing double '/' in
the middle of the path.
Keep same behavior than v15 and default Apache behavior.
domain.com//shop/product/1 -> 404
domain.com/shop//product/1 -> 301 -> /shop/product/1
opw-3063387
closesodoo/odoo#106137
X-original-commit: fdd6bf9942e2807b6d1460dba1ca5f61404c7b06
Signed-off-by: Julien Castiaux <juc@odoo.com>
Signed-off-by: Jérémy Kersten <jke@odoo.com>
Create a 15.0 database with website, access the home page via your
browser. Stop the server and migrate the database to 16.0. Restart the
server with a `--dbfilter` that rejects the database you created and
refresh your browser. 500 Internal server error, attribute error:
the `request` object as no `session`.
An error could occurs after a migration to 16.0 due to the presence of
the `geoip` key in the session. `request.session.geoip` has been made a
deprecated alias to `request.geoip` between 15.0 and 16.0, see 04e9726.
Because the session was created before 16.0, the session dict does
contain a `geoip` key. Upon logging the session out, the session dict
is cleared. The default implementation of `clear()`[^1] inside of
`collections.abc.MutableMapping` can be summarized for our usecase to:
for key in self:
value = self[key]
del self[key]
There is an extra `__getitem__` call due to `value = self[key]`, in the
case of the `geoip` key, it would access the alias. It is not possible
to accessing that alias inside of the `_get_dbname_and_session` method
of request as the session has not been set on `self` (the request) yet.
Yet inside of that method, we do `session.logout()` which `clear()` the
session which (wrongly) access the alias because `geoip` exists in the
internal dict (`'geoip' in self.keys() # True`).
The solution has been to implement the `clear()` function ourself
instead of using the mixin of `MutableMapping`.
[^1]: https://github.com/python/cpython/blob/b43496c01a554cf41ae654a0379efae18609ad39/Lib/_collections_abc.py#L925-L931closesodoo/odoo#105763
X-original-commit: b66e1ffa8e348eedf2de735babbc398290a8bffb
Signed-off-by: Julien Castiaux <juc@odoo.com>
The new `borrow_request()` function has been introduced to properly
separate the HTTP layer from the RPC layer. We forgot to protect some
RPC endpoints, mainly inside of `odoo.addons.web.controllers.database`.
This commit moves `odoo.service.dispatch_rpc` to `odoo.http` as we only
permorm RPC from the controllers and that we don't want to import
`borrow_request` inside of `odoo.service` (circular import).
X-original-commit: d0ee8615d8820e02232989c50a6e6ffbc66a9266
Part-of: odoo/odoo#105710
Install website then render a qweb template via xmlrpc, attribute error:
`request` has no `is_frontend` attribute.
Inside the qweb's `_prepare_environment` override of the http_routing
module (a dependency of website) is the following code snippet:
if (not irQweb.env.context.get('minimal_qcontext') and
request and request.is_frontend):
return irQweb._prepare_frontend_environment(values)
The conditionnal is about injecting extra informations in the qweb's
context in case we are serving a frontend request. By default, there is
no `is_frontend` attribute on the request object. That attribute is set
by the ir.http's `_match` override of the http_routing module (a website
dependency): it is set True when we don't match any endpoint or that we
match an endpoint that is `website=True`, it is set False otherwise.
The ir.http's `_match` method is called whenever we are serving a http
request whoose session is bound to a specific database. i.e. when there
is a valid database saved in the request's session. When there is not
database in the request's session (or that it is invalid) the matched
endpoint is directly called without going throught ir.http. Most
endpoints are only accessible via ir.http.
The two `/xmlrpc` and `/jsonrpc` endpoints are examples of endpoint that
do not require an established database connection to work. They perform
the request authentication and database connection themselves. It is
possible to call those two endpoints with no database saved in the
session, thus it is possible to call those two endpoints without going
throught ir.http. This is expected.
The two endpoints's duty is to execute public model methods and return
the xml/json serialized result. To do so, a registry is loaded on the
database with all the installed modules, including http_routing.
We fall in a situation where (1) there is a request, (2) we are using a
registry where http_routing is loaded, (3) there is no `is_frontend`
attribute on `request` as we didn't serve the endpoint via ir.http. This
situation is illegale.
To solve the problem, instead of working on the `if request.is_frontend`
bit of the above conditional, we decided to work on the `if request`
bit. ISO-model wise, RPC is an extra 8th layer built on top of HTTP.
HTTP is merely a transparent transport between a RPC client and a RPC
server, any other request-response capable procotol could fit. The
method executed via RPC must be independant from the usage of HTTP as
mean of transportation thus it should not be capable of using the
current request.
The proposed change is to temporary un-expose the current request from
the local-stack during the execution of the RPC method. This fixes the
problem as the code now run like it was executed from the shell or from
a cron. The other benefit is that the pattern used inside the condition:
`if request and request.is_frontend` doesn't need to change.
X-original-commit: f28863bfdaac3f644fcb274c856e0770783ed75c
Part-of: odoo/odoo#104567
When `make_json_response` was added in
034d01b2f3 and `_response` was updated
to use it, the `http_status` extracted from the error object was
removed.
But it's still set by `handle_error` and a local is defined for
it. Drop that.
closesodoo/odoo#104507
X-original-commit: b0ea1cd6a2628a6acd1e2d79cc862976c63b1565
Signed-off-by: Julien Castiaux <juc@odoo.com>
When calling a `auth=none` route with a non-existing DB name, the server
redirects to `/web/database/selector`. Such a route can be called
without database, therefore it is expected to work if the database
doesn't exist as well.
Before the HTTP refactoring, the fallback was `_dispatch_nodb` [1]. We
roll back to the same behavior since there is no good reason to redirect
to `/web/database/selector`.
Commits 4b330f3173 and de4e67dcc529163 are reintroduced as well.
[1] https://github.com/odoo/odoo/blob/b3199e949ae70307e8181ba0d6237871d6d07af7/odoo/http.py#L1529closesodoo/odoo#104174
X-original-commit: 1565e85890f1d14e067edd1c198c7199ab536307
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
Signed-off-by: Olivier Dony <odo@odoo.com>
* clean and improve docstrings in orm
* fix typos found with codespell
* rely on the Environment class docstring instead of doc content (and
therefore move part of the doc inside the class docstring)
closesodoo/odoo#102969
X-original-commit: 8250cd4b210005d223a4cdb8afa4014425ca6fa3
Related: odoo/documentation#2803
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
Both implementations of `set_cookie` in `http.py` and the one of
`setCookie` in `cookie_utils.js` are slightly different.
This commit unifies both Python implementations and replicates the same
behavior in JS:
- default the cookie type to "required"
- if setting that type is not allowed, delete that cookie if it was
previously set
task-2800976
closesodoo/odoo#102380
X-original-commit: fa8fdab2894607f1f89cdd195f62ec37af7b0342
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
*: im_livechat, survey, utm, website_crm_iap_reveal, website_forum,
website_livechat, website_sale, website_sale_comparison
Before this commit all cookies were considered essential.
This commit makes some of them optional. It also makes it possible for
the website visitor to only accept the essential cookies.
task-2800976
X-original-commit: 9a8a9463289a7446e9be0ef62ff895feb37a4de4
Part-of: odoo/odoo#101845
Co-authored-by: Benoit Socias <bso@odoo.com>
This commit avoid to have a dict by reference that will be global.
Now get_default_session return a new dict each time for the context key.
From this way the session.context['lang'] is not shared between several
users on the same worker.
To reproduce the bug, restart the server with 2 workers, make request in
lang A on these 2 workers. DEFAULT_SESSION['context']['lang'] now is set
to this lang A.
Now, make request to an url without lang in path and without cookies and
withtout session, you should be redirected to lang B (preferred lang
from the request header) but you will be redirect to lang A due to the
dict session.context that is shared for the worker...
When we initialize the new Session, we get the wrong lang A as value for
context.lang, so we don't recompute the expected lang for the end user.
X-original-commit: 62179de74862210fe2a055d15b367b1850c24263
fwd-port of #100102closesodoo/odoo#100910
X-original-commit: 42e46b2d89dde276f796b980f29e33cc216e7cb2
Signed-off-by: Jérémy Kersten <jke@odoo.com>
XML files are now declared in python module manifests. During the qweb
't-call-asset' directive, assetbundle will fetch the declared xml files,
apply the inheritance (t-inherit) and create a javascript service (for
eg: 'web.assets_backend.bundle.xml') which is added at the end of the
*.js mimifier file.
When the debug mode is activated, comments are added in the template
indicating which file the template comes from as well as the
inheritances applied to it.
****
JavaScript:
assets.js (module @web/core/assets) takes care of loading libraries,
javascripts and styles.
`loadJS(url)` (loads the javascript and returns a resolved promise when
the templates are also loaded via the '*.bundle.xml' service)
`loadCSS(url)` (loads the style a resolved promise when the file is
loaded)
`loadXML(xml, app=assets.defaultApp)` (load template into
application/owl, used by the `*.bundle.xml` services)
`getBundle(bundleName)` (get the bundle descriptor)
`loadBundle(desc)` (load the files and bundle from a descriptor)
templates (XML element content all owl templates)
A new `ready(serviceName)` method on boot.js lets you know when a
service is loaded are the require.
The xmlDependencies attribute no longer exists.
Python:
The xmls taken into account by assetbundle.py, applying `t-inherit`
inheritances and adding an `name_of_the_bundle.bundle.xml` service in
the generated JavaScript file.
****
Every manifest changes is into the next commit, except 'web_tour' in
this current commit as example.
Part-of: odoo/odoo#95500
When a file is immutable the different network layers can cache it.
Services receiving this header should never invalidate these files.
Part-of: odoo/odoo#95500
*: hr_presence, web_editor.
This commit is part of the websocket integration in Odoo.
It focuses on adapting the bus to support websockets:
- last notification id is now kept on the server
- channel list is built by overriding the `_build_bus_channel_list`
method of the `ir_websocket` model instead of overriding the `_poll`
method of the bus controller.
- The bus presence was updated during polls, since there is no more poll,
bus presence update will be the responsability of the client.
- The `/websocket/peek_notifications`, `/websocket/update_bus_presence`
routes will be available so that odoo sh can access notifications/update presence
from http requests.
- /longpolling routes are now prefixed with /bus thus won't be redirected to the
gevent worker anymore except for `/longpolling/health` which is the
health check route of the gevent server.
Since websocket now handle incoming messages, a way to manage authentication
have been introduced :
- The session is retrieved from the HTTP handshake.
- When a websocket message comes/leaves the session is retrieved
on the file system so that we're sure it still exists and that
it is up to date.
- The session is checked
- If no session is found on the file system or `check_session`
fails, the websocket connection is closed with the `SESSION_EXPIRED`
close code (which is a custom close code: 4001).
- Note that websocket connections are closed every `KEEP_ALIVE_TIMEOUT`
seconds to ensure no websocket connection will stay open if the user
clears its cookies.
- Note that a wsrequest object is available when processing incoming
messages. It is similar to the http request and contains various
useful informations (session, env, ...).
Part-of: odoo/odoo#75510
This commit is the first commit of the websocket integration in Odoo.
It focuses on the implementation of the websocket protocol as per RFC6455.
The implementation is tested thanks to the autobahn test suite.
A config parameter is available to customize the websocket connection:
- websocket_keep_alive_timeout (default 600): Integer specifying how
many seconds a websocket connection should be kept alive
Part-of: odoo/odoo#75510
It allows to always have a OdooResponse Object and don't allow redirect
to external except when you allow it explicitly with local=False.
Always return to a local url:
/website/add
/slides/slide/<model("slide.slide"):slide>
/microsoft_outlook/confirm
Allow previously external redirect without reason, now blocked
/website/lang/<lang> -> open redirect
Allow external redirect for good reason and url is controlled by code.
/social_facebook/redirect_to_profile/
PS: HTTP Code 303 is a better default for generic redirects. It's not
historically the default for werkzeug.utils, but it is what we want in
general. Contrary to 302, there is no browser-dependent behavior, and
no risk of asking the user whether they want to accept the redirect if
the original method wasn't GET. It's always a non-permanent GET on the
target location.
closesodoo/odoo#95019
X-original-commit: 77f8d9c5d96a9274785ffc2ad83b95ea157d26ad
Related: odoo/enterprise#29000
Signed-off-by: Olivier Dony <odo@odoo.com>
Signed-off-by: Jérémy Kersten <jke@odoo.com>
Leftovers of previous fixes of this issue.
closesodoo/odoo#95162
X-original-commit: dffc184862018670008ff4da839fc520a72bddca
Signed-off-by: Christophe Simonis <chs@odoo.com>
Create an empty database and start 2 http workers, go on the web app
menu and install website (don't install website via -i). Once website is
installed, you are redirected on `/website/configurator` but the route
does not exist and it fails with a 500 internal server error.
The problem is due to an invalid registry manipulation introduced in the
saas-15.3's httpocalypse. When new modules are installed the registry
must be reloaded in all workers. The function that determine if the
registry must be reloaded and reloads it is `check_signaling`.
When the current registry is up-to-date, it is returned as-is by
`check_signaling`. When it is outdated, `check_signaling` creates and
returns a new fresh registry; it does not nor discard nor change
in-place the previous (outdated) registry, it is up to the callee to
discard the previous registry itself.
closesodoo/odoo#93579
X-original-commit: 23bdcc3fd31e6908ee6737cb74787aefa5d057a5
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
This commit tries to avoid a traceback when no postgresql connexion is available.
closesodoo/odoo#92195
X-original-commit: 8199daf4bcba76277fed39554863efa3f1585b20
Signed-off-by: Josse Colpaert <jco@odoo.com>
When started with the CLI option --dev=werkzeug, errors in controllers
are caught by a friendly web debugger. This debugger should not be
started in case of error in a JSON-RPC controller.
closesodoo/odoo#90420
Task: 2837457
X-original-commit: 711a29e76f55e87d73dad89ddbef889dd5c7af10
Signed-off-by: Julien Castiaux <juc@odoo.com>
With this work we relax the http controller so that it accepts all
requests, including `Content-Type: application/json`. We also enrich
the framework with two new helper methods dedicated to serialaze json
requests and responses:
- `request.get_json_body()`, loads the json content from the request's
body and returns the corresponding python object (usually a dict).
- `request.make_json_response(data)`, dumps `data` to json and makes an
http response out of it.
closesodoo/odoo#86300
Task: 2779837
Signed-off-by: Julien Castiaux <juc@odoo.com>
Every request comes with a session, a dictionary that is persisted on
the filesystem and that saves various information such as the user
cart on the ecommerce.
When a user simply visits the website, a default session is created and
saved on disk, this bloats the filestore with many sessions. Creating
the session on-the-fly is cheaper than loading it from the filesystem.
With this work the default session is not saved on disk anymore unless
explicitly asked via `session.touch()`.
An exception to the statement "creating the session on-the-fly is
cheaper" is geoip, the ip geolocalization is not cheap. In this work,
geoip have been moved from http_routing/request.session.geoip to a
lazy property core/request.geoip. When requested the info is persisted
on the session. Like other keys from the default session, geoip will not
be persisted unless there is non-default stuff in the session.
Because the CSRF-TOKEN is based on the session-id, it is important the
session-id stays the same across multiples requests even when the
session is not persisted on disk. Even when a session is not persisted
on disk, the session-id cookie is still set so that the next session
created on-the-fly uses the same session-id.
Technical note regarding the session, it has been decided to drop the
session-snapshot protocol and to reintroduce a "modified" flag. It has
been decided not to use werkzeug's session (which natively comes with a
"modified" flag) and to keep our own session object. We decided to
extend MutableMapping instead of dict; using MutableMapping we only
have to override __setitem__ and __detitem__; using dict we would had to
override update()/pop()/... too.
Task: 2789035
Part-of: odoo/odoo#86015
Steps to reproduce:
1) start from a clean database (http_routing should not be installed).
2) go to the app menu and install project (don't install via `-i`!!).
3) traceback: `request` has not attribute `is_frontend` while rendering
a template.
The `is_frontend` attribute on request is set by the http_routing
module. At the moment the "install now" button is clicked, http_routing
was not installed so the `is_frontend` attribute was not set.
FF to the end of the installation: the registry is reloaded to include
the modules that have been installed, http_routing among them.
We are in a tricky situation: (1) there is a request, (2) http_routing
is installed and (3) the `is_frontend` attribute is missing from the
request.
This situation is illegal, when http_routing is installed, the
`is_frontend` attribut should always be set. In this work we reset the
missing attributes using sensitive default values via post-init hooks.
closesodoo/odoo#87684
Signed-off-by: Julien Castiaux <juc@odoo.com>
With the iot-box it is possible to download modules from an odoo server
into the iot-box. Those odoo modules are `exec`-like and exposed on the
box. With those modules come some controllers. Because they were
`exec`-like, they lack a python module name which makes
`_generate_routing_rules` to reject them.
With this fix, we authorise to declare controllers outside of odoo
addons. Those controllers cannot be extended, nor can they override
other controllers.
closesodoo/odoo#86275
Signed-off-by: Julien Castiaux <juc@odoo.com>
* http.py:docstring of odoo.http.route:19: WARNING: Unexpected
indentation.
* http.py:docstring of odoo.http.route:20: WARNING: Block quote ends
without a blank line; unexpected unindent.
* http.py:docstring of odoo.http.route:30: WARNING: undefined label:
csrf
-> No section in the doc exists with reference csrf
* Add docstring for the default_lang method.
closesodoo/odoo#85505
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
This commit is the 14th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
* `request.uid = x` => `request.update_env(user=x)`.
* `request.context = x` => `request.update_env(context=x)`.
* `request.context = dict(request.context, x=y)`
=> `request.update_context(x=y)`.
* `request.cr = None` => `request.cr.close()`.
* `http.mono_db()` => `request.db`.
* `http.dispatch_rpc()` => `service.dispatch_rpc()`.
* `@service.model.check` => `service.model.retrying()`.
* `request.endpoint`
=> `env['ir.http']._match(request.httprequest.path)[0].endpoint`.
* `request.routing_iteration `=> `removed`.
* `request.jsonrequest` => `request.dispatcher.jsonrequest`.
Note that `request.params` is now set much later in the process. If you
are in a situation where you values from the query string or the
http body you can use `request.get_http_params()`.
Note that using the new `request.future_response`, it is possible to
add headers and cookies on the response object before the response
object is initialized. Please note that headers/cookies saved on
the future response will NOT be injected in case of error.
PR: odoo#78857
Task: 2571224
This commit is the 12th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
The web module is twofold, on one side there are many controllers: /,
/web, /web/login, /web/database/selector, /web/dataset/call_kw, etc, on
the other side there is `session_info`: the method responsible to create
the web client's environ.
This module is kinda an exception as it is (with base) a server wide
module. In the case of the HTTP framework, it means that the controllers
of web are always accessible, i.e. going to / or /web/login will never
return a 404 Not Found even if the user is not connected to a database.
This is both a blessing and a curse. It is a blessing because the
controllers are always accessible it means that a new users can freely
access those routes. It is a curse because *any* user can access them,
even user who don't have a session yet thus who are not connected to a
database yet. From a developer standpoint, we have to put extra care to
correct serve users with and without a database. An example is the
/web/login route, the login/password pair is stored in a database,
without database it is impossible to validate a user login but users can
still access this route without db.
To solve this problem, there is the `ensure_db` function. This function
attempts to find a database using various sources (?db= query-string,
session db, mono db) and to save it on the user session. In case no db
is found, the user is redirected to the database selector. In a way,
this function grants a database to the user in a seamingly experience.
In a way, this function brings a welcome differentiation between
`auth='none'` with a database and `auth='none'` without a database. Such
differentiation only matters for the server wide modules as "regular"
module controllers are only accessible via the ir.http routing map, i.e.
it is not possible to declare a nodb controller outside of server wide
modules.
An important changement is the `session.authenticate` method, before it
was possible to call the method when the cursor was not yet initialized,
authenticate would open a cursor against the given database, setup a
registry and an environment and ultimately save everything on the
current request. Because the cursor is now greedily created, it is no
more possible to update the request environment when authenticating on
another database.
PR: odoo#78857
Task: 2571224
This commit is the 11th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals. See also [REF] core: HTTPocalypse (9) ORM initialization.
Where `request._serve_db` is reponsible for initializing the ORM,
`request._serve_ir_http` is responsible to do the match -> authenticate
-> pre dispatch -> dispatch -> post dispatch sequence. This sequence is
very much similar to `_serve_nodb` with the notable addition of the
authenticate step.
Where `_serve_nodb` was working in a db-free environment, calling the
methods of (Http|Json)Dispatcher right away, `_serve_ir_http` delegates
to the `ir.http` abstract model which in turn use the dispatchers.
The ir.http model is crucial for several modules, web, portal and
website on top. They override the various methods to add advenced
capabilities to the http framework.
An important changement to the ir.http model in this work is regarding
the `_dispatch` method. Before this work it was that method that was
responsible of doing the sequence match -> pre dispatch -> ... . It was
decided to move the responsibility to http.py in order to prevent
modules from taking over the entire framework. One such take-over was
done in the `http_routing` addon and resulted in arguably very poor and
fragile code.
All modules must now carefully override the correct method. You need to
do smart stuff on the URL prior to matching an endpoint? You should
override `_match`. You need to save a special option from the query-
strings to the session before calling the controller? Override
`_pre_dispatch`.
PR: odoo#78857
Task: 2571224
This commit is the 10th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
One of the objectives of this comprehensive refactor was to simplify the
error reporting, to show developers shorter and more informative
traceback upon errors.
We argue that the current error reporting is too much bloated:
1. There is a "dual" traceback, one traceback for the frames more recent
than `/base:ir.http_dispatch` (`@route` wrappers + controller) and a
second traceback for the former frames (http framework + web server).
2. We trace too far back, we argue that it is useless to trace back to
the thread/worker creation, that we should only trace back until the
WSGI entry point.
3. There is too much noise, many frames are either implementation
details or useless information. e.g. middlewares, `_dispatch`
overrides, wrappers, "checks".
To solve the above problem we did the following:
1. All errors bubble up to `request.__call__` where it is logged, any
wrapper that catch an error must log it or re-raise it. By logging
exceptions in `__call__`, we ensure the traceback at that function.
2. When one raises an HTTP error, e.g. 404/NotFound, the object is at a
same time an exception and a response. We don't log it and use the
response right way.
3. When one raises an exception, e.g. ValueError, the exception is sent
to `handle_error`, its job is to return the HTTP error corresponding
to the given exception, e.g. 500/InternalError in case of ValueError.
The given exception's `__traceback__` and `__clause__` are not
modified, we log the real exception but use the http error as
response.
Minimalistic example showing a traceback before and after this work:
https://gist.github.com/Julien00859/d6a48d523cac42d0df2e3136f5ba3e54
PR: odoo#78857
Task: 2571224
This commit is the 9th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals. See also [REF] core: HTTPocalypse (11) ir.http base model.
This is a two-part commit with "(11) ir.http base model". In this commit
we focus on the initialization of the various ORM objects, namely: the
registry, the cursor, the environment, the user and the context. In the
other n°11 commit we focus on the ir.http model and its relation to the
current http.py module.
One of the objectives of this comprehensive refactor was to ease the
cognitive complexity of the http framework, in other words to make it
simplier. One of the problem identified quite early during the
preparation of this work is the way the various ORM objects are
initialized, modified and cleaned during the request lifetime.
Before this work, all the ORM internals were lazily initialized via
properties. It is the first time one uses `request.cr` that a cursor is
opened to `request.db` and stored on `request._cr`. It is the first time
one uses `request.env` that an environment is create with the current
`request.user` and `request.context`. Upon user or context modification,
the current environment is discarded, the next usage of `request.env`
will create yet another environment on the fly using the modified user
and/or context.
Using this model, no ressource is initialized if not necessary. It is
possible for nodb-compatible endpoint to be served via the db-compatible
router and ir.http without ever opening a cursor to the database.
But this model is harder to reason about and ultimately to maintain.
In this work we propose to drop the lazy approach for a greedy one. In
this work the first steps of `_serve_db`, the db-compatible counter-part
of `_serve_nodb`, are dedicated to setup a registry, open a cursor to
the database and create an environment using the session's user and
context. In this work, when one wants to change the environ's user or
context, he must call `request.update_env` or `request.update_context`,
both method will recreate the environment *now* with the given values.
The downside of this approach is that resources are always allocated
even when it is not necessary. We argue that, in general, the
controllers that do not use the ORM are rare thus it is rare we allocate
unecessary resources. We also argue that the cognitive benefits are more
than welcome and that the new APIs will help at writing more robust
applications.
PR: odoo#78857
Task: 2571224
This commit is the 8th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
PR: odoo#78857
Task: 2571224
See also: [REF] core: HTTPocalypse (6) serve db-free routes
This commit is the 7th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
PR: odoo#78857
Task: 2571224
See also: [REF] core: HTTPocalypse (6) serve db-free routes
This commit is the 6th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
There are some routes that must work even when the user is not connected
to a database yet. Two obvious examples are the index (/) and the
database selector (/web/database/selector). Accessing those routes
should be possible even when the user is not connected to a database yet,
the `nodb_routing_map` and the `_serve_nodb` method fulfill this need.
Along with the introduction of the `_serve_nodb` method, we also
introduce the match -> pre dispatch -> dispatch -> post dispatch
sequence. `match` matches an endpoint using the http path,
`pre_dispatch` prepares the system, `dispatch` is the actual endpoint
call, `post_dispatch` cleanups the system and add some http headers to
the http response.
At the beginning, when an endpoint is matched, we retrieve the `routing`
dictionnary that is attached to the endpoint method. This dictionnary
contains the information set by the `@route` decorator, among other
values is the routing type `http` or `json`. This routing type is used
to specialize the request via one of the dispatchers: HttpDispatcher for
http and JsonDispatcher for json. Via the dispatchers, it is possible to
add custom behavior in the form of (pre_|post_)dispatch override. One
example of such custom behavior is the way the request body will be
loaded: urllib.parse.parse_qs (http) vs json.loads (json).
Before this work, it was not possible for multiple dispatchers to be
compatible with a same request mimetype. Because `JsonRequest` was
implementing the support for `application/json` (and like) mimetypes,
all requests having a body's mimetype `application/json` were considered
JSON-RPC2 (the protocol implemented by `JsonRequest`) even if the
request was actually bare json data. With the introduction of the
`is_compatible_with` classmethod, multiple dispatchers can be compatible
with a same mimetype and it is up to the `@route(type=...)` param to
define which dispatcher should be used with this route.
PR: odoo#78857
Task: 2571224
Benoit Socias