Replace all the calls to get_resource_path to the better file_path or
directly use file_open when not needed
Doing both a get_resource_path and file_open means checking twice that
the file exists.
Doing a simple path concatenation before a file_open is safe.
If given to another method (e.g. etree.parse), calling file_path is
the prefered method.
Note that get_resource_path used to return False when the file does
not exists while file_path/file_open raises a FileNotFoundException
closesodoo/odoo#135607
Related: odoo/upgrade#5187
Related: odoo/enterprise#47475
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Purpose
=======
When a user tries to reach a course, an AccessError can occur when we
unslug the URL. Instead of the traditional error page, we want to
redirect the users to /slides, and the error will be displayed there.
Task-3477630
closesodoo/odoo#135926
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.
Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.
The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.
Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.
The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.
We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.
Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.
Note that this poc is inspired from the long term cache but not all
use case where applie yet.
Part-of: odoo/odoo#119813
If a user is not present in the request, he is in no group at all and
can not access any model, including the one available for public
users.
Avoid ambiguity by using sudo or add a user specifically.
Part-of: odoo/odoo#125216
Before 16.0 and https://github.com/odoo/odoo/pull/78857 the session
cookie duration was set to 3 months, but the server-side garbage
collection of inactive session was reaping them after 7 days of
inactivity. The cookie lifetime was essentially superseded by the
server-side GC.
After https://github.com/odoo/odoo/pull/78857 these limits were made
consistent with each other, but the lifetime value was kept at 3 months,
which is a bit too long as a default.
This commit changes the default SESSION_LIFETIME back to 7 days for both
limits.
In addition, since the server-side GC is now implemented by a
database-specific cron job, this commit introduces an optional system
parameter `sessions.max_inactivity_seconds` that can be set to override
the default server-side GC threshold, to make it shorter.
Note 1: the ICP does not modify the cookie lifetime which will remain set
to the default 7 days. This means normal browser sessions won't stay
alive for longer than 7 days of inactivity. So `sessions.max_inactivity_seconds`
can't be effectively set to a longer expiration time.
This seems like a reasonably safe default.
Note 2: the session GC happens during the execution of the autovacuum
cron job ("Base: Auto-vacuum internal data") which is scheduled once per
day by default. When setting a small `sessions.max_inactivity_seconds`
value, it may be necessary to increase the frequency of that cron job
accordingly.
closesodoo/odoo#122964
X-original-commit: 05ff9a2db32c2fb1afa107ac005423218f452290
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
When generating the routing map, `Rule.add` will call
`Rule._compile_builder` twice, representing almost one third of the
routing map generation time. `_compile_builder` is actually stored
in `Rule._build` and `Rule._build_unknown` for latter use, in the
`Rule.build` method.
The `Rule.build` method is used to transform a rule, lets say
`/forum/<model("forum.forum"):forum>` in `/forum/basics-of-gardening-2`
Even if a deeper investigation could be interresting, it looks like it
is only used for `is_frontend_multilang` routes and `_enumerate_pages`,
used in the `sitemap` and `search_pages`.
The proposed solution si to make this part lazy, in order to call
_compile_builder on demand, once per rule. This will speedup the initial
routing map generation and postpone the heavy work when we need it,
only for the part we need most of the time.
On a database with all modules installed, generation of the routing map:
Before: ~600 ms
After: ~200 ms
An alternative implementation was also overriding the Rule.build method
instead of having a callable LazyCompiledBuilder, this implementation
was choosen since it is less dependant off the werkzeug implementation.
closesodoo/odoo#120542
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
A visitor could visit a web page having a lang in its context that is
not installed in the databased he is connected to. The problem is that
visitors are not logged-in thus it is not possible to determine their
lang via their `res.users` preferences. The lang used instead is the
lang set in the `Accept-Language` header of the incoming request, that
header is set by various browsers in accordance to the user system
preferences or browser settings.
The browser lang (`Request.best`) is only parsed according to the
`babel` database, it is a lang syntactically speaking but not necessary
a lang that is installed in the database.
At the moment the browser lang is set in the context (inside of
`Request._get_dbname_and_session`), it is not possible to verify it is
installed in the database as no connection to any database as been
established yet. Instead the lang is validated inside of
`ir.http._pre_dispatch` which is the method responsible to prepare/fix
various stuff on the request/session/context.
In regard to 93b684d3c7, we prefer to fallback on English, hence the
modification in `get_lang`.
closesodoo/odoo#116683
Reference-to: 93b684d3c7 ([FIX] base: lang should fallback on english instead of arab)
X-original-commit: 743e97667e44cdaab6db334d807cf3d409cea621
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
Some cookies were left around even when the user logged out
The next user should log into the default company instead of the company
of the last user.
task-3077421
closesodoo/odoo#108998
Related: odoo/enterprise#35685
Signed-off-by: Julien Castiaux <juc@odoo.com>
Commit [1] introduced the creation of a website in the tests, but
wrongly set `user_id` to the current environment user.
It doesn't make any sense as this field is supposed to hold the website
public user, which is not a real user and is supposed to be a public
user, as the field name hints..
On top of that, the current environment user (the demo user for tests,
which is wrongly set as the website public user) is then used to login
into the website, which is then making things even more weird/wrong: you
are not supposed to login with the public user (which is not really a
public user here but still marked as the public user from the website).
It was preventing the fix from this same PR (see previous commit) to be
working, as it was making this tour crash.
Not setting the `user_id` property will simply let the system create one
for you (a real public user..).
[1]: https://github.com/odoo/odoo/commit/3611aaf0983b3d25822022ebeddd781f6d082bf7closesodoo/odoo#107759
X-original-commit: 68fc99cf944090a37ee2d392990f39adb4899076
Signed-off-by: Julien Castiaux <juc@odoo.com>
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
One can very well select a "auth=user route" as homepage for his
website, like /my.
There would then be an issue with such an URL being set as homepage:
- As the user landed in the homepage controller which is auth=public,
the system will add the public user as env user (see
`_auth_method_public()`).
```
@http.route('/', type='http', auth="public", website=True, sitemap=True)
def index(self, **kw):
```
- Then, that controller will reroute to the homepage url (/my). The
request.httprequest.path will now be /my
- Then, that controller will recall the dispatcher stack:
`request._serve_ir_http()`
- From there, this call won't fail as it should because the user is
considered as logged in as it went already through the
`_auth_method_public()`, adding the public user as env.user.
`_auth_method_user()` won't raise its error.
- The /my page will be rendered despite not being logged in.
Once the user land on that page, the system will actually detect him as
logged out on a page supposed to be accessed when logged in.
It will then:
1. Show a toaster to inform the user
2. Auto reload the page
As the current URL is still `/` (due to the reroute and not a redirect),
the same flow will happen again, and again, looping forever.
--------
Note that accessing /my directly won't be an issue, as it won't go
through the homepage controller (which is auth=public), so there won't
be a user_id set on the env (the public user), meaning that the dispatch
layer will reject the access and raise an access error (through
`_auth_method_user()`.
Also note that the same behavior will occur when going through the first
menu fallback mechanism:
- if the user didn't setup any homepage_url
- and deleted his / website.page
- and his first website menu is /my
In that case, it will go through the first menu redirect fallback, which
will be working fine (as it's a redirect and not a reroute).
With both those 2 flows (going through redirect), the user will
correctly land on the login page (which will redirect to /my once logged
in).
------
Finally, the other solution would be to prevent such a configuration
(/my as homepage_url) but it was not easily doable (if doable at all),
see https://github.com/odoo/odoo/pull/99100#discussion_r963055251
------
A test is also added and over the more complexe case (which should cover
everything):
- With /my as homepage_url
- With the / website.page deleted
- With /my as first menu URL
-> Accessing / as public user should:
1. Reroute to /my (because of the homepage_url set to it) which
should fail now thanks to this commit
2. Since the reroute / re-serve failed, it should reach the
"first menu fallback" mechanism, which is also /my
3. That fallback should be a redirect, not a reroute, so the user
should actually land on the login page
4. Once logged in on that page, it should properly redirect to /my
opw-3077339
X-original-commit: 4f5899f769266edc27a0b2d812df092b95f43c71
Part-of: odoo/odoo#107759
This fix allows generating the sitemap (via `website.search_pages`) via
RPC. The changes are necessary as since #99667 the `request` object is
no more available to RPC-executed functions.
See also: the `test_search` test case of `website.tests.test_page`.
closesodoo/odoo#104567
X-original-commit: 8f4e214417cfb1aed7e595b8d6ab71f33a347b29
Signed-off-by: Julien Castiaux <juc@odoo.com>
*: im_livechat, survey, utm, website_crm_iap_reveal, website_forum,
website_livechat, website_sale, website_sale_comparison
Before this commit all cookies were considered essential.
This commit makes some of them optional. It also makes it possible for
the website visitor to only accept the essential cookies.
task-2800976
X-original-commit: 9a8a9463289a7446e9be0ef62ff895feb37a4de4
Part-of: odoo/odoo#101845
Co-authored-by: Benoit Socias <bso@odoo.com>
Translated fields no longer use the model ir.translation. Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table. The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages). Empty text is allowed in translation values, but not NULL.
Here are examples for a field with translate=True:
NULL
{"en_US": "Foo"}
{"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
{"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}
Like before, writing False to the field makes it NULL, i.e., False in all
languages. However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.
Here are examples for a field with translate=xml_translate:
NULL
{"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}
Change for callable(translate) fields: one can now write any value in any
language on such a field. The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values. Basically the
structure of the value must remain the same in all languages, like before.
Reading a translated field is now both simpler and faster than the former
implementation. We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:
SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...
The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages). For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.
Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language. Before this change, only the source value of the
field ('en_US') could be indexed.
Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages. We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.
Code translations are no longer stored into the database. They become static,
and are extracted from the PO files when needed. The worker simply uses a cache
with extracted code translations for performance. This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker. Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).
Performance summary:
(+) reading 'model' translated fields is faster
(+) reading 'model_terms' translated fields is much faster (no need to inject
translations into the source value)
(+) searching translated fields with operator 'ilike' is much faster when the
field is indexed with 'trigram'
(+) updating translated fields requires less ORM flushing
(-) importing translations from PO files is 2x slower
Some extra fixes:
- make field 'name' of ir.actions.actions translated; because of the PG
inheritance, this is necessary to make the column definition consistent in
all models that inherit from ir.actions.actions.
- add some backend API for the web/website client for editing translations
- move methods get_field_string() to model ir.model.fields
- move _load_module_terms to model ir.module.module
- adapt tests in test_impex, test_new_api
- because env.lang is injected into SQL queries, its returned value is
now guaranteed to correspond to a valid active language or None
- remove wizard to insert missing translations (no longer makes sense)
task-id: 2081307
Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Rationnals
----------
Web servers can serve some resources (e.g. static files) right away
without any interaction with the web application. The network model of
most web servers makes them capable of handling thousands of
simultaneous requests when it comes to intensive IO operations such as
streaming data from a file. The network model of Odoo is different: it
is capable of a lot of processing power but can only serve a handful of
requests at a time, i.e. Odoo (with some help from postgres) is
optimized for CPU operations, not IO.
Some users don't configure their web server, they use a basic
configuration that relay all requests to Odoo. The result is that many
Odoo HTTP Workers can be busy streaming static files instead of
processing other requests. This can lead to a worker starvation, i.e.
all workers are busy streaming files and cannot process new requests.
X-Sendfile
----------
In this work, we add the support for the [X-Sendfile] header family,
they are multiples http headers that can be used by the web application
to communicate with the web server in order to delegate the delivery of
files stored on the file system. Odoo still receives the request but it
does no more stream the file content from within its HTTP worker,
instead it skips the response body altogether and sets the `X-Sendfile`
special header with the path of the file on the filesystem. The web
server intercepts that special header, open the file and stream it.
Using those headers, we can use the best of both the web application and
the web server. The web application is still responsible to locate the
resource and verify the access rights, the web server is still
responsible of streaming the content.
Using X-Sendfile is opt-in via the `--x-sendfile` CLI flag. We set both
`X-Sendfile` (apache) and `X-Accel-Redirect` (nginx). If you are using
apache, make sure `mod_xsendfile` is enabled. If you are using NGINX
you have to add the following location block:
location /web/filestore { # custom path, hardcoded within Odoo
# Prevent access from the outside world, i.e. makes this
# route only accessible via X-Accel. MANDATORY!!!
internal;
# Give access to the filestore using this server's
# permissions. Odoo is in charge of verifying the access
# rights.
alias /path/to/odoo/data-dir/filestore;
}
The Odoo [deployment documentation] has been updated accordingly.
[X-Sendfile]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
[deployment documentation]: https://www.odoo.com/documentation/master/administration/install/deploy.html#serving-static-files-and-attachments
Changes to the API
------------------
To benefit most from X-Sendfile, all APIs related to streaming content
over HTTP has to be adapted. They are: (1) `request._serve_static`,
(2) `ir.http._serve_fallback`, (3) `/web/content` and (4) `/web/image`.
Each used it own way to deliver content: (1) `_serve_static` was using
`send_file` (flask's send_file that as been vendored with odoo 10
years ago and not maintenained since then), (2) _serve_fallback was
handcrafting a `werkzeug.wrappers.Response`, (3) /web/content-image were
using the "binary server" `ir.http.binary_content` API.
I has been decided to remove all 3 APIs and to merge the code inside of
the new `http.Stream` object and the `ir.binary` helper model.
A Stream wraps what is going to be sent to the browser, it can be a path
to a file on the locale filesystem, a blob of raw data or an URL to an
external resource. The Stream also holds various metadata that are
mainly used for caching. The preferred way to create a Stream is via one
of its three factories so that all the metadata are set. The factories
are: `from_path`, `from_attachment` and `from_binary_field`. A stream
instance exposes a single method `get_response()` used to create the
corresponding HTTP response object out of the stream.
Inside of `ir.http` were a few methods that were not related to the http
routing and formed what was called the "binary server". All those
methods have been removed and the feature have been refactored inside of
the new `ir.binary` model. The removed methods are:
- `_xmlid_to_obj`
- `_get_record_and_check`
- `_binary_ir_attachment_redirect_content`
- `_binary_record_content`
- `_binary_set_headers`
- `binary_content`
- `_response_by_status`
- `_get_content_common`
- `_content_image`
- `_content_image_get_response`
- `_placeholder_image_get_response`
The new `ir.binary` abstract model exposes the following utilities:
**`_find_record`**
Find an attachment or a record with a binary-field out of an xmlid or
out of a pair record-model/record-id. Check the access rights and the
access token.
**`_get_stream_from`**
Create a Stream from an attachment or a record with a binary-field.
**`_get_image_stream_from`**
Same as `_get_stream_from` but adapted for images. It sets a sensible
ETag on the stream and has image resizing support.
**`_placeholder`**
Get the image placeholder blob.
Testing
-------
It is possible to test the web server configuration using the
`test_http` module. Install the module then run the unittest using the
`webserver` test-tag. By default it attempts to connect to a web-server
running on `http://localhost:80`, you can change this URL by setting the
`WEB_SERVER_URL` environment variable.
odoo-bin -i test_http --stop-after-init
WEB_SERVER_URL='http://localhost:80' odoo-bin --test-tags webserver --stop-after-init
closesodoo/odoo#88134
Task: 2801675
Related: odoo/documentation#2083
Related: odoo/enterprise#26191
Signed-off-by: Julien Castiaux <juc@odoo.com>
Since PR #90855, we add extension if mimetypes doesn't match the type.
Since guess_type don't know some extension, we prefer considere all string
of less of 8 char as an extension before to fallback on the mimetypes lib.
With this commit, a filename filename.scss will be considered with an
extension .scss by our own helper instead of a fallback on .bin or .a as
returned by guess_extension of mimetype.
closesodoo/odoo#91905
X-original-commit: ca49c314970f32d051a449b0c0aa1aef30ca9a8b
Signed-off-by: Jérémy Kersten <jke@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
The file extension detection previously used to determine whether a
filename required an extension was not very smart and in fact only
checked for a dot in the filename.
`mimetypes.guess_type` is now used on the filename to better determine
whether the filename still needs a file extension or not.
This commit is a follow-up to: https://github.com/odoo/odoo/pull/90614
Which aimed to fix the same issue.
TaskId-2826061
closesodoo/odoo#91254
X-original-commit: bf904a7f98ba5bcc0a8909cca3241e2965c4464c
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Signed-off-by: William Braeckman (wbr) <wbr@odoo.com>
Every request comes with a session, a dictionary that is persisted on
the filesystem and that saves various information such as the user
cart on the ecommerce.
When a user simply visits the website, a default session is created and
saved on disk, this bloats the filestore with many sessions. Creating
the session on-the-fly is cheaper than loading it from the filesystem.
With this work the default session is not saved on disk anymore unless
explicitly asked via `session.touch()`.
An exception to the statement "creating the session on-the-fly is
cheaper" is geoip, the ip geolocalization is not cheap. In this work,
geoip have been moved from http_routing/request.session.geoip to a
lazy property core/request.geoip. When requested the info is persisted
on the session. Like other keys from the default session, geoip will not
be persisted unless there is non-default stuff in the session.
Because the CSRF-TOKEN is based on the session-id, it is important the
session-id stays the same across multiples requests even when the
session is not persisted on disk. Even when a session is not persisted
on disk, the session-id cookie is still set so that the next session
created on-the-fly uses the same session-id.
Technical note regarding the session, it has been decided to drop the
session-snapshot protocol and to reintroduce a "modified" flag. It has
been decided not to use werkzeug's session (which natively comes with a
"modified" flag) and to keep our own session object. We decided to
extend MutableMapping instead of dict; using MutableMapping we only
have to override __setitem__ and __detitem__; using dict we would had to
override update()/pop()/... too.
Task: 2789035
Part-of: odoo/odoo#86015
When a filename is given to the function binary_content it was shadowed
by _binary_ir_attachment_redirect_content if it is an ir.attachment.
To reproduce:
1. Start a brand new database in V14, install any app on which you can add an attachment (like Project or CRM)
2. Add a file as an attachment
3. Try to call the route "/web/content/<string:model>/<int:id>/<string:field>/<string:filename>"
closesodoo/odoo#87879
Solution: Use another variable to store the return value of _binary_ir_attachment_redirect_content and use it if the filename is not provided.
X-original-commit: 55e6097502da6169b85056b4769f4bf7fd230968
Signed-off-by: Wanderscheid Mathieu (mawa) <wama@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
Before this commit, any field requested to `_binary_record_content()`
when the model was `ir.attachment` would be considered as `raw`.
This assumption works at the moment but could lead to unexpected
behavior down the line.
see https://github.com/odoo/odoo/pull/86005#discussion_r824420219closesodoo/odoo#86524
X-original-commit: eeb93287f9a9027d0ce22f9fd3305a7ab081d69f
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Before this commit, since the refactor of some base64 decoding flows and
the addition of the raw field*, reading the content of a field that was
related to the `raw` field of `ir.attachment` was treated as reading a
base64 encoded content. This commit fixes this issue.
*https://github.com/odoo/enterprise/commit/680db8197731dae175027f81a705c38b0abf94da
task-2785146
closesodoo/odoo#86403
X-original-commit: 6aa984f072dde5667d07516888c6a4bf19aead8e
Related: odoo/enterprise#25272
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
This commit is the 11th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals. See also [REF] core: HTTPocalypse (9) ORM initialization.
Where `request._serve_db` is reponsible for initializing the ORM,
`request._serve_ir_http` is responsible to do the match -> authenticate
-> pre dispatch -> dispatch -> post dispatch sequence. This sequence is
very much similar to `_serve_nodb` with the notable addition of the
authenticate step.
Where `_serve_nodb` was working in a db-free environment, calling the
methods of (Http|Json)Dispatcher right away, `_serve_ir_http` delegates
to the `ir.http` abstract model which in turn use the dispatchers.
The ir.http model is crucial for several modules, web, portal and
website on top. They override the various methods to add advenced
capabilities to the http framework.
An important changement to the ir.http model in this work is regarding
the `_dispatch` method. Before this work it was that method that was
responsible of doing the sequence match -> pre dispatch -> ... . It was
decided to move the responsibility to http.py in order to prevent
modules from taking over the entire framework. One such take-over was
done in the `http_routing` addon and resulted in arguably very poor and
fragile code.
All modules must now carefully override the correct method. You need to
do smart stuff on the URL prior to matching an endpoint? You should
override `_match`. You need to save a special option from the query-
strings to the session before calling the controller? Override
`_pre_dispatch`.
PR: odoo#78857
Task: 2571224
Before this commit, when the current user is in the '/my/projects'
route and selects a project the route becomes '/my/project/<id of
project>'. There is no reason to change the route and then adding the id
of the project. In sales order route, we keep the route and we add the
id of the SO to see the portal form view of the SO selected.
This commit renames the route of project and task form view to keep the
same route and the same logic of SO and quotations routes for instance.
task-2648955
Closes#82379
Avoid to base64 encode, then decode to process assets and images for a ~25% speed improvement.
Change image processing tool to work on images, rather than base64 encoded strings.
Performance is ~25% faster on assets & images:
/web/assets/...frontend.min.css: 13ms to 7ms, base64 enc/dec: 2 -> 0
/web/image/XML_ID: 10ms to 8ms, base64 enc/dec: 3 -> 0
/web/image/res.users/2/avatar_128: 40ms to 20ms, base64 enc/dec: 6 -> 2
closesodoo/odoo#82851
Related: odoo/enterprise#23537
Signed-off-by: Fabien Pinckaers <fp@odoo.com>
Core client remains incompatible with CSP, however it can't hurt to
CSP the sub-resources.
Current scheme is simplistic, however if useful of necessary it could
be made more flexible e.g. there could be a map of mimetypes to CSP
configuration, that sort of things.
This branch adds request.redirect on all requests.
In case of a front end request, we do an url_for to the location.
We removed redirect_with_hash that was only for retro compatibility
local_redirect has been renamed to redirect_query, and param keep_hash has been
removed and moved.
Default code for redirect is 303 now instead of 302.
Now redirect and redirect_query make local redirect by default, you need to
pass local=False to make external redirect.
All werkeug.utils.redirect has been replaced by request.redirect.
Http.redirect now use an http.Response type, and it become easy to add an
override like 'set_cookies' e.g.
Dispatch of a website.page return an http.response too, so we first need to
check if it is a cached version before to check if it is an Odoo Response.
Migrate your code:
http.redirect -> request.redirect(location, code, local)
http.local_redirect -> request.redirect_query(location, query, code, local)
http.redirect_with_hash -> request.redirect
Courtesy of odony for help and review ;)
closesodoo/odoo#72599
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Before this commit, we only check if another endpoint can be serve in case of
404 not found. Idea was that each url has only one endpoint.
Fun effect is that if record doesn't exist, route doesn't match, so in reality
we had one endpoint but we fallback too to check if another match.
But it doesn't match a recurring case, when you archive a record, and want to
create a redirect for your seo and customer to another record, it is just
ignored because the url math a controller (that return 403 since you don't have
anymore the access right)
Since ask to the customer to delete the record is not always possible
(e.g. a product already sold) now we check if a fallback match for 403.
closesodoo/odoo#66030
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Use case: website_event_track(_online)
======================================
On sponsor model, image_128 field is defined in website_event_track module as
computed stored Image field. It means that when this binary field is set, an
attachment is created in ir.attachment model -> attachment is set to True.
In website_event_track_online module this field is modified. It is set as
a related (resized) version of image_512 and not stored anymore.
However attachments still exist in ir.attachment table for sponsor records.
This is not an issue when trying to access the image_128 field using the ORM
as the compute (or related) is correctly called. However '/web/image' does
checks if the field has an attachment and loads it. In our case old attachments
still in database are therefore displayed instead of the new related image.
Fix
===
When checking for an existing attachment in _binary_record_content we also
check that the field is not a related. This fixes the current issue. It also
makes _binary_record_content work as the ORM, aka using the computed value
and not any stored information.
DB Cleaning
===========
In 14.0 a script will be added to clean existing attachments for sponsor
model. Indeed there is no need to keep unused attachments. Especially in
14 website_event_track_online has been merged in website_event_track, meaning
only existing db have to be cleaned. New DBs will never have this attachment
issue as the field is always computed.
Task ID: 2341108
closesodoo/odoo#60259
X-original-commit: ada1eab338c6cffb526b03377ec01962f9c26c4c
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Odoo provides basic handling of CORS preflight requests: if an
endpoint is marked as `cors=<truthy value>` then it'll automatically
reply allowing the request.
*However* this is performed in `HttpRequest.dispatch` (likely in order
to correctly handle the nodb case), which means it's executed after
the auth handler has run... which means custom auth handlers will be
called on preflight requests.
This is a problem because they are missing relevant
information (e.g. which endpoint they're invoked for), plus having to
deal with preflight requests in every custom auth handler is annoying,
and simply allowing preflights could cause issues if the decision
diverges between the auth handler and the automatic
handling.
To fix this issue, extract the preflight *decision* into a separate
method so we get the same decision-making process everywhere, and in
case of CORS preflight set the auth to none to limit the eventual
capacity for nuisance in the span between the bypassed auth and the
automated preflight handling.
Also change the signature of IrHttp._authenticate so it's clearer if a
callsite was forgotten somehow (and this makes for less changes and
duplication at the callsites).
closesodoo/odoo#56029
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Before this commit, in case you were in --dev-mode, if a json request crash,
it was wrongly intercepted and return a Internal Server Error with status 500
and without the JSON Response
For the change of http code, since it is only in dev mode, it could not impact
a production server (in theory) with custom code based on it.
After this commit, if your rpc fails, you will not have anymore a breakpoint in
the code to help the developer to debug. But you will have a status 200 on the
rpc request.
closesodoo/odoo#54668
X-original-commit: 534cca311db1ecf1005a24d5cbd50e166094ba2a
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Http routing and websits module will already do it later and handle the case.
The only ModelConverter used that will be impacted is:
/web_editor/attachment/<model("ir.attachment"):attachment>
that is a json controller called with eisting data
task-2211013
Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
Werkzeug 1.0 adds a `merge_slashes` feature which calls rule.build()
*during match*[0] (= during dispatch).
However at this point our environment still contains an invalid /
placeholder UID, so `display_name` / `name_get()` fails dramatically.
One possibility would be to update the slugifying to not access the
record (outside of the id) if the environment is not "proper", an
other alternative is to just disable the feature since it seems to not
have been necessary so far.
The latter seems less hacky so do that for now, we can always swap the
solution later if we need to.
[0] https://github.com/pallets/werkzeug/blame/048cdfd9b969c0c3a133d7ff43b8ad1ad6a673ec/src/werkzeug/routing.py#L904
This reverts commit f7d7c53cd5.
Need to investiguate debug=assets mode.
Revert to unlock others devs
closesodoo/odoo#48675
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Similarly to method _binary_set_headers, when we serve an attachment
(e.g. image) where an unique is present in URL we add a long time of cache.
We considere that the hash will be changed if attachment is updated, so
while we ask for same checksum, lets the browser return the cached response.
closesodoo/odoo#48653
Signed-off-by: Christophe Simonis <chs@odoo.com>
If model is already an ir.attachment we don't need to do a new search_read.
We can use it record directly.
After this commit, we don't do extra request if we already have the info,
else we don't change the behavior.
task-2211013
Most of the time, the record exists, but we still ensure it does, every single
time, making an SQL query.
Doing a try catch will result in the same behavior, but won't make that query
most of the time.
task-2211013
When reading binary content such as `image_128` on `res.users`,
`AccessError` should be raised when necessary.
Steps to reproduce:
- Populate cache in superuser mode.
- Access cached field with public user.
- Read access is allowed but should not.
Concrete example:
- Unpublish `demo` user.
- Access `/slides` with `public` user.
- The template data is generated as `sudo`.
- The same data is then accessed as `public`.
- AccessError should be raised when requesting
`/profile/avatar/<int:user_id>` but is not.
Closes#43826closesodoo/odoo#45033
X-original-commit: e0112db4d6131751475365ab4c42de848f925dce
Signed-off-by: Christophe Simonis <chs@odoo.com>
The filename can be obtained in 3 ways when downloading a file, in order:
- by the filename argument
- by the filename_field argument
- a default one is computed as backup.
In the last case, there is by construction no file extension.
The filename is made from the record's model name, id and field.
However the model name almost certainly contains a ".",
which is the standard extension separator for filenames.
As a result <model_name_end-id-field> is considered to be the existing
extension, so we don't try to guess it from the mimetype.
To keep the existing default filename convention, we always add the
guessed extension in this case.
opw 2149612
closesodoo/odoo#41924
X-original-commit: f62a49a2f83bcce66eb9c3b08687ee5a37def503
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
Missing / wrong indented code during rewrite of binary controller:
https://github.com/odoo/odoo/commit/7d85ab1#diff-1407a8ce197a04eefaefa26c127a4418L343-L348
In case you have:
<record id="s_cover_default_image" model="theme.ir.attachment">
<field name="key">website.s_cover_default_image</field>
<field name="url">/web/image/theme_treehouse.bg_img_15</field>
</record>
theme_treehouse.bg_img_15 will be not found in /web addons.
So we need a 301 redirect too.
opw-live
How to reproduce:
odoo.com/trial > Website > Install theme Clean
Drop first snippet > background is 404
closesodoo/odoo#40605
X-original-commit: b8ce93e4eb397e1b7a0e1d408d61d2f283223fb9
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Initial b6ed34e1 idea was to ensure that we always have
_rewrite_len and _routing_map defined. Unfortunately, this
is not correct since class attribute defined dynamically
are actually added in registry, and recomputed on each install.
More than that, the call si shared between multiple database
meaning that routing_map cache may be shared between multiple
database whcich is not correct.
After b6ed34e1, when installing discuss on a fresh database without
demo, all discuss routes will be unknown since None is already in
_routing_map and thus routing_map is not recomputed.
When routing_map is on the model class, in registry,
a new install will reset the class, remove the _routing_map attribute,
which will fix the problem.
Task #2117275closesodoo/odoo#39640
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Actually, the two class attributes are only set when calling the
routing_map method. As a consequence, when the url_for function in the
http_routing module tries to access the _rewrite_len attribute, it
may crashe.
This problem arrise in the TestQwebProcessAtt from the website module
when the test is run alone.
It was hidden on the runbot because some missconfigured HttpCase tests were run
at_install before TestQwebProcessAtt. In that particular case, the
routing_map method is called and sets the attribute.
With this commit, the _routing_map and _rewrite_len class attributes are
set on the class at class declaration time.
Commit 5a9e1af64a has the unfortunate side-effect of crashing early
if for any reason the content cannot be decoded.
However, simply ignoring that the content cannot be decoded is no better idea:
some functions pipe the result to decoding functions that crash the same.
The resulting traceback pollutes the log with uninformative message such as:
binascii.Error: Incorrect padding 5 0.002 0.016
In case the content cannot be decoded (data corruption, or simply missing file)
we return a clean 404 instead, which is morally almost equivalent,
and is clean even from functions that depend on binary_content.
opw 2072586
closesodoo/odoo#38318
X-original-commit: a68f7e6e72dbfd44e14c2f293fa6dc6aa402308a
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
Redirect_type was renamed from 3xx to redirect_3xx, but name is used as http
status code. It is better a redirect(301) instead of redirect(redirect_301)
Re-add the bind_to_environ, it was removed because it seems not useful, but
once you are behind a proxy, werkzeug need the environ to find the correct
hostname and more ... Transformed relative url into absolute by werkzeug was
wrong.
closesodoo/odoo#37733
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
After this commit, you will be able (in technical mode) to update the url for
the python controllers.
Eg.
You can now rename /shop in /garden and /shop/product/ in /garden/vegetable/
Most of urls will be replaced at fly in the renderd qweb, with the function
url_for but all old urls will keep available. So if you access url /shop you
will be automatically redirected to /garden (308 Permanent Redirect).
As for cdn and other post-process of att, the automatically replacement in the
rendered qweb is only done when you will be not website editor. But the new
dispatch of URL will be applied in all cases.
For developper, since it is Permanent Redirect, don't forget to clear cache or
open chrome debug tool (with option 'Disable cache while DevTools is Open) to
see your lasts changes.
closesodoo/odoo#36555
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
* base, web
Google now recommends a TTL of one year for static contents. We used to
use 1 week in almost every case. This commit increases that value to one
year for safe resources, like assets bundles which contain a specific
hash in the URL which changes if the bundle is recomputed anyway.
Note: this commit refactors the code so that both the one week and one
year durations are defined in http.py and used by others apps. Loading
the library "locale" file used to be done with 10-hours-cache, this has
been increased to 1-week-cache by using the http.py STATIC_CACHE var.
closesodoo/odoo#37402
Signed-off-by: Quentin Smetz (qsm) <qsm@odoo.com>
* = stock, test_website, web, website_forum, website_slides, base
Replace KarmaError with AccessError and remove the related override made
on crash_manager and ir_http.
task-2069890
closesodoo/odoo#36655
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
SHA-1 is a cryptographic hash function that have weaknesses known since
2005, it has been deprecated by the NIST [1] about 10 years ago in 2011
and Google [2] have been able to perform a collision attack in 2017.
We use SHA-1 in order to generate unique URL for resources that can be
cached by the browser: assets bundle, translations, qweb templates and
qweb images.
Although practical attacks still requires quite a lot of computational
resources, it is time to upgrade SHA-1 to SHA-2.
We have selected the SHA-512/256 variant of the SHA-2 algorithm as
replacement for SHA-1 for the following reasons:
* On 64 bits platform, SHA-512 is the fastest SHA-2 variant, it is only
~1.5x slower than SHA-1. [3]
* Keeping only the 256 foremost bits protects against both collision
attacks and length extension attacks.
* The hexadecimal digest is only 24 chars longer than SHA-1 which is
nice to have somewhat short URLs.
We have not used SHA-3 because:
* At the moment of writing, it is too slow (~3x slower than SHA-1) [3]
* It is not guaranteed to be available with the Python 3.5 `hashlib`
module.
* One of the author of SHA-3 is Belgian.
[1] https://csrc.nist.gov/projects/hash-functions/nist-policy-on-hash-functions
[2] https://shattered.io/
[3] http://bench.cr.yp.to/results-hash.html
[4] http://www.commitstrip.com/en/2017/02/27/the-sha-1-alternative/
This branch is the combination of several optimizations in the ORM:
* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;
* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;
* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);
* make method `modified` take advantage of inverse fields to inverse
dependencies;
* filter records by evaluating a domain on records in Python;
* a computed field with `readonly=False` behaves like a normal field
with an onchange method;
* computed fields are computed in superuser mode by default.
Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.
closesodoo/odoo#35659
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
Martin Trigaux