* = gamification, test_website, stock
- KarmaError is now handled as a 400 exception.
- test_website has been updated.
- NO_POSTMORTEM is now clean it was referencing duplicates as most the
exceptions inherit from except_orm.
- serialize_exception from http.py has been moved to ir_http
to take advantage of the odoo inheritance system. We can then
extend ir_http serialize_exception method to add the KarmaError
logic if and only if gamification is installed.
Places where serialize_exception was previously used are updated.
Part of https://github.com/odoo/odoo/pull/32132
task-1894820
* = account, iap, point_of_sale
Removing the CrashManager templates from base.xml and moving
it to a new file (crash_manager.xml) to allow lazy loading the required
xml only when a crash occurs (necessary for the frontend as no xml is
pre-loaded).
All rpc errors/warnings are now displayed using the same modal system as
the backend (crash_manager.js).
CrashManager will now return the WarningDialog and ErrorDialog
in addition to the CrashManager itself.
In some cases an instance of CrashManager was created just to
display a warning/error dialog.
In the backend all rpc errors but sessionExpired will be logged.
If the exception is a warning, NotFound (403) or except_orm, it will be
a warning log. The rest will be exception log.
In the frontend all rpc errors will be logged using console.debug
instead of console.warn and console.error. This way we can trigger
rpc errors in tests without failing them. If a real error happend
during a test it will fall back on the backend logger to fail the test.
Part of https://github.com/odoo/odoo/pull/32132
task-1894820
On `WebRequest` `__exit__`, when an exception occured,
(in `self.registry.signal_changes` or `self.registry.reset_changes`)
cursor were left unclosed as `self._cr.close` was not called
in such cases.
Having exceptions in the above mentioned method do not happen
often, but when it does it left unclosed and unusable cursors
in the connection pool, and in the extreme case explained below,
it left the connection pool with only unclosed and unusable cursors.
The entire server was then unusable as it no longer had working cursors.
Case:
- Start a multi-thread server with db_maxconn set to 5
- Ensure you do not send any request to the server,
not even with a left open tab on `http://localhost:8069` in your browser
- Send 6 parallel HTTP requests to `/web/login`
thanks to an external thread python script
(See below, at the end of this long commit message)
According to your registry state (if you have a lot of modules installed or not),
and the native Python Garbage Collecting state,
you might end with
- either warnings telling some unclosed cursor were garbage collected,
and therefore closed (by a kind of luck thanks to the Python garbage collecting),
- either, a server completely blocked not accepting any other request
(you can try for instance `curl http://localhost:8069`
and you end up with a `500 Internal Server Error`
This observed issue looks to appear only in 11.0. Not 10.0 or 12.0.
This is because only 11.0 clear the cache during registry loading:
`https://github.com/odoo/odoo/blob/f1706c848d41c47646dabca771996e9b9f788241/odoo/modules/loading.py#L236`
This cache clearing doesn't happen in 10.0 nor 12.0
(in 12.0, thanks to e181f592f3)
When sending the 6 parallel requests,
it uses instantly all the 5 available cursors of the connection pool to handle these requests,
and when each request exits, in `__exit__`, it calls `self.registry.signal_changes()`
which tries to open a new cursor because of
- `self.cache_invalidated` which is True, for all the 6 requests, thanks to the call to `clear_caches`
explained above during the registry loading and the fact all requests have been treated in parallel,
- `with closing(self.cursor()) as cr:`, `self.cursor()` attempting to use a new cursor
(the `closing(...)` does not have any incidence on this issue, despite it could look like guilty)
The attempt to use a new cursor fails, as there is no more available (`db_maxconn` is reached),
raising a `PoolError('The Connection Pool Is Full')` exception.
In the request `__exit__` method, because of this exception raised when calling `signal_changes`,
`self._cr.close` is never reached, and the parallel request therefore left only unclosed
cursors in the connection pool,
therefore leaving the server in a state where it only has unusable cursors
and therefore can't do anything more.
This might look like really bad luck to land in such a state,
but we observed multiple actual case on Odoo.sh,
the one referenced in this commit (opw-2008340) was because of an Outlook client
which launched 18 parallel requests to fetch the email images,
and the server wasn't spawned, therefore neither was the registry.
The server registry was therefore just loaded when it received the 18 parallel requests,
and it therefore triggered this extreme use case.
The server was left unusable for several minutes, until a forced restart.
For reference, here is the script that has been used to trigger the 6 parallel requests:
```
import requests
import threading
threads = []
for i in range(6):
threads.append(threading.Thread(target=lambda: requests.get('http://localhost:8069/web/login')))
for thread in threads:
thread.start()
```
opw-2008340
closesodoo/odoo#34071
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
It seems to be a better solution to keep starting tests without being in debug
mode to avoid having new fields and menus appearing.
This commit simply drop the debug mode in test mode and retrieve the test
assets by checking if test mode is enabled directly.
closesodoo/odoo#34012
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Before this commit, call a controller defined as:
```
@http.route('/route', type='http', auth='public')
def controller_func(self, foo):
do_it()
```
and called with url like /route?foo=1&bar=2
will crash with an exception:
`TypeError: controller_func() got an unexpected keyword argument 'bar'`
Now, we remove the extra parameters if the controller doesn't support it.
This case is not uncommon, you can easily arrive in this case with utm or
debug as extra parameter.
closesodoo/odoo#33962
Signed-off-by: Christophe Simonis <chs@odoo.com>
This commit goal is to encapsulate every tour-test into a separate assets
bundle that would be called only during tests (command line) or by URL
(debug=tests). That way, a lot of .js files would not be loaded anymore
uselessly outside test mode and will speed up the page loads (especially in the
frontend as the backend do not reload the page anyway).
In order to do that, we needed to propagate the `debug` state from page to
page.
Otherwise, every page change during a test would simply lose the debug mode and
the test would stop as the test assets would not be loaded.
As we could not add the `&debug=tests` on every link (either hardcoded or
preprocess during the rendering), it has been decided to store it in session.
Technical summary:
1. `debug` state (currently only stored in URL) will be stored in session.
Either when adding the `debug` param in URL (handled with _dispatch) or by
starting Odoo with `test-enable` or `test-file` (handled by session init).
2. Once activated (and so set in session), debug mode will remain activated
even if not visible in URL (after a page navigation eg).
To deactivate it, set its value to nothing, `debug=`. That will exit debug mode
(whatever mode it is: debug, assets, tests).
3. As tour-test files are now in a separate bundle, every layout (when needed)
should `t-call="web.conditional_assets_tests"`.
As it would be redundant and verbose, no `t-if` is needed on the t-call to
load it only in test debug mode. That will be handled by
`compiled_assets_tests` that will actually do the conditionnal t-call-assets
to web.assets_tests, if tests debug mode is activated.
4. In addition to separating the tour-tests files in a separate bundle, we also
moved those files to a specific folder under /static/tests/tours next to
QUnit tests.
5. Also, tour files will be moved in a specific folder /static/src/js/tours for
cleanness purpose (those files will still be kept in their 'normal' assets
as needed outside test mode since it is tours).
This will be done in the next commit.
6. It is possible to enable multiple debug mode, such as 'tests' and 'assets'
together. Simply separate debug modes with a comma, eg '?debug=assets,tests'
task-1934445
Comes with https://github.com/odoo/enterprise/pull/4281Closes#33213
In the HTTP pipeline, there is a middleware called DisableCacheMiddleware
that is called on every request to check if we are in debug mode, to remove
the cache headers and replace them with a static 'no-cache' so the browser
has to contact the server and check for a new version of every ressouce.
There were 2 problems with it:
1. it was doing more work than strictly needed when not in debug, like
rebuilding a list of headers for every request
2. it was removing Etag (case sensitive) with a typo and other headers
that could have been left in place.
This commit fixes those 2 issues by only modifying the headers if we are
in debug mode, and only removing the header 'cache-control' and replacing
it with 'cache-control: no-cache'
* = website_profile, website_slides
Before this commit, the returned Content-Type was not always correct, for
example if the image tool was changing the format, which happens when given
a BMP (converted to PNG), or other types being converted to JPEG.
The previous method `force_contenttype` was too specific and it wasn't available
in every controller. It didn't even need to be a controller method because it
didn't use self.
Now we create a generic helper to ease updating headers.
The Content-Type is only updated when it is safe to do. It is especially unsafe
for example for SVG files.
task-1958000
PR: #31811
JSONP can be entirely replaced by the CORS mechanism, which is simpler.
We support CORS in our routes since 8.0 (odoo/odoo@9cce88a), so it's about time
to get rid of JSONP.
- This commit fixes a crash that happens when the server receive a
JSON-P call done in two requests (a POST followed by a GET).
The issue is due to the fact that when the first request is done (POST
one) the member `params` is never initialized.
This parameter is then used in the module `auth_signup` on an override
of the method `dispatch` thus crashing the code.
To avoid the crash, we now initialize `params`.
closesodoo/odoo#27369
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
reuse current_thread result
No needed to call threading.current_thread() on evry arg setting
closesodoo/odoo#32000
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
This revision aims to support the memory limits
on Linux, Windows and MacOSX according to their own spefication
regarding their memory management.
Among others, it brings the possibility to
use the multi-workers mode on Windows and MacOSX.
e.g.
- Windows does not support `resource`,
and therefore we skip to set the hard limit
- MacOSX allocates a large virtual memory for each process
even if the memory is not actually used,
and therefore using the VMS to limit the memory is pointless
as it will always exceeds the default soft memory limit.
We therefore choose to use the RSS to limit the memory
closesodoo/odoo#27848
Session rotation was introduced a long time ago, but deactivated at
login due to obscure side-effects related to #6949 (aka the "BBQ" PR).
This commit reinstates the rotation, which is better from a security
standpoint.
In order to also prevent session ID reuse, we force the renewal of
deleted sessions, at the SessionStore level (via `renew_missing`).
When there is a performance issue, it's sometimes difficult to discover
which request increased the query count or its duration.
With this commit, the query count, the query time and "python and io" time are displayed
in the logs at the end of each werkzeug request line.
Co-authored-by: Christophe Monniez <moc@odoo.com>
From this commit onwards, Date fields will return datetime.date objects and Datetime fields will return datetime.datetime objects, this implies a number of things that are clearly explained both in the ORM API for master.
This commit also introduces a number of helper functions for dates and datetimes that are exposed in tools.date_utils and fields.Date[time], explained in the documentation as well.
Task-ID: 47189
* Make Users._login and session.authenticate always raise AccessDenied
on authentication failure instead of only sometimes (cf
Session.authenticate calling security.check() which raises and not
catching the exception)
* Alter AccessDenied such that it's possible to add a custom access
message, for use with login rate limiting instead of smuggling the
information via the session
* Alter the RPC endpoints to catch and convert AccessDenied back to
a boolean sentinel
Browsers accept it as they are wont to do, but the Content-Disposition
lib (introduced in 35d452cffb) does
not. Simply don't mark them as safe when %-escaping the filename so
they do get %-escaped properly.
RFC5987 states
ext-parameter = parmname "*" LWSP "=" LWSP ext-value
ext-value = charset "'" [ language ] "'" value-chars
value-chars = *( pct-encoded / attr-char )
attr-char = ALPHA / DIGIT
/ "!" / "#" / "$" / "&" / "+" / "-" / "."
/ "^" / "_" / "`" / "|" / "~"
; token except ( "*" / "'" / "%" )
Neither : nor / are in attr-char. This is further confirmed by
checking out RFC2616:
CTL = <any US-ASCII control character (octets 0 - 31) and DEL (127)>
token = 1*<any CHAR except CTLs or separators>
separators = "(" | ")" | "<" | ">" | "@"
| "," | ";" | ":" | "\" | <">
| "/" | "[" | "]" | "?" | "="
| "{" | "}" | SP | HT
Here we can see "/" and ":" are both in "separators", which are
specifically *not* in token. attr-char restricts token further, so an
invalid token char can't be a valid attr-char.
- Each time we check if a session is valid we create a new cursor.
This could lead to issues with db_maxconn that limits the number of
connections to the postgresql server.
In a perfect world, a worker should use a single connection to
postgres to process the request.
The only known side effect is that the cursor is created earlier in
the execution of the code.
- This commit fixes issues with the longpolling raising
Psycopg2.PoolError exceptions on databases with a lot of clients.
Endpoints can be explicitly marked as `save_session=False` (default is
true across the board). In that case they will have an in-memory session
(either the existing one or a brand new one) but the session won't be
persisted to disk.
Currently used for non-browser RPC endpoints: the APIs don't use
cookies/sessions and we can't assume the RPC libraries keep cookies
across calls. This means a new session is created and saved to disk for
each RPC calls, for no useful reason.
Before this patch, early connections made to a 11.0 Odoo server running
on Python 3 and deployed in threaded mode with socket activation would
generate invalid registries.
With this patch it is now possible, when this deployment mode is used,
to opt-out addons preload by setting the following environment variable:
ODOO_PRELOAD_ADDONS=no
Note: this environment variable is only available for v11.0 as later
versions does not preload anymore (cf: 1a39c9b)
This way, XMLRPC calls can get request details form the standard
`odoo.http.request` system.
Move jsonrpc to the same controller while at it, for coherence.
Fix#24183
Replace `httprequest.stream.read()` by `httprequest.get_data()` so the
payload content remains available: get_data stores the request body (by
default) so it remains available for alternative processing or checks
(MAC checks for webhook validations for instance). With `stream.read()`,
once the data is read if it's not stored separately it is lost.
- Install Website
- Load the 'Norwegian Bokmål' translation, and choose to translate the
website
- Logout
When accessing the website, the language displayed is not consistent to
the browser language.
1. `no` shows homepage in `nb_NO` (Firefox only) => expected since
Bokmål is the main language in Norway
2. `nb` shows homepage in `en_US` => unexpected
3. `nn` shows homepage in `en_US` => expected since 'Norwegian Nynorsk'
is not a language available (this will be the topic of another PR)
The issues comes from Babel's side, since `nb` is not in the
`LOCALE_ALIASES` while `no` is. We monkey-patch the value to avoid this
while Babel is corrected.
opw-1827258
fja-odoo