1. move cache to _get_asset_paths
The `_get_asset_content` cache has many cache key that are related to a
posprocessing of the `_get_asset_paths` result, the heavy part of this
method. Moving the cache to _get_asset_content will have the benefit
to create less duplicates entries in the ormcache as well as less cache
miss.
To simplify even further, the css and js parameters are removed since
they only filter the output of get_paths, the heavy part of globing the
file will be done before that. Anyway, they are both true when called
from _get_asset_content, and the only other call, in
`_get_related_bundle` don't really need to filter them since it is not
a critical part regarding performance, and the funtional result will
stay the same.
The initial orm cache key was using `_get_template_cache_keys`, a little
overkill and possibly creating duplicates entries again. The only
context key needed is website_id for `_get_related_assets`.
Note that it is not really enough, the orm cache key should actually
contain `request.session.get('force_website_id')` as well has
`request.httprequest.host`. This will be addressed latter since a nicer
solution would be to have website_id as a unique parameter computed
earlier.
2. better _get_asset_paths cache key
The orm cache key was simplified in previous point but there is still
one concern, the website_id depends on more parameters than that:
- request.session.get('force_website_id')
- request.httprequest.host
- existing websites
The idea here is to call `get_current_website` instead of using all
parameters that could define the webiste.
In the same spirit of `_get_template_cache_keys` `_assets_path_params`
can be overriden to give extra params that are usefull to list assets
path. Those params are computed before entering the method
`_get_asset_paths`. This may latter put at a higher level latter, in
get_asset_node, to simplify the _generate_asset_nodes_cache key.
3. better assets_node caches key
The main purpose of this part is to improve ormcache containing assets
nodes. The ormcache key contains
- to much context key
- missing session/host/env info
- unwanted boolean options.
- keys leading to the same cache value
The main goal being to reduce the size of the cache keys, decrease the
number of cache entries and improve the cache hit.
This will also make the behaviour more coherent and hopefully less bug
prone because of mismatch in parameters.
The main reason of the orm cache is the slowness of the validation of
the assets. This includes:
- listing files (dedicated orm cache)
- computing version
The cache key was depending on
- `debug`
The only relevant value for debug is "contains assets"
We dont need to differ between debug='', debug='1', debug='test',
and 'debug=assets', 'debug=tests,assets', ...
- `defer_load`, `lazy_load`, `media`
Those values are only useful to generate html node, a leightweight
operations that does not really needs to be in cache. `media` was also
used in the generation but it looks useless if we have the media on the
node. THIS NEEDS TO BE VALIDATED but in any case, since media is not
used to generate the url, it doesn't make sence to use it in the
generation.
The main idea to remove them from the ormcache key is simply to generate
the nodes outide the ormcached values.
-`async_load`
This one is similar to `defer_load` and `lazy_load` but it looks like
it wasn't used anymore. This was simply removed
- context.get('lang')
The only information needed is the direction, rtl or ltr. This means
en and fr languages, despite sharing the same css assets, will duplicate
the ormcache entries.
-`_get_template_cache_keys`
Only the lang and webiste where really relevant in this flow. Other
keys are actually useless in this flow.
Some information used in the generation where not in the orm cache key
- `self.env.user.lang` if there is no lang in the context
- `request.session.get('force_website_id')`
- `request.httprequest.host`
- ...
The proposed solution is to:
- extract any informùation needed from thecontext, request, environment
before entering the ormcache, reduce it to the minimal possible set of
values needed
```
rtl = self.env['res.lang']._lang_get_direction(self.env.context.get('lang') or self.env.user.lang) == 'rtl'
assets_params = self.env['ir.asset']._get_assets_params() # website_id
debug_assets = debug and 'assets' in debug
```
and remove a leightweight part of the logic
```
def _get_asset_nodes(self, bundle, css=True, js=True, debug=False, defer_load=False, lazy_load=False, media=None):
links = self._get_asset_links(bundle, css=css, js=js, debug=debug)
return self._links_to_nodes(links, defer_load=defer_load, lazy_load=lazy_load, media=media)
```
Where _get_asset_links is the cached part, and _links_to_nodes is the
lightweight part generating the nodes based on the `defer_load`, ....
Additionnal notes:
- data-asset-version and data-asset-bundle are removed from the node
since they don't seem to be used anymore since 65d70acdbf
- async_load is removed since there is no occurence of this in the code.
- a small hack is still needed to pass javascript content instead of
links, this is only to manage css compile error and will hopefully be
removed in the future.
- a context key is still in use to generate the bundle, the
`commit_assetsbundle` but it has no impact on content and will hopefully
be removed in the future.
4. Add test for ormcache hit/miss
In this context, hit/miss is about having the same cache key for the
same result. This test demonstrates the current state, were entries are
create in the ormcache only if the key is really different and will lead
to a different result.
5. remove cache invalidation
This cache invalidation is quite agressive since everytime an
assetbundle is updated, all workers will clear their cache.
The concerned cache by this clear_cache is `_generate_asset_nodes_cache`
throug `_get_asset_nodes`.
The cache is ignored, both in dev=xml and debug=assets.
This clear cache was made conditionnal in 553ea82f81 but this does
not solve an issue we can have in production.
Lets imagine a clean solution
- all sources are updated
- all workers are restarted.
The orm caches are all empty, but since the sources
changed, all bundles will be recomputed. This means that every bundle
updated in database with save_attachement will invalidate the cache of
all workers. Rendering a pdf report of any kind using a specific bundle
will invalidate all cache. Starting a debug=assets for the first time
will invalidate all cache, even if the cache is not used in this case.
But for a regenerated bundle we would expect the ormcache to be:
- empty (did not generate the same bundle yet)
- have the same value (concurrent generation of the same bundle)
Having a different value would mean that the bundle was generated with
another version of the sources. In this case it is maybe even better not
to invalidate the cache since it could lead to an invalidation war
between two workers.
The only case where invalidating this cache is useful is when a bundle
changes, Usually if an ir_asset is created, modified, ...
There is still another rare but possible possibility to have a 404 if
the transaction is rollbacked after populating the assets node cache.
In this case, we only need to clear the cache locally in case of
rollback.
Part-of: odoo/odoo#121376
Previous behavior: when debugging the websockets worker, source code is always minified.
Now, if user is in `debug=assets` mode, the asset won't be minified and it will be easier to debug.
@moduon MT-1900
closesodoo/odoo#109583
X-original-commit: 6360bbdbdcc6f2c150ceaa9f7fb5b0760e0f8940
Signed-off-by: Julien Castiaux <juc@odoo.com>
Co-authored-by: Julien Castiaux <Julien.castiaux@gmail.com>
Before [1], the bus was started lazily: either as a consequence of
the addition of a channel to listen to or by manually calling the
`startPolling` method.
Before this commit, the websocket would have been started as soon as
the bus service starts which degrades performances.
This PR fixes the issue by re-introducing the same mechanism as before
that is by starting the websocket either by calling manually the `start`
method of the bus service or automatically when adding a channel.
[1]: odoo#75510
closesodoo/odoo#107878
X-original-commit: 5d7deacf54f37f0938b92a3c45c9f1d1325d1a9f
Signed-off-by: Alexandre Kühn (aku) <aku@odoo.com>
Signed-off-by: Stockbauer Matthieu (tsm) <tsm@odoo.com>
Before this PR, the last notification id known by the client
was not reset after restoring the database. This was an issue
since there can be a gap between the last notification and the
one that has been restored with the database. In this scenario,
messages are not received after restoring the database since the
client subscribes to higher notification ids that the ones that are
created.
This PR fixes this issue by defaulting to 0 if the one the client
passed is higher.
closesodoo/odoo#103311
X-original-commit: 5118da53a6e9adb94577a4f16999522a7e1e8c10
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Signed-off-by: Stockbauer Matthieu (tsm) <tsm@odoo.com>
The session used to call this websocket route needs to have been
initiated with a call to /websocket/peek_notifications first.
closesodoo/odoo#102009
X-original-commit: 49aa391b4ef93b2de578c73a1fe852cea32abb43
Signed-off-by: Julien Castiaux <juc@odoo.com>
In order to know whether or not the websocket session is up to date,
the session is saved when opening a websocket connection. Then, for
each incoming/outgoing message, we check if the session still exists
on the file system. When it does not exist anymore, the websocket
connection is refreshed. This ensures the session is always up to date.
Odoo sh proxies the websocket connection and needs a way to tell
when a websocket session is expired. This commit adds the same
check that is done for each incoming websocket message in the
websocket peek route in order to raise a `SessionExpiredException`
when the session is outdated.
This will allow odoo sh to catch this error and to refresh their
websocket connection accordingly.
closesodoo/odoo#100416
Signed-off-by: Julien Castiaux <juc@odoo.com>
Transform the /websocket/update_bus_presence route to type='json'
which makes more sense as a POST than a get, seeing it does things with
a parameter and doesn't return anything.
Also fixes the _update_bus_presence method that leads to access errors
when accessed from HTTP when the user is the public one (only portal/
internal users should update their presences).
Part-of: odoo/odoo#100416
*: bus, hr_holidays.
The `/bus/im_status` route polls the server every minute in order for
the user im_status to be up to date. This commit removes this poll
by sending the im_status on the bus when updating the current user
presence.
Moreover, before [1], the user bus presence was updated on each poll.
When the user didn't poll for 50 seconds, we assumed the user was
disconnected. Since [1], the bus presence is updated each 30 seconds
by the `im_status` service. This is too frequent: there is no need
to update the user presence so often.
In order not to overhelm the server with unnecessary requests, the update
presence interval as well as the delay to be considered disconnected
have been updated: the former from 30 to 60 seconds, the later from
55 to 65 seconds (assuming that a user that has missed an update
presence tick is disconnected).
[1]: odoo/odoo@a5623d2closesodoo/odoo#100249
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Before [1], only string channels were allowed for polling. This
ensured no one could send a server-side channel from the frontend,
this PR restores this behavior.
[1]: odoo/odoo@a5623d2closesodoo/odoo#100309
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
When executing python tests with Odoo opened in the browser, the
websocket connection coming from the browser can lead to issues
with savepoints/rollbacks. Indeed, during test set up, a savepoint
is created. This savepoint is released at the end of each test.
The issue occurs when the websocket connection opens a cursor (and
thus creates a savepoint) after the one created by the test but releases
it after the one created by the test:
- SAVEPOINT TEST
- SAVEPOINT WS
- ROLLBACK TO SAVEPOINT TEST
- SAVEPOINT WS DOES NOT EXIST
In order to solve this issue, let's prevent browsers from opening a
websocket connection during python tests. This does not apply to chrome
headless since remaining threads are awaited before the end of every tour
(which means the websocket connection will be close before releasing the
test cursor).
closesodoo/odoo#99538
Signed-off-by: Julien Castiaux <juc@odoo.com>
*: bus, calendar, hr_attendance, iap_mail, im_livechat, project, snailmail,
snailmail_account, survey, web, web_editor, website_livechat.
This commit is part of the websocket integration in Odoo.
This bus service now communicates with a shared worker in order to provide
a single websocket connection for multiple tabs. It is designed to
be used as a websocket except that events are slightly different,
re-connection is handled automatically. If the browser does not support shared
worker (Safari), the service fallback on a simple web worker.
Available events are:
- connect : fired upon a successful connection.
- disconnect : fired upon reception of the websocket close event.
The close code and reason are given to the listeners callback.
- reconnect : fired upon a successful re-connection.
- reconnecting : triggered when the worker starts to try reconnecting.
- notification : fired upon the reception of notifications.
Since multiple tabs are now handled by a worker, the cross_tab bus is no longer
required and has been removed.
Part-of: odoo/odoo#75510
*: hr_presence, web_editor.
This commit is part of the websocket integration in Odoo.
It focuses on adapting the bus to support websockets:
- last notification id is now kept on the server
- channel list is built by overriding the `_build_bus_channel_list`
method of the `ir_websocket` model instead of overriding the `_poll`
method of the bus controller.
- The bus presence was updated during polls, since there is no more poll,
bus presence update will be the responsability of the client.
- The `/websocket/peek_notifications`, `/websocket/update_bus_presence`
routes will be available so that odoo sh can access notifications/update presence
from http requests.
- /longpolling routes are now prefixed with /bus thus won't be redirected to the
gevent worker anymore except for `/longpolling/health` which is the
health check route of the gevent server.
Since websocket now handle incoming messages, a way to manage authentication
have been introduced :
- The session is retrieved from the HTTP handshake.
- When a websocket message comes/leaves the session is retrieved
on the file system so that we're sure it still exists and that
it is up to date.
- The session is checked
- If no session is found on the file system or `check_session`
fails, the websocket connection is closed with the `SESSION_EXPIRED`
close code (which is a custom close code: 4001).
- Note that websocket connections are closed every `KEEP_ALIVE_TIMEOUT`
seconds to ensure no websocket connection will stay open if the user
clears its cookies.
- Note that a wsrequest object is available when processing incoming
messages. It is similar to the http request and contains various
useful informations (session, env, ...).
Part-of: odoo/odoo#75510
This commit is the first commit of the websocket integration in Odoo.
It focuses on the implementation of the websocket protocol as per RFC6455.
The implementation is tested thanks to the autobahn test suite.
A config parameter is available to customize the websocket connection:
- websocket_keep_alive_timeout (default 600): Integer specifying how
many seconds a websocket connection should be kept alive
Part-of: odoo/odoo#75510
This commit is the 14th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
* `request.uid = x` => `request.update_env(user=x)`.
* `request.context = x` => `request.update_env(context=x)`.
* `request.context = dict(request.context, x=y)`
=> `request.update_context(x=y)`.
* `request.cr = None` => `request.cr.close()`.
* `http.mono_db()` => `request.db`.
* `http.dispatch_rpc()` => `service.dispatch_rpc()`.
* `@service.model.check` => `service.model.retrying()`.
* `request.endpoint`
=> `env['ir.http']._match(request.httprequest.path)[0].endpoint`.
* `request.routing_iteration `=> `removed`.
* `request.jsonrequest` => `request.dispatcher.jsonrequest`.
Note that `request.params` is now set much later in the process. If you
are in a situation where you values from the query string or the
http body you can use `request.get_http_params()`.
Note that using the new `request.future_response`, it is possible to
add headers and cookies on the response object before the response
object is initialized. Please note that headers/cookies saved on
the future response will NOT be injected in case of error.
PR: odoo#78857
Task: 2571224
* = auth_signup, calendar, im_livechat, snailmail_account, survey, test_mail,
web_editor, website_crm_iap_reveal, website_livechat
The aim of this PR is to improve/fix various flaws and limitation of the current
API, to make it easier to use and more efficient.
Notification are now defined with 3 distinct parts:
- the channel determines which client(s) should receive it
- the type determines how it should be handled
- the payload determines any extra information helpful for handling it
Channel
=======
Business code
-------------
- Record channel is introduced for ease of subscribing to and sending
notifications to specific partners, channels, documents, ...
- String channel is still supported (but it is converted internally to the tuple
channel).
- Tuple channel is still supported without any change (but should be avoided
whenever possible due to its complex syntax).
The channel is no longer sent to the client. When the channel was used for
business purpose, the information it contained has been moved into either the
new type, or the payload itself.
Technical note
--------------
All channels are now internally converted to the tuple (db, ...) channel, which
is necessary for the platform code (saas/sh).
Internally, the bus.bus table is not changed, type and payload are grouped
together into what was (and still is) called message.
Type
====
Type is introduced to uniformize the way notifications are sent and handled.
All existing notifications already had some kind of manually-built type in them.
This is now officially supported at the bus API.
In client code this will allow (to be done in future commits) to register one
handler per specific type, instead of having to iterate and to filter all
received notifications on every handler.
Payload
=======
Payload (ex message) did not change, it can still be anything depending on
business needs.
Few adaptations:
- When the type was included on the payload, the type has been moved to the new
type parameter.
- When the channel was used in business code, its data has been copied into the
payload.
task-1891151
closesodoo/odoo#79201
X-original-commit: 543af27c7d6836ffac9e80ff8490b6ddbd849221
Related: odoo/enterprise#21998
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
When trying to get the im_status of several partners, if one of them
does not exist, it will return an error.
To reproduce the error:
1. Go to Contacts
2. Open a contact
3. Action > Delete
4. Wait max 50 seconds
=> A Missing Record message is displayed. The im_status RPC includes the
deleted partner's identifier and the server looks for it.
Now, the server skips the missing identifiers.
OPW-2390719
closesodoo/odoo#63198
X-original-commit: a35c65032fc5ed1371957c362921bf5fa30062ff
Signed-off-by: adwid <adwid@users.noreply.github.com>
Before this commit, archived and deleted partners had not their
im_status being updated automatically. There were some issues
with this, notably:
- odoobot 'bot' status was not displayed in chatter
- im_status of partners becoming archived were not updated.
This commit fixes both issues.
task-2329591
closesodoo/odoo#60567
X-original-commit: 63b081c49da089c839479b807de8c0cea45221da
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>
Before this commit, mail status manager was regularly fetching
`im_status` of partners with `this._rpc()`. This used the main
thread worker, so it may be overloaded because of that.
To mitigate this issue, mail status manager now uses a new route
`/longpolling/im_status` to fetch `im_status` of partners using
the longpolling thread worker.
closesodoo/odoo#33238
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
When embedding the livechat on an external website, we used to make JSONP calls.
As the support of JSONP calls has been dropped, we now use the CORS mechanism
instead.
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
* remove references to basestring & unicode (use relevant pycompat
helpers)
* remove some str calls (either entirely or replaced by relevant
helper, either text or native)
* use better API to avoid unnecessary conversions
* remove some XML declarations in views
Commit 5d1b323. re-enabled the user presence updates and notifications.
Unfortunately, it showed poor performance due to the high frequency of bus
notifications triggered on our instance (~600k users).
In this rev., we don't trigger notifications on presence changes anymore, but
we rather send the presence of users we have a DM open with, at the end of each
poll period. For performance reasons, we ensure not to do that more than once
every 30 seconds.
We also refined the detection of the 'away' status client side. 'Last presence'
timestamps are stored in the local storage, and the current inactivity period
is sent at each poll. Those presence timestamps now rely on browser activity
detection (click, keypress... events) rather than on the focus on Odoo tabs.
Finally, we removed the no more necessary cron introduced in 5d1b323.
Before this rev., no notification was sent on the bus when the user presence
changed. Thus, the bullets displayed in Discuss were never updated and stayed
as they were on the initialilization of the chat_manager (on webclient launch).
This rev. makes the bus.presence notifications work, and handles them client
side.
Moreover, the disconnections detection is now performed at each poll (with a
maximum of 1 per minute), instead of randomly (1/100 chance) at each poll, as
it scales better than the former solution. We also added a cron that performs
this check every 5 minutes. It is needed to detect that the last connected user
just disconnected (useful for visitors in the website, trying to talk with a
livechat operator).
Also, the 'away' status is now handled client side, as it makes more sense
that way (being away at each poll, e.g. every 50seconds, during 10 minutes
doesn't mean that we didn't come back between two polls).
Finaly, in bus.js, CrossTabBus, we moved the code writing in/reading the local
storage after the tab registration as this code depends on the fact that the
tab is the master tab or not (and this is known only once the tab is
registered).
This rev. was necessary in stable because the livechat uses the user status to
detect if there is an operator available, and this was often inaccurate.
Moreover, it improves the user experience of the chat in the backend.
In test mode, only one cursor is available, and longpolling will take
it, and not give it back before 60s, causing a phantomjs timeout. This
commit simply return an error when the server is in test mode.
Sadly, I had to patch the web client to prevent logging the error in
this case, because that's the way phantomjs detect if there is a
problem.
- im_chat.session will be replaced by mail.channel
- im_chat.shortode is renamed into mail.shortcode
- im_chat.presence is moved to bus module
- js and controller code is moved from im_chat to mail module
This commit only move files, and modify manifests, bundles, ... The code will be adapt in the next commits.
Migrate the models and controller to new API. Renaming xml id according to convention, renaming openerp tag into odoo tag. Add comment strings and documentations.
Xavier-Do