The C implementation of the JS minification gives speedups between 6
and 55 times faster than the regex-based Python port, depending on
how compressed the input it (which is what our default implementation
does).
This is measurable when generating compiled assets bundle from scratch,
e.g. after installing/updating modules or source code.
As an illustration, the minification of a 2MB JS bundle can be 50x
faster:
```py
import rjsmin
from odoo.addons.base.models.assetsbundle import rjsmin as rjsm
js_source = open("web.assets_common_lazy.js").read() # 2MB JS
%timeit rjsm(js_source)
# -> 339 ms ± 495 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit rjsmin.jsmin(js_source)
# -> 6.88 ms ± 213 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
It's also a drop-in replacement, as long as you rjsmin 1.1.0 or better
is available (to support format strings properly, a.o.).
See also the documentation of rjsmin: http://opensource.perlig.de/rjsmin/closesodoo/odoo#104283
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The ofxparse module explicitly uses an html parser instead of xml but
since bs4 4.11.0, this triggers a warning.
closesodoo/odoo#113354
X-original-commit: e81ed97c9c54c69c68c54c69b4d3d80ed0515ae8
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Werkzeug 1.0 colorised some outputs on POSIX IFF `click` was
installed.
Since 2.0 (pallets/werkzeug#2012) werkzeug unconditionally colorises
the log on POSIX. This is annoying when using output redirection (let
alone logging to a non-stream), as werkzeug will dump ANSI color codes
to the non-term stdout and thus the logfile.
Werkzeug provides no official knob to control this behaviour, but it
does have a secret flag which is normally used to check if colorama is
available on windows (so the ANSI codes are not output if colorama
won't be interpreting and stripping them on the way out). Since
`werkzeug.serving` is available in pretty much all versions, we can
just (un)set this flag if not logging to a tty, and versions 2+ should
pick it up and disable colorisation.
closesodoo/odoo#112829
X-original-commit: 7c9f883dc508a7a8a45bf7bf7e900da0be9b34be
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Since Python 3.11, sampling from a set deprecated, the population must
be a sequence.
This commit applies the suggested fix.
Also, the filtering of the deprecation warning about sampling from set
can be disabled when the python version is not 3.9. This warning was
wrongly triggered since 3.9 because recordsets are bot a sequence and a
set.
This was fixed in python 3.10, see https://bugs.python.org/issue42470closesodoo/odoo#112450
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Fine tuning of 8eaac97, the `errors` attribute was added in py38[^1]
[^1]: python/cpython@ca7b504a4dclosesodoo/odoo#112588
X-original-commit: c6c19ff6a3093fe64035d9f970ab35522f8080f5
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
Remove custom open introduced by bpo-26789 as we do not need it.
closesodoo/odoo#112453
X-original-commit: 8eaac9744b93e7132827edb2a59c28bb43732ec1
Related: odoo/enterprise#36978
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
Co-authored-by: Martin Trigaux <mat@odoo.com>
Followup of: odoo/enterprise@b58637f1ab
That completely removed the firebase_admin library support, making this warning
ignore line not necessary.
Task-3000338
closesodoo/odoo#103474
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
* clean and improve docstrings in orm
* fix typos found with codespell
* rely on the Environment class docstring instead of doc content (and
therefore move part of the doc inside the class docstring)
closesodoo/odoo#102969
X-original-commit: 8250cd4b210005d223a4cdb8afa4014425ca6fa3
Related: odoo/documentation#2803
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
With version of urllib3 in ubuntu 22.04 (1.26.5) and with version of firebase_admin between 2.17.0 and 4.5.2 a gives the following
warning:
In firebase_admin/_http_client.py:30: DeprecationWarning: Using 'method_whitelist' with Retry is deprecated and will be removed in v2.0. Use 'allowed_methods' instead
It has been fixed in later version
https://github.com/firebase/firebase-admin-python/pull/532 since v4.5.2
closesodoo/odoo#101731
X-original-commit: 66934cde91843a90b99ec02557b41cb68c281119
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Check the PY_COLORS env variable and force colored output even if the
is_a_tty is no true
TT24972
closesodoo/odoo#71685
Signed-off-by: Olivier Dony <odo@odoo.com>
Since Python 3.10, the invalid escape sequence warning use quotes to
display the invalid sequence. Because of that, the filter does not catch
it anymore.
With this commit, they are caught in all supported versions.
Part-of: odoo/odoo#91927
Add a big fat warning when the qweb compiler finds a `t-raw`.
`t-esc` should now be used everywhere, the use-case for `t-raw` should
be handled by converting the corresponding values to `Markup`
objects. Even though it's convenient, this constructor *should never
be made available in the qweb rendering context* (maybe that should be
checked for explicitely?).
Replace `werkzeug.escape` by `markupsafe.escape` in
`odoo.tools.html_escape`, this means the output of `html_escape` is
markup-safe.
Updated qweb to work correctly with escaping and `Markup`, amongst
other things QWeb bodies should be markup-safe internally (so that a
`t-set` value can be fed into a `t-esc`). See at the bottom for the
attributes handling as it's a bit complicated.
`to_text` needed updating: `markupsafe.Markup` is a subclass of `str`,
but `str` is not a passthrough for strings. So `Markup` instances
going through would be converted to normal `str`, losing their safety
flag. Since qweb internally uses `to_text` on pretty much
everything (in order to handle None / False), this would then cause
almost every `Markup` to get mistakenly double-escaped.
Also mark a bunch of APIs as markup-safe by default
* html_sanitize output.
* HTML fields content, sanitization is applied on intake (so stripped
by the trip through the database) and if the field is unsanitised
the injection is very much intentional, probably. Note: this
includes automatically decoding bytes as a number of default values
& computes yield bytes, which Markup will happily accept... by
repr-ing them which is useless. This is hard to notice without `-b`.
* Script-safe json, it's rather the point (though it uses a
non-standard escaping scheme).
* Note that `nl2br`, kinda: it should work correctly whether or not
the input is markup-safe, this means we should not need to escape
values fed to `nl2br`, but it doesn't hurt either.
Update some qweb field serialisations to mark their output as
markup-safe when necessary (e.g. monetary, barcode,
contact). Otherwise either using proper escaping internally or doing
nothing should do the trick.
Also update qweb to return markup-safe bytes: we want qweb to return
markup-safe contents as a common use-case is to render something with
one template, and inject its content in an other one (with Python code
inbetween, as `t-call` works a bit differently and does not go through
the external rendering interface).
However qweb returns `bytes` while `Markup` extends `str`. After a
quick experiment with changing qweb rendering to return `str` (rather
unmitigated failure I fear), it looks like the safest tack is to add a
somewhat similar bytes-based type, which decodes to a `Markup` but
keeps to bytes semantics.
For debugging and convenience reasons, MarkupSafeBytes does *not*
stringify and raises an error instead (`__repr__` works fine). This is
to avoid implicit stringifications which do the wrong thing (namely
create a string `"b'foo'"`).
Also add some configuration around BytesWarning (which still has to be
enabled at the interpreter level via `-b`, there's no way to enable it
programmatically smh), and monkeypatch `showwarning` to show warning
tracebacks, as it's common for warnings to be triggered in the bowels
of the application, and hard to relate to business logic without the
complete traceback.
`t-out`
=======
`t-esc` is a bit confusing for the new behaviour of "maybe escape
maybe not", so add a `t-out` alias with the same behaviour.
Unlike `t-raw`, `t-esc` is only soft-deprecated for now: there are
thousands of instances, so editing all the templates is not
great. Eventually we'll add a `ci/style` to prevent addition of new
ones, and eventually we might do a bulk-replace and hard-deprecate.
Attributes handling
===================
There are a few issues with respect to attributes. The first issue is
that markup-safe content is not necessarily attributes-safe
e.g. markup-safe content can contain unescaped `<` or double-quotes
while attributes can not. So we must forcefully escape the input, even
if it's supposedly markup-safe already.
This causes a problem for script-safe JSON: it's markup-safe but
really does its own thing. So instead of escaping it up-front and
wrapping it in Markup, make script-safe JSON its own type which
applies JSON-escaping *during the `__html__` call.
This way if a script-safe JSON object goes through `markupsafe.escape`
we'll apply script-safe escaping, otherwise it'll be treated as a
regular strings and eventually escaped the normal way.
A second issue was the processing of format-valued
attributes (`t-attf`): literal segments should always be markup-safe,
while non-literal may or may not be. This turns out to be an issue if
the non-literal segment *is* markup-safe: in that case when the
literal and non-literal segments get concatenated the literal segments
will get escaped, then attributes serialization will escape
them *again* leading to doubly-escaped content in attributes.
The most visible instance of this was the `snippet_options` template,
specifically:
<t t-set="so_content_addition_selector" t-translation="off">blockquote, ...</t>
<div id="so_content_addition"
t-att-data-selector="so_content_addition_selector"
t-attf-data-drop-near="p, h1, h2, h3, .row > div > img, #{so_content_addition_selector}"
data-drop-in=".content, nav"/>
Here `so_content_addition_selector` is a qweb body therefore
markup-safe, When concatenated with the literal part of
`t-atff-data-drop-near` it would cause the HTML-escaping of that
yielding a new Markup object. Normal attributes processing would then
strip the markup flag (using `str()`) and escape it again, leading to
doubly-escaped literals.
The original hack around was to unescape() `Markup` content before
stringifying it and escaping it again, in the attribute serialization
method (`_append_attributes`).
That's pretty disgusting, after some more consideration & testing it
looks like a much better and safer fix is to ensure the
expression (non-literal) segments of format strings always result in
`str`, never `Markup`, which is easy enough: just all `str()` on the
output of strexpr. We could also have concatenated all the bits using
`''.join` instead of repeated concatenation (`+`).
Also add a check on the type of the format string for safety, I think
it should always be a proper str and the bytes thing is only when
running in py2 (where lxml uses bytestrings as a space optimization
for ascii-only values) but it should not hurt too much to perform a
single typecheck assertion on the value... instead of performing one
per literal segment.
Note: we may need to implement unescape anyway, because it's still
possible to get double-escaping with the current scheme: given an
explicitly escape-ed `foo` and `t-att-foo="foo"`, `foo` will be
re-escaped.
fixup! [CHG] core, web: deprecate t-raw
The astroid package where version is lower than 2.5.1 uses imports the `imp`
module that was deprecated in python 3.4 [0].
In Debian buster, astroid version is 2.1.0 and 2.3.3 in Ubuntu Focal.
With this commit, the deprecation warning is filtered out.
Note that pylint 2.5 avoid that deprecated import [1], that's the reason
why the warning was not showing on current runbot Docker image.
[0] https://docs.python.org/3/library/imp.html
[1] https://github.com/PyCQA/pylint/commit/51c646bf70a6e0a86492bfd2ddd1885671d64d67closesodoo/odoo#67469
X-original-commit: 8127bea147c093aa64ac7d31ebc69e4dcbef6eef
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Only updates outdated requirements which actively cause issues:
* freezegun broken in 3.8 (removal of time.clock)
* xlrd broken in 3.8 (removal of time.clock)
* also monkeypatches xlrd.xlsx for 3.9 (removal of
Element.getiterator, breaks because of defusedxml)
* jinja triggers DeprecationWarning in 3.8
* pillow triggers warning in 3.9
* lxml, greenlet don't compile in 3.9
* reportlab doesn't work in 3.9
New versions try to match those of Debian Bullseye.
Also adds a script to more easily compare dependency versions between
the requirements files and what's in various distributions (currently
supports checking against debian and ubuntu).
Furthermore updates warnings filtering:
* removes xlrd (mischeck was monkeypatched as noted above)
* removes setuptools (was for older versions, one would hope this
isn't an issue anymore)
* adds babel: python-babel/babel#684 fixes the deprecation warning but
is not part of any release yet
* ignores error related to `random.sample` on a set, this is a
diagnostics bug because recordsets implement both Sequence and Set,
and the stdlib checks for Set first (bpo-42470)
See #59980Closes#61103closesodoo/odoo#62510
X-original-commit: 648635deca67df09417ae55c6eb181c98524b74d
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
In current Debian version (0.19) ofxparse gives the following warning:
ofxparse/ofxparse.py:40: DeprecationWarning: Using or importing the ABCs from 'collections'
instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
return isinstance(candidate, collections.Iterable)
Since we can't do anything about it, we filter it for now.
ofxparse already fixed it https://github.com/jseutter/ofxparse/issues/155
A Debian bug report has been opened https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=973055closesodoo/odoo#60880
X-original-commit: 00497c03db080914fed81dc3fc66fa5d45f65a4e
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Signed-off-by: Thomas Dieuzeide <tdi-odoo@users.noreply.github.com>
When the PostgreSQLHandler is used to store some logs in the ir_logging
table of a database, the log record pathname is truncated to keep only
the relevant part of the path. In some circumstances, the path is
wrongly truncated, leading to totally invalid paths.
e.g.: When using an addon-path like `/data/build/enteprise` the removed
part correspond to the length of `/data/build/odoo/`. The resulting path
is `prise/....`.
With this commit, the full path is kept.
This commit is mainly a fix for the runbot logs and should not have an
impact on existing databases.
closesodoo/odoo#50246
X-original-commit: f3c96aa21ad0d8756fadc1608107438ba8cd6c05
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
DeprecationWarning was configured as `once` which might be why I
missed some previously: `once` means the *filter* will only signal
once, meaning only one DeprecationWarning gets shown per run. Even if
there are 5 *completely different* warnings getting generated by the
codebase.
`default` should be more suitable, I think I'd originally
misunderstood it as "whatever the default is for this warning" but
what it *actually* means is "print once for each (module, lineno)".
An alternative could be `module` which would show the warning once per
module but wouldn't trigger from different warnings in the same module.
Before this commit, trying to use Odoo as a library without initializing
the logging features would result in a crash because `Logger.runbot()`
is monkey-patched inside the function `init_logger()`, which is de facto
necessary for using Odoo as a lib.
With this commit, we do the monkey-patch at the module-level so whenever
the `netsvc` package is imported the patch is applied, which should be
well before the loading of the registry starts.
closesodoo/odoo#48396
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
* from collections import <ABC> is deprecated, unclear why the
deprecation warning didn't appear before (possibly only appears in
3.7/3.8?) either way `collections.abc` should be 3.3+ so switch
everything to it.
* add some more ignores on third-party packages deprecation
warnings (meh)
* while at it, mitigate generation of non-breaking space on some
versions of Babel (in the french locale used by our tests anyway)
closesodoo/odoo#47581
Related: odoo/enterprise#9214
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Performances from a general point of view can be difficult to track.
This commit proposes to improve logs in two ways:
The current logs only use the sql_counter, wich will only be updated
when a cursor is closed. In a test-enable install, this counter
is actually the queries of the tests wince the install cursor is
open untill the end. The first fix is to use bot sql_counter and
sql_log_count to have total queries untill now on closed cursor,
but also the current number of queries of the current cursor.
This means that the new log format will be
{nb} modules loaded in {time}, {loading_querie} (+{test_cr_queries}) queries
instead of
{nb} modules loaded in {time}, {tests_cr__queries}queries
Nothe that in the current version, {nb} is actually the total number of
loaded modules until now.
This commit also add an equivalent end log by module and change the
loglevel of module start on install (mainly usefull if an error occurs
before anything else is logged hidding the module causing this error.)
A cleaner runbot logger is also added, in order to be abble to call
_logger.runbot( instead of _logger.log(25. This will clarify the purpose
of such a log level.
closesodoo/odoo#47283
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Allows better formatted and commented log_handler items in
configuration files: configparser allows multiline values and
interspersed comments (stripping out comment lines), but while it
removes indent it leaves the linebreaks, so e.g.
foo =
bar,
# qux
quux
when parsed and split on "," results in `['\nbar', '\nquux']` which is
obviously an issue for logging (as it assumes the leading `\n` is part
of the logger name proper).
Since log_handler items can be fairly long and are not necessarily
self-descriptive as to *why* they were selected for re-configuration,
allowing one per line & comments is helpful.
Configuring the non-root logger based on --log-level had the
side-effect (unclear whether it was intended or not as none of the
relevant commits really documented the idea) that log-level could
override log handlers being set on the root logger e.g. if
`log_handler = :INFO` is set in the config file, `--log-level=warn` on
the CLI will override it.
This could be replicated by swapping `pseudo_config` and `logconfig`,
*however* it would also make the sub-loggers override
differently (currently on an exact logger match log-handler overrides
log-level).
The ideal fix would likely be to sequence log-level from the
configuration file, log-handler from the configuration file, log-level
from the CLI and log-handler from the CLI. However that doesn't really
work with the current structure.
closesodoo/odoo#41180
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
* root the base.partner.merge wizard's logger in odoo.addons so the
"odoo" logger configuration properly applies to it
* update the default logging configuration to set the root logger
rather than independently set the odoo and werkzeug loggers
closesodoo/odoo#40872
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
DROP CONSTRAINT (even with IF EXISTS is specified) acquires an ACCESS
EXCLUSIVE lock on the table, preventing e.g. inserts in an other
transaction, so ir_logging would systematically deadlock if configured
to the same database and a warning would be triggered during install
or update (if that ran ir.logging's init).
1. hand-roll the "IF EXISTS" bit, to avoid taking an ACCESS EXCLUSIVE
lock on the table if the problematic constraint does not exist and
thus doesn't need to be dropped (which by now should be the vast
majority of cases).
Replacing DROP CONSTRAINT with DISABLE TRIGGER does not fix the
issue as *that* acquires SHARE ROW EXCLUSIVE. While that's less
constraitning than ACCESS EXCLUSIVE, it still conflicts with an
insert's ROW_EXCLUSIVE.
2. add a timeout to the logging INSERT anyway, the deadlock is still
an issue if we're updating a database which does have the
problematic constraint, and we want to preclude the possible
eventual introduction of new deadlocks in the future.
closesodoo/odoo#34243
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This reverts commit c42a130a98
and commit fb7ac56d65.
The runbot is using cross db logging and those commit were breaking the
mechanism. As a result, the master builds logs were empty.
If we update the runbot to use logging_type instead, the other versions
builds will be impacted hence this revert.
closesodoo/odoo#28512
When there is a performance issue, it's sometimes difficult to discover
which request increased the query count or its duration.
With this commit, the query count, the query time and "python and io" time are displayed
in the logs at the end of each werkzeug request line.
Co-authored-by: Christophe Monniez <moc@odoo.com>
4e5119c6e5 didn't account for
`tools.config['workers']` being `None` on windows, resulting in the
entire thing crashing when trying to execute `None > 1` in Python 3.
Fixes#26447
Olivier Dony