Performances from a general point of view can be difficult to track.
This commit proposes to improve logs in two ways:
The current logs only use the sql_counter, wich will only be updated
when a cursor is closed. In a test-enable install, this counter
is actually the queries of the tests wince the install cursor is
open untill the end. The first fix is to use bot sql_counter and
sql_log_count to have total queries untill now on closed cursor,
but also the current number of queries of the current cursor.
This means that the new log format will be
{nb} modules loaded in {time}, {loading_querie} (+{test_cr_queries}) queries
instead of
{nb} modules loaded in {time}, {tests_cr__queries}queries
Nothe that in the current version, {nb} is actually the total number of
loaded modules until now.
This commit also add an equivalent end log by module and change the
loglevel of module start on install (mainly usefull if an error occurs
before anything else is logged hidding the module causing this error.)
A cleaner runbot logger is also added, in order to be abble to call
_logger.runbot( instead of _logger.log(25. This will clarify the purpose
of such a log level.
closesodoo/odoo#47283
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Before this commit:
- Install a module
- Uninstall the previously installed module
- The registry will complain that some dependencies may be missing
for the module being uninstalled
This happens because we check after the installation / upgrade of
modules that none have been left in a transient state to verify that new
dependencies have been properly installed and loaded, this applies to
'to install' and 'to upgrade' states however it's not the same for 'to
remove' states, as the process of uninstall happens much later in the
code.
After this commit, simply uninstalling modules will not trigger this
error log.
Do note that in case of a problem with an uninstall, the function
"reset_module_states" will tackle the case of leftover transient states.
closesodoo/odoo#47495
X-original-commit: c095a28314f78d1d9854e5e9bcb26014922a4f12
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Sometimes, constraints are not delayed during a module upgrade: the
module is 'to upgrade' but in 'init' mode :-/
Fix this by relying on the module's state only.
closesodoo/odoo#45179
X-original-commit: 878b6538961a9b6a5fe1c6ac68d4ff39391abcdf
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
In some case, a pre-migration script calls `xml_import.parse` to force
reload a no-update data in pre-script. It is possible that fields are
marked to recompute in the process. The problem is that the reference to
the field added in `env.all.tocompute` won't be the same as the
reference of the same field after calling `registry.setup_models()`,
leading to an infinite loop while recomputing this field, since
`fields.__get__` wont find `self` in `env.all.tocompute`.
This problem was discovered when trying to migrate a database with all
modules installed (no demo data) from 12.0 to 13.0 (for commit
references: http://runbot.odoo.com/runbot/build/1176576 with database
comming from http://runbot.odoo.com/runbot/build/1171714).
This commit adds a check before executing `registry.setup_models()` in
order to log when some fields to compute remain before breaking fields
references, and adds a `flush()` after pre-scripts to fix the current
issue.
X-original-commit: 41b9d810066774736c4077bba1dc9c9fba004f48
Before this commit models with the `_transient` flag were ignored in
ir.model.access verifications. Only an implicit ir.rule with the
domain (create_uid=user.id) was applied to avoid most side effects.
The problem is that, often, the security does not lie in side-effects
of abusing of somebody else's wizard record but in the fact that the
wizard methods blindly trust only the right users are creating these
records. Too often, too many sudo were used and creating wizard with
chosen values could lead to an abuse scenario.
Instead, explicitly require the developer to declare security rules
the same way as on any other model.
Studio customization is an exception, a data module added to ir_module
but that is never added to graph since there is no manifest.
OPW #2180885closes#43880closesodoo/odoo#43940
X-original-commit: 4051ac83b5e7cc1cb91a264b51401ad5601ad4e2
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
When migrating a database, load_marked_modules will be called multiple times, alternating
to upgrade and to install modules. The main reason for this is still a litle confusing
but it as the side effect to log "Unmet dependencies" error multiple time in add_modules,
even if the dependency will be resolved later.
This commit removes the error level for this log, and replace it by another check,
performed at the end, logging any module in "to install"/"to upgrade" state.
Also log removed module as info (25), not warning. This may be changed latter
closesodoo/odoo#43797
X-original-commit: c5d6a3977de85fb974a4940fb11d54ef847e08e4
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This branch is the combination of several optimizations in the ORM:
* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;
* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;
* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);
* make method `modified` take advantage of inverse fields to inverse
dependencies;
* filter records by evaluating a domain on records in Python;
* a computed field with `readonly=False` behaves like a normal field
with an onchange method;
* computed fields are computed in superuser mode by default.
Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.
closesodoo/odoo#35659
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
An unintended side-effect of #29431 is apparently that auto_install
applications automatically get reinstalled when uninstalled, which was
not the goal.
Move the auto_install selection back into db.py, this version seems to
work even though the previous attempt was apparently unsuccessful.
closesodoo/odoo#35180
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
1. Make test logs clearer & remove redundancies
Instead of having an ERROR log right when the test fails then print
the useful / relevant information at the end of the test suite,
immediately print the traceback. Keep the final summary. Also avoids
having to wait for the entire test suite to end before a dev' can know
the failure details of a specific test.
Done by working at a lower level and replacing the custom
test stream mess by a custom Result class which prints and formats the
information we want. Replace TextTestRunner by a bare-bones custom
Runner object to tie it in.
2. Provide useful location information on test failure
Leverage the work above to log the test function's failure location:
previously logging would point to within TestStream which is not
useful.
Here, on failure the traceback is used to discover the caller info and
point to the test line which fails instead. similar to unittest's
_exc_info_to_string (https://github.com/python/cpython/blob/93e8aa62cfd0a61efed4a61a2ffc2283ae986ef2/Lib/unittest/result.py#L173).
3. Replace direct logging in browser_js by raising errors
Properly marks the test as in error, and the error traceback points to
the tour definition / launcher (python side) rather than common.py
and/or module.py.
Also removes unused dbname parameter that was added in
/278ed718e9805edf088642ba10d3b7c4e5716c31/openerp/modules/module.py#L361
for nor visible reason
closesodoo/odoo#34996
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
As most of the tests are using demo data, a check is performed to verify
that they are loaded.
A best practice is to write tests that does not depend on demo data but
with this check, it's not possible to launch them without demo data.
With this commit, the check is removed, allowing to launch test even
without demo data loaded.
closesodoo/odoo#33531
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Currently, auto_install is triggered when all dependencies get
installed, but there are cases where one would want such trigger on
only a subset thereof.
e.g. we want `website_sale_dashboard` to auto-install when
`website_sale` is installed. Currently, it requires `web_dashboard` to
also be auto-installed otherwise `website_sale_dashboard` would "wait"
for both dependencies to be explicitly installed before the
auto-install triggers. That's despite `web_dashboard` not being very
useful on its own. More generally this is an issue with technical
modules which need to be marked as auto_install so as not to block
e.g. bridge modules from automatically installing.
This change allows setting `auto_install` to a subset of `depends`:
* if auto_install is set to `False`, the module does not get
automatically installed (no change in semantics)
* if auto_install is set to `True`, the module gets automatically
installed if and only if all its dependencies are installed (also no
change in semantics)
* if auto_install is set to a list of dependencies, the module will be
installed when all *these* dependencies are installed, other
dependencies (excluded from auto_install) will be installed
alongside as a consequence
* auto_install can be set to an empty list, in this case the module
will always be automatically installed regardless of its
dependencies (and will force their installation).
So after this change, `web_dashboard`'s auto_install can be set to
`False` (such that it's not installed if no module defining dashboards
is installed) and `website_sale_dashboard`'s manifest can be edited
to:
'auto_install': ['website_sale']
possibilities:
# no automatic installation
'depends': ['a', 'b'],
'auto_install': False
# automatic installation if both a and b are installed
'depends': ['a', 'b'],
'auto_install': True
# automatic installation if both a and b are installed (explicit)
'depends': ['a', 'b'],
'auto_install': ['a', 'b']
# automatic installation if b is installed, a will get forcefully
# installed if it isn't yet
'depends': ['a', 'b'],
'auto_install': ['b']
# always automatically installed, will cause the installation of
# its dependencies even if they're not marked explicitly
'depends': ['a', 'b'],
'auto_install': []
Task 1851328
closesodoo/odoo#29431
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
After a module upgrade, validate the architecture of the module's views that
are impacted by updates, but have not been checked yet. Before this patch,
views were checked twice on average.
This patch speeds up the installation of modules by about 15% without demo
data. With demo data, the speedup is around 10%.
The registry loading system should not alter databases unless it is
asked to do so, by a module installation or update instruction.
This property should hold true as well for database bootstrap, and this
is what this commit changes..
In order to avoid any behavior change for command-line users, an
implicit `-i base` is assumed when starting the server from the
command-line with `-d <db>`, causing the db boostrap to happen if the
database did not exist yet.
Task 1856935
Currently, if installing a module's demo data fails (because the
module was uninstalled and some were left over and can't be
reinstalled, or because they're broken, or…) the installation of the
module fails and further installations are skipped (?).
Since demo data are non-essential (though required to run tests),
rather than fail everything if they fail just roll them back and
notify the user.
This change generates a warning log *and* shows a notification popup
to the end user.
Try to remove cr.commit (and rollback) from module and db install:
* put a savepoint around test data loading
* remove a bunch of commits sprinkled throughout
* remove rollback on data loading failure (assuming it bubbles up, the
entire module's installation should be rolled back)
* add commit right before the tests are run, so they can run isolated
and still see whatever was done when installing their module
* convert a few explicit closing to context managers
Before this commit:
* Module A defines a field X of model M
* Module B inherits from model M without touching field X
* Module C inherits from model M and extends field X by giving an INDEX
/ NOT NULL constraint.
* Module B and C depend from Module A, but not each other
If all three modules are installed and Module B is updated, the INDEX /
NOT NULL constraint could be dropped.
This happens because Module B can be loaded before Module C is loaded,
if that's the case, then after the upgrade of Module B, during the
schema checking, we verify that the field object we have and the field
on the DB are the same, since Module B doesn't introduce the index then
this check is false and we drop the index. When we get to loading Module
C, we do not do any schema checking because the module is not marked as
`to upgrade`, therefore the index is lost forever.
To solve this, we re-init the models that belong to the set of the intersection
between upgraded and modified models and loaded and modified models.
Fixes#24958
Before this commit:
* Module A defines a field X of model M
* Module B inherits from model M without touching field X
* Module C inherits from model M and extends field X by giving an INDEX
/ NOT NULL constraint.
* Module B and C depend from Module A, but not each other
If all three modules are installed and Module B is updated, the INDEX /
NOT NULL constraint could be dropped.
This happens because Module B can be loaded before Module C is loaded,
if that's the case, then after the upgrade of Module B, during the
schema checking, we verify that the field object we have and the field
on the DB are the same, since Module B doesn't introduce the index then
this check is false and we drop the index. When we get to loading Module
C, we do not do any schema checking because the module is not marked as
`to upgrade`, therefore the index is lost forever.
To solve this, we re-init the models that belong to the set of the intersection
between upgraded and modified models and loaded and modified models.
Fixes#24958
Before this commit:
* Install any module that creates a SQL view and another that extends
this view.
e.g.: sale and pos_sale for `report.all.channels.sales`.
* Uninstall the module that extended the SQL view.
e.g.: uninstall pos_sale.
* Try to access the view from the web client -> Traceback, table not
found.
This happens because when reloading the registry, postgres drops the sql
view and it must be re-initialized.
After this commit:
We solve this issue by calculating all missing tables/views during the
uninstallation process, and re-initializing all of them right before
reloading the registry.
Also remove sql-view hacks in the modules `sale` and `sale_margin`.
Fixes#23528, #23529, #23530
Commit 763d714 introduced cron job locking for databases which had
modules with states set to 'to x', however if an
installation/uninstallation/upgrade fails, the state will stay at 'to
x', and it may stay in that state for an indefinite amount of time,
meaning that cron jobs could stay locked forever.
This commit fixes this in part by adding a cleanup function to loading.py that
will be executed whenever load_modules fails, the function will change
every 'to x' module to their original state, effectively unlocking the
execution of cron jobs.
This however only works to prevent "zombie" transient states for
brand new databases, however for existing databases which already
contain some modules in a zombie state it won't do anything unless
a module is installed/uninstalled/upgraded, which may never happen.
This is where the second part comes in (ir_cron.py), when failing to
execute crons, we check if the failure was due to bad module state
and if an arbitrary amount of time (5 hours as of this commit) has passed
since the last time it was supposed to be executed, if it is the case, it means
that the cron execution failed around 5 * 60 times (1 failure per minute for 5h)
in which case we assume that the crons are stuck because the db
has zombie states and we force a call to reset_module_states.
Previous to this rev., sometimes None could be added
to the list of _init_modules in the registry, this would
then be problematic in ir_http since a sorted would be
performed on this list, which works in py2 but in py3
None and string can't be compared implicitly.
Was only done when updating base.
The cost of scanning the addons path should small in comparison to the update
time itself.
Followup of #9133Closes#9140
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
Xavier-Do