This provides a new API for those operations, in order to make the
distinction between the use cases more explicit. The former API was
using obscure parameter combinations to correspond to various cases.
In the summary below, `fnames` is an iterable of field names. If the
parameter is not given, it means "all fields" in the given context.
Note that method recompute() is now mostly private, as it should not be
used in business code.
# process pending computations and updates
records.env.flush_all() # all fields of all models
records.flush_model(fnames) # the fields of all records of the model
records.flush_recordset(fnames) # the fields of the given records
# process pending computations, became non-public methods
records.env._recompute_all() # all fields of all models
records._recompute_model(fnames) # the fields of all records of the model
records._recompute_recordset(fnames) # the fields of the given records
# invalidate the cache of fields
records.env.invalidate_all() # all fields of all models
records.invalidate_model(fnames) # the fields of all records of the model
records.invalidate_recordset(fnames) # the fields of the given records
Part-of: odoo/odoo#87527
* add configuration for `flake8[flake8-rst-docstring]`
* enable docstring-related checks
* fix invalid docstrings in odoo's core & `base`
* fix a few more bits (mostly missing or incorrect `:param:` info
fields) are out of scope for the lint but my editor catches
closesodoo/odoo#74604
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This reverts commit 9cc4d6956b71291958114196107b1889109e92a1.
How to reproduce the issue:
- Starts from a database with some modules installed (account).
- Install a module with data only (l10n_generic_coa)
The models are not setup and the data fails to install.
A proper fix would be to mark the registry as dirty when new models are
added and only setup models when needed.
Since the faulty commit was part of a bunch of optimization and the
impact of this particular one is quite small, reverting it is a quick
and easy fix waiting for a better one (maybe, one day).
closesodoo/odoo#79552
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Some modules are only adding data/tests/routes/static content.
In this case, it is useless to check the database schema or setup models
in registry.
closesodoo/odoo#78898
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
When installing a database with all modules in enterprise,
install take around 20 minutes and almost half of that is spent in the
`setup_model` method.
There is actually two calls to `setup_models` for each module.
One of them was introduced in b5c50fa824
and only looks useful when upgrading a module with migration scripts.
This first fix proposes to skip `setup_models` if the module state is
`to install`.
closesodoo/odoo#78808
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Since
https://github.com/odoo/odoo/commit/8a3e1a0ccdad25ba5b4b99639bdaeb9073a27f1f,
`_update_user_groups_view` is called only once per module installation. However,
module installation is not the only scenario when data files are loaded and
hence we need to add the method call.
---
opw-2602541
closesodoo/odoo#76869
X-original-commit: a46d20c89c24a4e60faef09cf8ad7721bd29d6cc
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Refactor the Environments object into a Transaction object, which is
bound to one cursor, and is no longer shared among several cursors.
The following methods/properties have been changed:
- Environment.envs no longer works (because of the design change);
- Environment.manage() is deprecated (no longer useful);
- Environment.reset() is now an instance method;
- env.clear_upon_failure() is deprecated in favor of cr.savepoint().
closesodoo/odoo#75598
Related: odoo/enterprise#20451
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
When installing some modules, if an other action is undertaken around
the same time it is possible for the two to deadlock, leading to the
two workers being killed by the wallclock limit watcher. This should
only be an issue on the *threaded* server, workers should not be
affected, meaning the issue is not reproducible on runbot.
A relatively reliable way to trigger this issue manually is to create
an empty database, on the "apps" kanban view install the "sales"
application, and as soon as the UI gets unblocked install the "CRM"
application. An other method (also reliable but not really doable by
hand) is to simultanously log in and install the sales
application (`sale_management` module).
The core of the issue seems to be in `_button_immediate_function`:
1. `Other` has an env ready for use and has accessed various
models (so has pretty shallow locks on the tables e.g.
ACCESS SHARE).
2. `Install` takes the registry lock to create the new registry.
3. `Install` needs to update one of the tables `other` has touched,
starts waiting on the `ACCESS EXCLUSIVE` lock (the only one
`ACCESS SHARE` conflicts with) in order to execute DDL (most `ALTER
TABLE` forms require exclusive access to the table).
4. `Other` needs a new environment (e.g. `sudo()`, `with_user`,
`with_context`, ...), starts waiting on the registry lock.
At this point the two threads are deadlocked, `other` waits on the
registry lock which `install` holds, while `install` waits on a table
lock which `other` holds. Since one of the waits is on the application
side, Postgres' deadlock detector can not notice the issue. That one
of the locks is on the Python side is why only the threaded
server *should* be affected.
An initial seemingly promising mitigation attempt was to
LOCK res_partner IN ACCESS EXCLUSIVE MODE
in the prelude of `_button_immediate_function` as `res.partner` is one
of the most commonly modified models, this would force
`_button_immediate_function` to wait until all existing requests have
completed and prevent later requests from progressing.
This turns out to be unreliable, as later requests could already have
acquired an environment and would race ahead as soon as the
transaction is committed if the scheduler lets them. Trying to lock
the registries earlier doesn't work as the locking is interleaved in
normal operation and we'd just deadlock there. The commit is because
`load_modules` does not take an externally provided cursor and instead
creates its own (thus its own connection and transaction). And because
of its lack of atomicity the issue might occur regardless.
An alternate mitigation is instead to set (or drastically reduce) the
lock wait delay during module installation, installation should
normally be entirely uncontended (or infeasible in production with
large traffic) so there is limited reason it'd be waiting several
seconds on a lock. Conveniently, this means instead of the thread
being killed entirely, the install request gets aborted *and retried*,
so it can succeed a little more slowly if that allows the other
request to complete and no other concurrent request causes the same
issue.
Other alternate mitigation which got discarded: reusing registries
when creating new environments (from existing ones) if the database is
the same, that works for some case of switching environments, it
doesn't work for other where we actually fetch a registry e.g. assets
generation calls `get_modules_order` which calls `module_boot` which
calls `module_installed_bypass_session` which gets a
registry. `get_modules_order` is the last place we know we have an
existing registry. Though maybe we could strip out the entire thing
and call `module_installed(self.env)` directly?
Issue 2581648
closesodoo/odoo#73906
X-original-commit: ab84d970dcf1a9dbd5697b6600930cd1bcba3634
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When a custom module is in 'to upgrade' state,
and the code has a dependency that is not yet
installed, Odoo refuses to upgrade it, and
says the new dependency is unmet.
This commit fixes this by also calling button_upgrade() in this situation,
and not only for modules in 'installed' state.
This situation arises in a version migration scenario. Custom modules are
in 'to upgrade' state after migration.
If one of these custom modules has a new dependency
after migration, it refuses to upgrade.
closesodoo/odoo#72942
X-original-commit: d5ffe0159ada953985a23c29e36763599bd10ed9
Signed-off-by: Adrian Torres (adt) <adt@odoo.com>
Translation overwriting was not working when forced by command line
arg --i18n-overwrite
This is due to the change of signature of the method, no longer
relying on the context
Fixesodoo/odoo#67419closesodoo/odoo#67873
X-original-commit: 44624f5d51a266c4fc37644d3fc36b810e722ee4
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
We have some tests in odoo/upgrade that are sensitive to the order on
which they are executed. Specifically: IntegrityCase tests need to be
run after all UpgradeCase tests across all Odoo modules.
To support this we implemented a sorting mechanism for tests based on
the test_sequence class attribute. This is intended to be used by meta
cases, not by individual tests.
closesodoo/odoo#66521
Related: odoo/upgrade#2184
Signed-off-by: Denis Ledoux (dle) <dle@odoo.com>
Before this patch, adding a field on a custom model discards all
automated actions on that model. The explanation is relatively simple.
When models are set up in the registry, the classes of custom models are
dropped then recreated. Given that automated actions are implemented as
monkey-patches on model classes, the setup of models simply loses those
monkey-patches, which explains why they stop working on custom models.
The fix introduces an `_unregister_hook()` method, that is expected to
clean up what has been done in `_register_hook()`. When the registry is
ready (i.e., not being loaded), the setup of models first invokes
`_unregister_hook()` on models, proceeds with the setup, and finally
invokes `_register_hook()` to reinstall the hooks.
OPW 2362308
closesodoo/odoo#60833
X-original-commit: 67152bf82da2674179297d32e4cec9dd534fa0c9
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Since e1a5ed51db, missing ir_model_access are warned at the end of an install.
Unfortunately, it is still possible that a module add a model and another module add the corresponding ir_model_access.
Since runbot install all module at once, this won't be spot until a single module build is ran when each module is
installed independently.
This commit proposes to move the check at the end of each module.
This commit also format the log in orther to ease copy/paste of proposed rules in case of multiple new models
and add module to xmlids.
Note that the log may be repeated multiple times if multiple modules redefine this model.
Linked to #59193closesodoo/odoo#59213
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
* remove useless OdooTestRunner
* don't log results & time per-file, log a module-level tally instead
* add number of tests to post-test results
* generate a single test suite per module (see note)
* use the previous item to split out the at_install test-running in
two steps: generating the suite for the module then running that
suite, this way for modules which have no test, or for
which all tests have been deselected by test tags, we can avoid some
of the setup necessary to prepare for running tests but possibly
quite expensive (e.g. `setup_models`)
Note: single test suite per module
I wanted to stop creating a test result for (essentially) every file
in the module, however because of the class-level ``addCleanup``, a
TestResult can't be reused by independent suites:
In order to run class-level cleanup, the test suite checks between
tests if the test it's *preparing* to run is in the same class as the
last test it ran, and if not applies the class-level cleanup.
The problem is that the "previous test class" is stored on the result
object, which is never cleaned up, and the "between tests" check is
really performed *before each test*.
This means when reusing results across suites it will run the
class-level cleanup at the end of one suite and immediately at the
start of the next, which will cause issues if class-level cleanups are
not idempotent (thankfully ``TestTestCursor`` has a non-idempotent
``tearDownClass` which let me discover the error).
Possible fixes are:
* don't reuse results
* clear the relevant states / attributes between suites
* put individual suites in a Big Suite for running
The latter seems simpler: just create a single suite for the entire
odoo-level module instead of creating one suite per test module.
Note to the note: the case of nested suite is taken in account, the
"end of suite" cleanup only runs at the end of the top-level suite, so
technically we don't have to unwrap suites for *that* purpose, we're
doing so in order to filter the test cases inside the suites. But
maybe we could integrate this feature to the suites themselves...
closesodoo/odoo#55185
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
That's a not-very-useful subset of OdooTestResult, so:
* make results merge-able (aka add ability to update a result with the
contents of another)
* remove support for test data files, and transmission of the
assertion report thing through the data-files loading
* replace "legitimate" uses of assertion report by test result
* have run_unit_tests manipulate and return a result instead of weird
flags & ternaries
Currently there are a few issues with testing reporting (at the
command-line):
1. there is a global report after at_install tests, but it gets
"scrolled off" by long post_install tests, and is thus easy to
miss
2. with test tags, it's easy to fat finger a typo and run 0 tests,
which look like everything's running fine (no failure)
To improve this, print a global report at shutdown (in
`--stop-after-init` mode if tests are enabled) which recapitulates the
test results *and prints a warning if no tests were run at all*.
Also update the reporting collection to make this more reliable:
* have `run_unit_tests` return `None` if it has run no tests, the
assertion reporting machinery counts this as neither success nor
failure which is exactly what we want
* have load_test only report a success *if files were actually
loaded* (by having `load_data` return that information)
Task 2301268
closesodoo/odoo#54812
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Debugging/improvement of an at install test may be tedious because of
the need to update the module, spending most of the time checking
tables and xml file.
1. This commit proposes to allow to execute test without installing or
updating a module. The test is still executed during the loading, and
the behavior should be close to an execution of tests during an update.
Co-authored-by rco-odoo <rco@odoo.com>
2. keep previous behavior if -i or -u is given
Three solutions were possible:
- The clean one that changes dev habits.
When test-enable or test_tags is given, all tests are always executed,
a test_tags is needed to select tests to execute:
Example: `-u module --test_enable` becomes `-u module --test-tags /module` to keep the same behaviour
This solution is the simplest, and executed tests does not depends on database already installed modules.
- The conservative solution.
When giving -i or -u, the behavior stays the same as before. When giving test-enable
without -i/-u, test are executed on all installed modules.
- The intermediate solution:
When no test_tags is given but a -i and -u is given, only the given modules are tested.
This is quite close to the second solution except that a -i module on a new database won't
test all dependencies on the first install.
The chosen solution is the second one to minimize changes on dev old habits,
only an almost unused feature is impacted: using test-enable without any -i or -u.
Before this pr only post install tests were executed in this case. Now at_install tests are also executed.
This combination is actually used by runbot to execute post_install test in parallel, but a `--test-tags -at_install`
tag is given so nothing to worry about here.
closesodoo/odoo#53499
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
When installing a database from scratch, marking all packages (in the
module graph) as 'update' forces the migration manager to retrieve all
migrations scripts... for nothing.
X-original-commit: fd8f3c73c8c30162034c7712fa0993e2408fba89
Performances from a general point of view can be difficult to track.
This commit proposes to improve logs in two ways:
The current logs only use the sql_counter, wich will only be updated
when a cursor is closed. In a test-enable install, this counter
is actually the queries of the tests wince the install cursor is
open untill the end. The first fix is to use bot sql_counter and
sql_log_count to have total queries untill now on closed cursor,
but also the current number of queries of the current cursor.
This means that the new log format will be
{nb} modules loaded in {time}, {loading_querie} (+{test_cr_queries}) queries
instead of
{nb} modules loaded in {time}, {tests_cr__queries}queries
Nothe that in the current version, {nb} is actually the total number of
loaded modules until now.
This commit also add an equivalent end log by module and change the
loglevel of module start on install (mainly usefull if an error occurs
before anything else is logged hidding the module causing this error.)
A cleaner runbot logger is also added, in order to be abble to call
_logger.runbot( instead of _logger.log(25. This will clarify the purpose
of such a log level.
closesodoo/odoo#47283
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Before this commit:
- Install a module
- Uninstall the previously installed module
- The registry will complain that some dependencies may be missing
for the module being uninstalled
This happens because we check after the installation / upgrade of
modules that none have been left in a transient state to verify that new
dependencies have been properly installed and loaded, this applies to
'to install' and 'to upgrade' states however it's not the same for 'to
remove' states, as the process of uninstall happens much later in the
code.
After this commit, simply uninstalling modules will not trigger this
error log.
Do note that in case of a problem with an uninstall, the function
"reset_module_states" will tackle the case of leftover transient states.
closesodoo/odoo#47495
X-original-commit: c095a28314f78d1d9854e5e9bcb26014922a4f12
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Sometimes, constraints are not delayed during a module upgrade: the
module is 'to upgrade' but in 'init' mode :-/
Fix this by relying on the module's state only.
closesodoo/odoo#45179
X-original-commit: 878b6538961a9b6a5fe1c6ac68d4ff39391abcdf
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
In some case, a pre-migration script calls `xml_import.parse` to force
reload a no-update data in pre-script. It is possible that fields are
marked to recompute in the process. The problem is that the reference to
the field added in `env.all.tocompute` won't be the same as the
reference of the same field after calling `registry.setup_models()`,
leading to an infinite loop while recomputing this field, since
`fields.__get__` wont find `self` in `env.all.tocompute`.
This problem was discovered when trying to migrate a database with all
modules installed (no demo data) from 12.0 to 13.0 (for commit
references: http://runbot.odoo.com/runbot/build/1176576 with database
comming from http://runbot.odoo.com/runbot/build/1171714).
This commit adds a check before executing `registry.setup_models()` in
order to log when some fields to compute remain before breaking fields
references, and adds a `flush()` after pre-scripts to fix the current
issue.
X-original-commit: 41b9d810066774736c4077bba1dc9c9fba004f48
Before this commit models with the `_transient` flag were ignored in
ir.model.access verifications. Only an implicit ir.rule with the
domain (create_uid=user.id) was applied to avoid most side effects.
The problem is that, often, the security does not lie in side-effects
of abusing of somebody else's wizard record but in the fact that the
wizard methods blindly trust only the right users are creating these
records. Too often, too many sudo were used and creating wizard with
chosen values could lead to an abuse scenario.
Instead, explicitly require the developer to declare security rules
the same way as on any other model.
Studio customization is an exception, a data module added to ir_module
but that is never added to graph since there is no manifest.
OPW #2180885closes#43880closesodoo/odoo#43940
X-original-commit: 4051ac83b5e7cc1cb91a264b51401ad5601ad4e2
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
When migrating a database, load_marked_modules will be called multiple times, alternating
to upgrade and to install modules. The main reason for this is still a litle confusing
but it as the side effect to log "Unmet dependencies" error multiple time in add_modules,
even if the dependency will be resolved later.
This commit removes the error level for this log, and replace it by another check,
performed at the end, logging any module in "to install"/"to upgrade" state.
Also log removed module as info (25), not warning. This may be changed latter
closesodoo/odoo#43797
X-original-commit: c5d6a3977de85fb974a4940fb11d54ef847e08e4
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This branch is the combination of several optimizations in the ORM:
* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;
* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;
* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);
* make method `modified` take advantage of inverse fields to inverse
dependencies;
* filter records by evaluating a domain on records in Python;
* a computed field with `readonly=False` behaves like a normal field
with an onchange method;
* computed fields are computed in superuser mode by default.
Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.
closesodoo/odoo#35659
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
An unintended side-effect of #29431 is apparently that auto_install
applications automatically get reinstalled when uninstalled, which was
not the goal.
Move the auto_install selection back into db.py, this version seems to
work even though the previous attempt was apparently unsuccessful.
closesodoo/odoo#35180
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
1. Make test logs clearer & remove redundancies
Instead of having an ERROR log right when the test fails then print
the useful / relevant information at the end of the test suite,
immediately print the traceback. Keep the final summary. Also avoids
having to wait for the entire test suite to end before a dev' can know
the failure details of a specific test.
Done by working at a lower level and replacing the custom
test stream mess by a custom Result class which prints and formats the
information we want. Replace TextTestRunner by a bare-bones custom
Runner object to tie it in.
2. Provide useful location information on test failure
Leverage the work above to log the test function's failure location:
previously logging would point to within TestStream which is not
useful.
Here, on failure the traceback is used to discover the caller info and
point to the test line which fails instead. similar to unittest's
_exc_info_to_string (https://github.com/python/cpython/blob/93e8aa62cfd0a61efed4a61a2ffc2283ae986ef2/Lib/unittest/result.py#L173).
3. Replace direct logging in browser_js by raising errors
Properly marks the test as in error, and the error traceback points to
the tour definition / launcher (python side) rather than common.py
and/or module.py.
Also removes unused dbname parameter that was added in
/278ed718e9805edf088642ba10d3b7c4e5716c31/openerp/modules/module.py#L361
for nor visible reason
closesodoo/odoo#34996
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
As most of the tests are using demo data, a check is performed to verify
that they are loaded.
A best practice is to write tests that does not depend on demo data but
with this check, it's not possible to launch them without demo data.
With this commit, the check is removed, allowing to launch test even
without demo data loaded.
closesodoo/odoo#33531
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Currently, auto_install is triggered when all dependencies get
installed, but there are cases where one would want such trigger on
only a subset thereof.
e.g. we want `website_sale_dashboard` to auto-install when
`website_sale` is installed. Currently, it requires `web_dashboard` to
also be auto-installed otherwise `website_sale_dashboard` would "wait"
for both dependencies to be explicitly installed before the
auto-install triggers. That's despite `web_dashboard` not being very
useful on its own. More generally this is an issue with technical
modules which need to be marked as auto_install so as not to block
e.g. bridge modules from automatically installing.
This change allows setting `auto_install` to a subset of `depends`:
* if auto_install is set to `False`, the module does not get
automatically installed (no change in semantics)
* if auto_install is set to `True`, the module gets automatically
installed if and only if all its dependencies are installed (also no
change in semantics)
* if auto_install is set to a list of dependencies, the module will be
installed when all *these* dependencies are installed, other
dependencies (excluded from auto_install) will be installed
alongside as a consequence
* auto_install can be set to an empty list, in this case the module
will always be automatically installed regardless of its
dependencies (and will force their installation).
So after this change, `web_dashboard`'s auto_install can be set to
`False` (such that it's not installed if no module defining dashboards
is installed) and `website_sale_dashboard`'s manifest can be edited
to:
'auto_install': ['website_sale']
possibilities:
# no automatic installation
'depends': ['a', 'b'],
'auto_install': False
# automatic installation if both a and b are installed
'depends': ['a', 'b'],
'auto_install': True
# automatic installation if both a and b are installed (explicit)
'depends': ['a', 'b'],
'auto_install': ['a', 'b']
# automatic installation if b is installed, a will get forcefully
# installed if it isn't yet
'depends': ['a', 'b'],
'auto_install': ['b']
# always automatically installed, will cause the installation of
# its dependencies even if they're not marked explicitly
'depends': ['a', 'b'],
'auto_install': []
Task 1851328
closesodoo/odoo#29431
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
After a module upgrade, validate the architecture of the module's views that
are impacted by updates, but have not been checked yet. Before this patch,
views were checked twice on average.
This patch speeds up the installation of modules by about 15% without demo
data. With demo data, the speedup is around 10%.
The registry loading system should not alter databases unless it is
asked to do so, by a module installation or update instruction.
This property should hold true as well for database bootstrap, and this
is what this commit changes..
In order to avoid any behavior change for command-line users, an
implicit `-i base` is assumed when starting the server from the
command-line with `-d <db>`, causing the db boostrap to happen if the
database did not exist yet.
Task 1856935
Currently, if installing a module's demo data fails (because the
module was uninstalled and some were left over and can't be
reinstalled, or because they're broken, or…) the installation of the
module fails and further installations are skipped (?).
Since demo data are non-essential (though required to run tests),
rather than fail everything if they fail just roll them back and
notify the user.
This change generates a warning log *and* shows a notification popup
to the end user.
Try to remove cr.commit (and rollback) from module and db install:
* put a savepoint around test data loading
* remove a bunch of commits sprinkled throughout
* remove rollback on data loading failure (assuming it bubbles up, the
entire module's installation should be rolled back)
* add commit right before the tests are run, so they can run isolated
and still see whatever was done when installing their module
* convert a few explicit closing to context managers
Raphael Collet