Xavier-Do
8a7990735d
This pr proposes an auto retry mechanism for tests. This shouldn't impact normal testing: tests are not supposed to fail, but the growing number of tests and pull requests can lead to some bottleneck when a staging fails because of a random error. This mechanism should help to reduce splits/the need to retry a failed pr. This branch have been tested with the nightly multi build, creating 40 identical build without test-tags to disable know random errors. This multi build is used to detect test failing randomly, this is an excellent candidate to detect the effect of the retry. On average, with the current base of this pull request, there is between 10 en 15 failures over 40 build. With the auto retry mechanism, only 1 build failed over 40 builds since the same error was triggered twice. This is simply because with the retry mechanism, an error that has a probability of p to fail randomly will still have a probability of p² to fail with the retry mechanism. A error that occurs 10% of the time should only appear 1% of the time with one retry. In most of the case, the retry is sucessfull: https://runbot.odoo.com/runbot/build/10053257 The current solution to allow to enable this mechanism only in some cases (staging) is to check an environment variable "ODOO_TEST_FAILURE_RETRIES" that defines a number of retry. This will allow to retry more than once if an error still occurs to ofen with the autoretry. The mechanism will run multiple time the same test on the same test_case, meaning that some modification on self may impact the second execution. The following code is an example of how this could be problematic, but also a good example to test the auto-retry mechanism. ```python class TestRetry(HttpCase): def test_fail(self): self.t = getattr(self, 't', 0) + 1 if True or self.t == 1: import logging _logger = logging.getLogger('test_a') with self.assertLogs(level="ERROR"): _logger.error("This shouldn't be log at all") with mute_logger('test_a'): _logger.error("This shouldn't be logged (mute)") _logger.error("This should be log") ``` As we can see here the error logs are also managed, and emit at a lower level the first time, butany log higher than 25 will make the test "failed" and the autoretry mechanism will be triggered. The second time, everything is logged normally. We also need to replace Traceback by _Traceback to avoid being catched by runbot Traceback detection regexes. The inspiration here commes from the assertLogs, that replace all handlers. The mute_logger had to be adapted to use the same strategy, so that quite_logger won't detect logs catched by mute_logger or assertLogs. closes odoo/odoo#76336 Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Odoo
Odoo is a suite of web based open source business apps.
The main Odoo Apps include an Open Source CRM, Website Builder, eCommerce, Warehouse Management, Project Management, Billing & Accounting, Point of Sale, Human Resources, Marketing, Manufacturing, ...
Odoo Apps can be used as stand-alone applications, but they also integrate seamlessly so you get a full-featured Open Source ERP when you install several Apps.
Getting started with Odoo
For a standard installation please follow the Setup instructions from the documentation.
To learn the software, we recommend the Odoo eLearning, or Scale-up, the business game. Developers can start with the developer tutorials