Victor Piryns (pivi)
8fd1b20d20
Issue: On databases with lots of products (400k+ `product.product`), the action Inventory > Operations > Replenishment takes a lots of memory to complete, leading to "Out of Memory" errors in some use cases. Analysis: When triggering the action, we need to recompute the `virtual_available` of a lot of products (regular occurrence when a DB for example has a lot of variants). The compute `_compute_quantities` uses the ORM `filtered` method to filter out `services` out of the `product.product` we have. This method is memory hungry, as for each `_prefetch_ids` (capped at 1000, cf. `PREFETCH_MAX`) it will store in cache the `product_tmpl_id.type`, which stores all fields of the product_template + the type, bloating the cache with *useless* data, as it's not used beyond this filtering process. Solution: Disable the prefetcher, the bulk of the work is done in `_compute_quantities_dict`, which makes a lot of `read_group` and doesn't benefit much from fields already cached (this function itself also disables the prefetcher internally at some point). Also correcting the looping order in `_get_orderpoint_action` to avoid multiple cache invalidation when those can be batched in 1 call. The `PREFETCH_MAX` makes sure the compute is only executed with 1k records at a time, so explicitly dividing `all_products_ids` into batches of 5k is unnecessary. Results: - Memory usage of `_get_orderpoint_action`: | Memory usage | Before | After | |--------------|---------|---------| | Total Memory | 1.8 GiB | 363 MiB | | Allocations | 10170 | 2804 | (profiler data available on referenced ticket) - Execution time of `_get_orderpoint_action`: | `product.product` count | Before | After | |-------------------------|----------|----------| | 1k | 1.3s | 1.1s | | 10k | 4.1s | 3.8s | | 100k | 30s | 25.9s | | 400k+ | 2min 25s | 1min 39s | As we can observe, we've reduced drastically the memory usage of the function and also the number of allocations quite drastically. As a side effect there is also a slight speed improvement at small scale, and at large scale the gains are up to ~40% faster. Reference: opw-3415087 closes odoo/odoo#138423 X-original-commit: 41442574159429ce29aa5c5adf0e1a0d8a7046dd Signed-off-by: William Henrotin (whe) <whe@odoo.com> Signed-off-by: Piryns Victor (pivi) <pivi@odoo.com>
Odoo
Odoo is a suite of web based open source business apps.
The main Odoo Apps include an Open Source CRM, Website Builder, eCommerce, Warehouse Management, Project Management, Billing & Accounting, Point of Sale, Human Resources, Marketing, Manufacturing, ...
Odoo Apps can be used as stand-alone applications, but they also integrate seamlessly so you get a full-featured Open Source ERP when you install several Apps.
Getting started with Odoo
For a standard installation please follow the Setup instructions from the documentation.
To learn the software, we recommend the Odoo eLearning, or Scale-up, the business game. Developers can start with the developer tutorials