Issue:
On databases with lots of products (400k+ `product.product`), the action
Inventory > Operations > Replenishment takes a lots of memory to
complete, leading to "Out of Memory" errors in some use cases.
Analysis:
When triggering the action, we need to recompute the
`virtual_available` of a lot of products (regular occurrence when a
DB for example has a lot of variants). The compute
`_compute_quantities` uses the ORM `filtered` method to filter out
`services` out of the `product.product` we have. This method is
memory hungry, as for each `_prefetch_ids` (capped at 1000, cf.
`PREFETCH_MAX`) it will store in cache the `product_tmpl_id.type`,
which stores all fields of the product_template + the type,
bloating the cache with *useless* data, as it's not used beyond this
filtering process.
Solution:
Disable the prefetcher, the bulk of the work is done in
`_compute_quantities_dict`, which makes a lot of `read_group` and
doesn't benefit much from fields already cached (this function
itself also disables the prefetcher internally at some point).
Also correcting the looping order in `_get_orderpoint_action` to
avoid multiple cache invalidation when those can be batched in 1 call.
The `PREFETCH_MAX` makes sure the compute is only executed with 1k
records at a time, so explicitly dividing `all_products_ids` into
batches of 5k is unnecessary.
Results:
- Memory usage of `_get_orderpoint_action`:
| Memory usage | Before | After |
|--------------|---------|---------|
| Total Memory | 1.8 GiB | 363 MiB |
| Allocations | 10170 | 2804 |
(profiler data available on referenced ticket)
- Execution time of `_get_orderpoint_action`:
| `product.product` count | Before | After |
|-------------------------|----------|----------|
| 1k | 1.3s | 1.1s |
| 10k | 4.1s | 3.8s |
| 100k | 30s | 25.9s |
| 400k+ | 2min 25s | 1min 39s |
As we can observe, we've reduced drastically the memory usage of the
function and also the number of allocations quite drastically. As a
side effect there is also a slight speed improvement at small
scale, and at large scale the gains are up to ~40%
faster.
Reference:
opw-3415087
closesodoo/odoo#138423
X-original-commit: 41442574159429ce29aa5c5adf0e1a0d8a7046dd
Signed-off-by: William Henrotin (whe) <whe@odoo.com>
Signed-off-by: Piryns Victor (pivi) <pivi@odoo.com>
Victor Piryns (pivi)
8fd1b20d20