pengyuthon/odoo_source
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[IMP] stock: add least_packages retrieval strategy

Browse Source 
Community PR: https://github.com/odoo/odoo/pull/91187

## Proposed feature and relevance
Add a new retrieval strategy for gathering products: "Least packages".
The goal of this strategy is to select specific product quantities from stock by using as few packages as possible, to allow for efficient logistics operations and avoid unnecessary opening/unpacking of packages.

This strategy will behave as follows:
- In case a negative qty is requested this strategy will behave as a FIFO strategy.
- Otherwise, the strategy will determine a combination using the least amount of packages that sum to the requested quantity, and adapt the provided domain to select only the quantities in these packages. This way no packages need to be opened to select the requested quantity.
- In case an exact matching of current packages to get the requested quantity is not possible, a selection of packages is made where the largest packages are selected one by one until the requested quantity is reached, or until no more quantities are available. For this approach, one package needs to be unpacked as it will not be used completely. The final selected package that will be unpacked will be the smallest one possible, to try to unpack smaller packages over bigger packages.
- In case the search for a minimal matching of packages results in an out-of-memory situation, the strategy will fall back to a FIFO strategy.
- Loose items (not tied to a package) are considered as single-item packages for the purposes of package selection.
- **NOTE:** To force the selection strategy to "clean up" certain packages that have previously been opened, it is recommended to unpack these packages into single items, as they will then be consumed to "fill up" requested quantities in later retrievals.

## Changes to allow more complex strategies
To allow for the integration of this new strategy the structure of `_gather()` needed to be adapted to allow a removal strategy to provide not only the SQL `ORDER BY` string but also adapt the domain used to retrieve the requested quantities.
This change is reflected in the method signature of `_get_removal_strategy_domain_order()`.

## How the strategy works
The strategy does one extra query of the database, to determine the different packages and their respective quantities (using the provided domain).
Loose items (that do not belong to a package) are added as single-item packages to also consider them.

For minimal exact package selection, an A* algorithm is used with 2 performance modifications:
- Using the information that the packages are sorted by size (large -> small).
- Using the information that the generated subtree for a specific package of size x is the same as all other packages of size x at a specific node.

By using the A* algorithm, as soon as a matching packaging selection is found, it is ensured to be optimal and other combinations do not need to be considered.
In case the A* algorithm runs out of memory the provided domain is just returned, which equates to a FIFO strategy.
In case the A* algorithm does not find an exact matching of packages for the requested quantity, a selection is made using the least amount of packages, unpacking the smallest possible package (or until no more quantities are available, in which case effectively all quantities in the domain are returned).

## Used heuristic
The heuristic used for the A* algorithm exploits the knowledge that our packages to select from are ordered by the highest to the lowest quantity (on the database side, SQL `ORDER BY`).
It assumes the "best case scenario" as an estimate for the amount of packages remaining, which is if we were to "match" the required quantity using fractional multiples of the next biggest package size.

***Example**:
Remaining quantity to select: 27
Remaining package sizes to select from: [10, 10, 5, 5, 5, 1, 1, 1]
Heuristic for choosing package with qty 10 as next package: 27/10 = 2.7
Heuristic for choosing package with qty 5 as next package: 27/5 = 5.4
Heuristic for choosing package with qty 1 as next package: 27/1 = 27*

This way the heuristic is always a "best case" (fractional) for the amount of packages that still need to be selected.

As required by the A* algorithm this also results in an underestimation as the fractional result is not possible, and it is not certain that there are still enough packages of this size that can actually be selected, but favours larger packages to make the search converge more quickly.

## Performance
### CPU
Two test cases were run to evaluate the performance.

***Total nr of pkg**: The total amount of different packages generated during data generation
**Qty to select**: The total quantity that needs to be selected
**Selected nr of pkg**: The amount of packages that was ultimately selected by the algorithm
**Data generation time**: The time required to generate all packages
**Assignment time**: The total time to run `_action_assign()` for the move
**A\* time**: The time to run the `_run_least_packages_removal_strategy_astar()` method*

#### First test case
This scenario simulates a case with a large quantity to select from a large amount of packages, but of relatively homogenous size.

Generate 30 different package sizes between 2 and 1000. For each package size, generate 100 to 250 packages each. Also add 100 to 250 "loose items".

The quantity to select is chosen between 1 and 50000.

|         RESULTS          |  RUN 1   |   RUN 2   |   RUN 3    |   RUN 4   |  RUN 5   |    AVG     |
|--------------------------|----------|-----------|------------|-----------|----------|------------|
| Total nr of pkg          |     5298 |      5772 |       5855 |      5533 |     5179 |     5527.4 |
| Qty to select            |    10319 |     20762 |      44422 |     39306 |     1783 |    23318.4 |
| Selected nr of pkg       |       11 |        22 |         48 |        41 |        3 |         25 |
| Data generation time (s) | 64.59124 | 92.916682 | 133.673384 | 82.711438 | 85.54542 | 91.8876328 |
| Assignment time (s)      | 4.060216 | 11.212937 |   4.763776 |   8.60248 | 3.461941 |    6.42027 |
| A* time (s)              |     0.28 |      6.88 |       0.21 |      3.96 |     0.03 |      2.272 |

#### Second test case
This scenario generates a large amount of small packages of different sizes.

Generate 250 different package sizes between 2 and 10. For each package size, generate 5 to 15 packages each. Also add 5 to 15 "loose items".

The quantity to select is chosen between 1 and 10000.

|         RESULTS          |   RUN 1   |   RUN 2   |   RUN 3   |   RUN 4   |  RUN 5   |    AVG     |
|--------------------------|-----------|-----------|-----------|-----------|----------|------------|
| Total nr of pkg          |      2485 |      2406 |      2491 |      2555 |     2542 |     2495.8 |
| Amount to assign         |        27 |      4029 |      3966 |      3049 |     1833 |     2580.8 |
| Selected nr of pkg       |         3 |       411 |       411 |       306 |      184 |        263 |
| Data generation time (s) | 18.502933 | 18.692048 | 19.969955 | 19.169106 | 19.16439 | 19.0996864 |
| Assignment time (s)      |   1.35635 |  7.958852 |  8.470518 |  6.300406 | 3.895457 |  5.5963166 |
| A* time (s)              |      0.01 |      0.07 |      0.77 |      0.56 |     0.31 |      0.344 |

### Memory
In cases with large amounts of different-sized packages (200+ different sizes and 200+ packages required to get an exact match) the memory performance of the A* strategy might become problematic.
When lots of packages have the same size (which would be considered typical in cases where there are large amounts of packages) the specific modifications to the A* algorithm ensure that the strategy still delivers good memory performance for this use case as the tree size can be significantly reduced, even with large quantities.

### IO
This strategy only requires one extra SQL query to fetch the different quantities and their respective packages.

## Limitations
No tests for this strategy were done for non-integer quantities. In these cases the performance of the A* algorithm might not be satisfactory and another approach might be preferred.
More research could be done for specific use cases where packaged quantities are non-integer values.

closes odoo/odoo#91187

Signed-off-by: William Henrotin (whe) <whe@odoo.com>
This commit is contained in:
Pieter Claeys (clpi)
2023-03-24 23:41:54 +01:00
parent 67e197870a
commit 1beb99b73d
4 changed files with 303 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
addons/product_expiry/models/stock_quant.py
+3 -3
View File
@@ -28,7 +28,7 @@ class StockQuant(models.Model):
return res
@api.model
def _get_removal_strategy_order(self, removal_strategy):
def _get_removal_strategy_domain_order(self, domain, removal_strategy, qty):
if removal_strategy == 'fefo':
return 'removal_date, in_date, id'
return super(StockQuant, self)._get_removal_strategy_order(removal_strategy)
return domain, 'removal_date, in_date, id'
return super(StockQuant, self)._get_removal_strategy_domain_order(domain, removal_strategy, qty)
addons/stock/data/stock_data.xml
+4
View File
@@ -13,6 +13,10 @@
<field name="name">Closest Location</field>
<field name="method">closest</field>
</record>
<record id="removal_least_packages" model="product.removal">
<field name="name">Least Packages</field>
<field name="method">least_packages</field>
</record>
</data>
<data noupdate="1">
<!-- Resource: stock.location -->
addons/stock/models/stock_quant.py
+118 -10
View File
@@ -1,7 +1,8 @@
# -*- coding: utf-8 -*-
# Part of Odoo. See LICENSE file for full copyright and licensing details.
import heapq
import logging
from collections import namedtuple
from ast import literal_eval
from collections import defaultdict
@@ -564,20 +565,126 @@ class StockQuant(models.Model):
loc = loc.location_id
return 'fifo'
def _run_least_packages_removal_strategy_astar(self, domain, qty):
# Fetch the available packages and contents
query = self._where_calc(domain)
query_str, params = query.select('package_id', 'SUM(quantity - reserved_quantity) AS available_qty')
query_str += ' GROUP BY package_id ORDER BY available_qty DESC'
self._cr.execute(query_str, params)
qty_by_package = self._cr.fetchall()
pkg_found = False
# Items that do not belong to a package are added individually to the list, any empty packages get removed.
for idx, elem in enumerate(qty_by_package):
if elem[0] is None:
del qty_by_package[idx]
qty_by_package.extend([(None, 1) for _ in range(int(elem[1]))])
elif elem[1] == 0:
del qty_by_package[idx]
else:
pkg_found = True
if not pkg_found:
return domain
size = len(qty_by_package)
class PriorityQueue:
def __init__(self):
self.elements = []
def empty(self) -> bool:
return not self.elements
def put(self, item, priority):
heapq.heappush(self.elements, (priority, item))
def get(self):
return heapq.heappop(self.elements)[1]
def heuristic(node):
if node.next_index < size:
return len(node.taken_packages) + node.count_remaining / qty_by_package[node.next_index][1]
return len(node.taken_packages)
def generate_domain(node):
selected_single_items = []
single_item_ids = False
for pkg in node.taken_packages:
if pkg[0] is None:
# Lazily retrieve ids for single items
if not single_item_ids:
single_item_ids = self.search(expression.AND([[('package_id', '=', None)], domain])).mapped('id')
selected_single_items.append(single_item_ids.pop())
expr = [('package_id', 'in', [elem[0] for elem in node.taken_packages if elem[0] is not None])]
if selected_single_items:
expr = expression.OR([expr, [('id', 'in', selected_single_items)]])
return expression.AND([expr, domain])
Node = namedtuple("Node", "count_remaining taken_packages next_index")
frontier = PriorityQueue()
frontier.put(Node(qty, (), 0), 0)
best_leaf = Node(qty, (), 0)
try:
while not frontier.empty():
current = frontier.get()
if current.count_remaining <= 0:
return generate_domain(current)
# Keep track of processed package amounts to only generate one branch for the same amount
last_count = None
i = current.next_index
while i < size:
pkg = qty_by_package[i]
i += 1
if pkg[1] == last_count:
continue
last_count = pkg[1]
count = current.count_remaining - pkg[1]
taken = current.taken_packages + (pkg,)
node = Node(count, taken, i)
if count < 0:
# Overselect case
if best_leaf.count_remaining > 0 or len(node.taken_packages) < len(best_leaf.taken_packages) or (len(node.taken_packages) == len(best_leaf.taken_packages) and node.count_remaining > best_leaf.count_remaining):
best_leaf = node
continue
if i >= size and count != 0:
# Not enough packages case
if node.count_remaining < best_leaf.count_remaining:
best_leaf = node
continue
frontier.put(node, heuristic(node))
except MemoryError:
_logger.info('Ran out of memory while trying to use the least_packages strategy to get quants. Domain: %s', domain)
return domain
# no exact matching possible, use best leaf
return generate_domain(best_leaf)
@api.model
def _get_removal_strategy_order(self, removal_strategy):
def _get_removal_strategy_domain_order(self, domain, removal_strategy, qty):
if removal_strategy == 'fifo':
return 'in_date ASC, id'
return domain, 'in_date ASC, id'
elif removal_strategy == 'lifo':
return 'in_date DESC, id DESC'
return domain, 'in_date DESC, id DESC'
elif removal_strategy == 'closest':
return 'location_id ASC, id DESC'
return domain, 'location_id ASC, id DESC'
elif removal_strategy == 'least_packages':
if qty > 0:
return self._run_least_packages_removal_strategy_astar(domain, qty), 'in_date ASC, id'
return domain, 'in_date ASC, id'
raise UserError(_('Removal strategy %s not implemented.') % (removal_strategy,))
def _gather(self, product_id, location_id, lot_id=None, package_id=None, owner_id=None, strict=False):
def _gather(self, product_id, location_id, lot_id=None, package_id=None, owner_id=None, strict=False, qty=0):
removal_strategy = self._get_removal_strategy(product_id, location_id)
removal_strategy_order = self._get_removal_strategy_order(removal_strategy)
domain = [('product_id', '=', product_id.id)]
if not strict:
if lot_id:
@@ -593,7 +700,8 @@ class StockQuant(models.Model):
domain = expression.AND([[('owner_id', '=', owner_id and owner_id.id or False)], domain])
domain = expression.AND([[('location_id', '=', location_id.id)], domain])
return self.search(domain, order=removal_strategy_order).sorted(lambda q: not q.lot_id)
domain, order = self._get_removal_strategy_domain_order(domain, removal_strategy, qty)
return self.search(domain, order=order).sorted(lambda q: not q.lot_id)
@api.model
def _get_available_quantity(self, product_id, location_id, lot_id=None, package_id=None, owner_id=None, strict=False, allow_negative=False):
@@ -647,7 +755,7 @@ class StockQuant(models.Model):
"""
self = self.sudo()
rounding = product_id.uom_id.rounding
quants = self or self._gather(product_id, location_id, lot_id=lot_id, package_id=package_id, owner_id=owner_id, strict=strict)
quants = self or self._gather(product_id, location_id, lot_id=lot_id, package_id=package_id, owner_id=owner_id, strict=strict, qty=quantity)
reserved_quants = []
if float_compare(quantity, 0, precision_rounding=rounding) > 0:
addons/stock/tests/test_quant.py
+178 -1
View File
@@ -1,6 +1,5 @@
# -*- coding: utf-8 -*-
# Part of Odoo. See LICENSE file for full copyright and licensing details.
from contextlib import closing
from datetime import datetime, timedelta
from unittest.mock import patch
@@ -796,6 +795,184 @@ class StockQuant(TransactionCase):
quant.invalidate_recordset(['quantity'])
self.assertEqual(quant.quantity, 11)
class StockQuantRemovalStrategy(TransactionCase):
def setUp(self):
super().setUp()
self.least_package_strategy = self.env['product.removal'].search(
[('method', '=', 'least_packages')])
self.product = self.env['product.product'].create({
'name': 'Product',
'type': 'product',
})
self.product.categ_id.removal_strategy_id = self.least_package_strategy.id
self.stock_location = self.env['stock.location'].create({
'name': 'stock_location',
'usage': 'internal',
})
def _generate_data(self, packages_data):
move = self.env['stock.move'].create({
'name': 'Test Least Package',
'product_id': self.product.id,
'product_uom': self.product.uom_id.id,
'location_id': self.ref('stock.stock_location_suppliers'),
'location_dest_id': self.stock_location.id,
})
move._action_confirm()
ml_vals_list = []
ml_common_vals = {
'move_id': move.id,
'product_id': self.product.id,
'product_uom_id': self.product.uom_id.id,
'location_id': self.ref('stock.stock_location_suppliers'),
'location_dest_id': self.stock_location.id,
}
packages = self.env['stock.quant.package'].create(
[{}] * sum(p[1] for p in packages_data if p[0]))
for package_size, number_of_packages in packages_data:
if not package_size:
ml_vals_list.append(dict(**ml_common_vals, **{
'qty_done': number_of_packages,
}))
continue
for dummy in range(number_of_packages):
package = packages[0]
packages = packages[1:]
ml_vals_list.append(dict(**ml_common_vals, **{
'qty_done': package_size,
'result_package_id': package.id,
}))
self.env['stock.move.line'].create(ml_vals_list)
move._action_done()
def test_least_package_removal_strategy_priority_to_package(self):
"""
Tests the least package removal strategy in a use case where only one package needs to be selected.
It should only return the quantity of a single size 1000 package.
"""
packages_data = [
(False, 2000),
(5, 10),
(50, 10),
(1000, 2),
]
self._generate_data(packages_data)
# Out 1000 should selecte a package with 1000 units inside
move = self.env['stock.move'].create({
'name': 'Test Least Package',
'product_id': self.product.id,
'product_uom': self.product.uom_id.id,
'location_id': self.stock_location.id,
'location_dest_id': self.ref('stock.stock_location_customers'),
'product_uom_qty': 1000,
})
move._action_confirm()
move._action_assign()
self.assertEqual(len(move.move_line_ids), 1, 'Only one pack could be use')
self.assertTrue(
move.move_line_ids.package_id,
'A package should be selected, priority to package even if there is enough quantity without package'
)
def test_least_package_removal_strategy_simple_usecase(self):
"""
Tests the least package removal strategy in a simple "typical" use case.
It should return a minimal exact matching for the requested quantity.
"""
packages_data = [
(5, 10),
(50, 10),
(1000, 2),
]
self._generate_data(packages_data)
# Out 1000 should select a package with 1000 units inside
move = self.env['stock.move'].create({
'name': 'Test Least Package',
'product_id': self.product.id,
'product_uom': self.product.uom_id.id,
'location_id': self.stock_location.id,
'location_dest_id': self.ref('stock.stock_location_customers'),
'product_uom_qty': 1280,
})
move._action_confirm()
move._action_assign()
self.assertEqual(len(move.move_line_ids), 12)
self.assertRecordValues(
move.move_line_ids,
[{'reserved_qty': 1000}] +
[{'reserved_qty': 50}] * 5 +
[{'reserved_qty': 5}] * 6
)
def test_least_package_removal_strategy_not_possible(self):
"""
Tests the least package removal strategy in the case where an exact matching
of packages is not possible for the requested amount.
It should return the best leaf from the A* search.
"""
packages_data = [
(False, 2),
(5, 2),
(10, 5),
]
self._generate_data(packages_data)
move = self.env['stock.move'].create({
'name': 'Test Least Package',
'product_id': self.product.id,
'product_uom': self.product.uom_id.id,
'location_id': self.stock_location.id,
'location_dest_id': self.ref('stock.stock_location_customers'),
'product_uom_qty': 13,
})
move._action_confirm()
move._action_assign()
self.assertEqual(len(move.move_line_ids), 2)
self.assertRecordValues(
move.move_line_ids,
[{'reserved_qty': 10}] + [{'reserved_qty': 3}]
)
# Make sure it selects the smallest possible package as best leaf.
self.assertEqual(
move.move_line_ids[1].package_id.quant_ids.quantity,
5
)
def test_least_package_removal_strategy_not_enough(self):
"""
Tests the least package removal strategy in the case where not enough quantity
is available to fill the requested amount.
It should just return all the quantities in the domain.
"""
packages_data = [
(False, 2),
(5, 2),
(10, 5),
]
self._generate_data(packages_data)
move = self.env['stock.move'].create({
'name': 'Test Least Package',
'product_id': self.product.id,
'product_uom': self.product.uom_id.id,
'location_id': self.stock_location.id,
'location_dest_id': self.ref('stock.stock_location_customers'),
'product_uom_qty': 90,
})
move._action_confirm()
move._action_assign()
self.assertEqual(len(move.move_line_ids), 8)
self.assertRecordValues(
move.move_line_ids,
[{'reserved_qty': 2}] +
[{'reserved_qty': 10}] * 5 +
[{'reserved_qty': 5}] * 2
)
def test_quant_reserve(self):
""" Tests the reserve stock wizard which allows to choose a specific quant to reserve from
also checks if editing the reserved_uom_qty on stock.move.line updates the quants