Currently, the domain expression on a many2one field with
negative operator + string values leads to a extra SQL request on the
target model of the many2one which is not optimal and lead to
performance degradation.
Example:
--------
With a simple model A with a one2many (`b_id`) targeting model B (with
a field `b_name` and `_rec_name = b_name`):
The `A.search([('b_id', 'not like', 'OneName')])` will
generate two SQL requests:
- A _name_search on model B and returning ids who match
`('b_name', 'not like', 'OneName')`.
- A other on the model A with the application of the result
of the first one `('b_id', 'in', <returning ids> + [False])`.
Solution:
---------
Avoid the extra SQL request (the first one) and use subquery instead.
We do it by translate the `('b_id', 'in', <returning ids> + [False])`
expression into `['|', ('b_id', 'in', _subquery), ('b_id', '=', False)]`
task-2671476
Part-of: odoo/odoo#78948
This fixes a performance issue: ORDER BY clauses in subqueries can make
the query unexpectedly slow. We should avoid this situation, since
ORM-generated queries have an ORDER BY clause which is not relevant in
the context of a subquery.
The following example was found:
SELECT "pos_payment"."id" AS "id"
FROM "pos_payment"
WHERE ("pos_payment"."pos_order_id" in
(SELECT "pos_order".id
FROM "pos_order"
WHERE ("pos_order"."company_id" in (1))
ORDER BY "pos_order"."id"))
AND "pos_payment".id IN (1285508)
Here are the query plans made by PostgreSQL on this query with and
without the ORDER BY clause. The query time went from 1240ms to
0.402ms, which is 3000 times faster!
EXPLAIN ANALYZE SELECT "pos_payment"."id" as "id" FROM "pos_payment" WHERE ("pos_payment"."pos_order_id" in (SELECT "pos_order".id FROM "pos_order" WHERE ("pos_order"."company_id" in (1)) ORDER BY "pos_order"."id" )) AND "pos_payment".id IN (1285508);
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------
Merge Semi Join (cost=2.88..82726.85 rows=1 width=4) (actual time=1239.361..1239.364 rows=1 loops=1)
Merge Cond: (pos_payment.pos_order_id = pos_order.id)
-> Sort (cost=2.46..2.46 rows=1 width=8) (actual time=0.021..0.022 rows=1 loops=1)
Sort Key: pos_payment.pos_order_id
Sort Method: quicksort Memory: 25kB
-> Index Scan using pos_payment_pkey on pos_payment (cost=0.43..2.45 rows=1 width=8) (actual time=0.014..0.015 rows=1 loops=1)
Index Cond: (id = 1285508)
-> Index Scan using pos_order_pkey on pos_order (cost=0.43..66770.53 rows=1282120 width=4) (actual time=0.013..1148.194 rows=1182463 loops=1)
Filter: (company_id = 1)
Planning time: 0.272 ms
Execution time: 1239.396 ms
(11 rows)
EXPLAIN ANALYZE SELECT "pos_payment"."id" as "id" FROM "pos_payment" WHERE ("pos_payment"."pos_order_id" in (SELECT "pos_order".id FROM "pos_order" WHERE ("pos_order
"."company_id" in (1)) )) AND "pos_payment".id IN (1285508);
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.85..4.89 rows=1 width=4) (actual time=0.047..0.049 rows=1 loops=1)
-> Index Scan using pos_payment_pkey on pos_payment (cost=0.43..2.45 rows=1 width=8) (actual time=0.027..0.028 rows=1 loops=1)
Index Cond: (id = 1285508)
-> Index Scan using pos_order_pkey on pos_order (cost=0.43..2.45 rows=1 width=4) (actual time=0.018..0.018 rows=1 loops=1)
Index Cond: (id = pos_payment.pos_order_id)
Filter: (company_id = 1)
Planning time: 0.322 ms
Execution time: 0.080 ms
(8 rows)
closesodoo/odoo#77357
X-original-commit: 947c3bd72ee84c3a298e0c9c365340ddf16b307d
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Stanislas Sobieski (sts@odoo.com)
For searches, Odoo uses `unaccent` if it's available. On some
technical fields this is completely unnecessary and precludes the use
of indexes.
This PR provides:
* an opt-out (`unaccent = False`) on `String` and `Text` fields
* a warning if `unaccent` is enabled on *parent_path* fields as their
performance can be rather critical and not using the index is quite
an issue (note: the check that `parent_path` fields have been moved
outside of the check for their existence as we want to check that
the field is declared and correctly configured in all cases,
probably)
Task 2627454
Part-of: odoo/odoo#76436
`website` added its own sql-compatible version of
`get_unaccent_wrapper` in order to use `psycopg2.sql` for safety.
That is very sad.
Add a condition in the "standard" unaccent wrapper which checks
whether the input type is a `Composable`, and in that case return an
`SQL`. This increases the cost of the wrapper a tad but probably not
to a really sensible amount.
Task 2634851
Part-of: odoo/odoo#76436
Before this commit, the implementation crashes when the ancestors of a
record contain non-accessible records. This fixes the code to avoid it
to crash.
We also make the semantics of hierarchical searches more consistent in
this case: searching with operators 'child_of' and 'parent_of' returns
the subset of accessible records that satisfy the hierarchy operator.
We actually make it coincide with the results of the search when using
the "_parent_store" optimization.
closesodoo/odoo#73962
X-original-commit: 3e1b960acf3f6a726338476ba9e62e5ef5c84ef9
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Domain terms of the form `('field', 'not in', [...])` generate incorrect
queries for translated fields, example (model `res.country`, field `name`):
```
psycopg2.errors.SyntaxError: syntax error at or near "ARRAY"
LINE 1: ...M "res_country" WHERE "res_country"."name" not in ARRAY['No ...
```
The root cause is that the right part of the term is not converted to
tuple.
Observed during the upgrade request 16639
opw-2525553
closesodoo/odoo#73030
X-original-commit: 7c4db97e21f74a429e56f6cae4d4786ee74f7e22
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
When building a domain `(=|!=)` with
a Binary field stored in attachment and
the right part of the domain is not null,
the full binary content is logged, giving
the logs a huge size.
Now, the content of the binary is cropped to
20 chars as it is useless to log more.
opw-2527629
closesodoo/odoo#70740
X-original-commit: 0ed46ae94b20b30c10025b2a458e1ecf92c2712d
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
A domain with a negated thruty leaf raises an exception here
https://github.com/odoo/odoo/blob/835197e48fbbee1c8ac35d52287537fa6f0bdd36/odoo/osv/expression.py#L614
due to
https://github.com/odoo/odoo/blob/835197e48fbbee1c8ac35d52287537fa6f0bdd36/odoo/osv/expression.py#L670
and the fact that `FALSE_LEAF != (1, "!=", 1)` and `TRUE_LEAF != (0, "!=", 1)`
Current wrong behaviour:
```
>>> distribute_not(['!', TRUE_LEAF])
[(1, '!=', 1)]
>>> is_leaf((1, '!=', 1))
False
```
This issue was noticed when adapting domains for fields that have been
removed during migration. Example (extract of) traceback:
```
Traceback (most recent call last):
File "/home/odoo/src/odoo/13.0/odoo/http.py", line 624, in _handle_exception
return super(JsonRequest, self)._handle_exception(exception)
...
File "/home/odoo/src/odoo/13.0/odoo/osv/expression.py", line 677, in __init__
self.parse()
File "/home/odoo/src/odoo/13.0/odoo/osv/expression.py", line 814, in parse
self.stack = [ExtendedLeaf(leaf, self.root_model) for leaf in self.expression]
File "/home/odoo/src/odoo/13.0/odoo/osv/expression.py", line 814, in <listcomp>
self.stack = [ExtendedLeaf(leaf, self.root_model) for leaf in self.expression]
File "/home/odoo/src/odoo/13.0/odoo/osv/expression.py", line 562, in __init__
self.check_leaf(internal)
File "/home/odoo/src/odoo/13.0/odoo/osv/expression.py", line 618, in check_leaf
raise ValueError("Invalid leaf %s" % str(self.leaf))
ValueError: Invalid leaf (0, '!=', 1)
```
closesodoo/odoo#68665
X-original-commit: e20c23658d15a135e5f35b0e8dec6e3202cecaab
Signed-off-by: Christophe Simonis <chs@odoo.com>
Assume field B depends on A, and searching on B is rewritten as a domain
that mentions A. If A is modified, and then we search on B, one has to
flush A to the database before searching.
closesodoo/odoo#57963
X-original-commit: b242b7b1382ee39ebefa58c556931b7dae77f669
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
By adding a parameter `_table_query` on the Model, we can now have views
that depend on the context. The query is used instead of the table name
in ORM operations. This allows to pre-compute some values to improve
performance, instead of storing context values in the database.
Co-authored-by: william-andre <wan@odoo.com>
Co-authored-by: rco-odoo <rco@odoo.com>
Make the method `_search` return a `Query` object, and make that object
generate a subquery when used on the right-hand side of a condition.
closesodoo/odoo#52403
Related: odoo/enterprise#10945
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Introduce better methods for introducing tables, join clauses and where
clauses to the Query object, and other methods to generate a complete
SELECT query. The Query object can also behave as a tuple of ids, where
the query is made on demand, and the result is memoized.
This avoids the formatting part of those methods (`browse` in the case
of `search`, `display_name` in the case of `name_search`), which is
useless in most cases.
This also enables further optimizations where `_search` and
`_name_search` return subqueries instead of record ids.
Steps to reproduce:
- install helpdesk, hr_expense (the demo data has a clear example of the bug)
- go to helpdesk > tickets > set a custom filter > "activities is set"
Previous behavior:
the results show the ticket with id=2 as if it had an activity set.
Because the search sql query collides with other activities set on other models
(hr.expense.sheet in this example)
Current behavior:
the sql query returns the proper result
opw-2273680
closesodoo/odoo#55297
X-original-commit: 6e90c41413f6a726c08071c71d01c738a6538079
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Performance fix: when parsing domains using special operators
(`child_of`, ...), or with dotted field on the left value
(`['partner_id.name', ...]`), the ORM starts by a first call on
`search()` and builds a new domain so the final query has "IN (<ids
found by sub-search>)".
The sub-search issued on the comodel uses the default "order by".
The order of the ids is irrelevant here, we only need to know the ids.
A good example is when the sub-search happens on `product.product`,
which is ordered by `default_code, name, id`. The field `name` is
inherited and translated, so the ordering requires a JOIN on
`product_template` and a LEFT JOIN on `ir_translation`. Ordering by `id`
avoids these 2 JOINs.
Some discussion took place about modifying `_generate_order_by()` to
remove the `ORDER BY` clause when the `order` argument is False or ''.
Apart the fact that it would be a breaking change, @odony has shown that
sorting by `id` is so cheap that removing the `ORDER BY` might not
bring significant performance boost [0].
[0] https://github.com/odoo/odoo/pull/52368#issuecomment-646643773
opw-2270690
closesodoo/odoo#53629
X-original-commit: fc91f7b4e8f9f8a95862df1ce95a9639cf7d1363
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Replaces the auto_join scheme which is pretty fast but doesn't quite
work (it's *completely* broken for read_group) by a subquery which is
slower but has the same semantics as the ORM version.
Ideally we'd use `EXISTS (select 1 from ...)` but that has no support
inside the SQL compiler right now, so use `id in (select inverse_field
from ...)`.
Here are some measures of a `read_group` on a simple object with lines
filtered on line values, similar to
read_group([('order_line.product_uom', '=', xxx)],
fields=['total_amount'], groupby=['user_id'])
+-------+-------+-------+--------+
| | ORM | join |subquery|
+-------+-------+-------+--------+
| large | 160 | 10 | 20 |
+-------+-------+-------+--------+
| many | 36000 | 50 | 180 |
+-------+-------+-------+--------+
* values of data fields (filtering, grouping and aggregating) randomly
picked between 10 different values
* "large" is 1000 objects with 1000~15000 lines each
* "many" is 1000000 objects with 1~40 lines each
closesodoo/odoo#48494
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Instead of merging Query objects (main domain and access rules domains),
which requires renaming aliases in query strings, make `expression` push
its result in an existing Query object. This removes tricky code.
This also avoids using `IrRule.domain_get()`, which has become unsafe,
since it returns a list of tables (for the FROM clause) which does not
include the joins from the generated Query object.
closesodoo/odoo#49999
Related: odoo/enterprise#10126
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Use a LEFT JOIN on table `ir_translation` instead of a subquery, by
using the same implementation as the one used for ordering by a
translated field. In particular, this avoids having both the LEFT JOIN
and subquery when searching and sorting on a given translated field.
This fixes some cases where search results are actually missing. For
instance, searching with a domain like
['|', ('parent_id.name', '=', 'foo'), ('name', 'like', 'foo')]
where `parent_id` is auto-joined, misses results where `parent_id` is
NULL, because the actual query makes cartesian products for auto-joins:
SELECT "res_partner".id
FROM "res_partner", "res_partner" AS "res_partner__parent_id"
WHERE ("res_partner"."parent_id" = "res_partner__parent_id"."id")
AND ("res_partner__parent_id"."name" = 'foo' OR "res_partner"."name" == 'foo')
Using LEFT JOIN instead of cartesian product above, the query includes
all results where `parent_id` is NULL:
SELECT "res_partner".id
FROM "res_partner"
LEFT JOIN "res_partner" AS "res_partner__parent_id" ON
("res_partner"."parent_id" = "res_partner__parent_id"."id")
WHERE ("res_partner__parent_id"."name" = 'foo' OR "res_partner"."name" == 'foo')
The management of the join contexts now relies on the `Query` object,
that has all the necessary logic to do so. The refactoring also removes
the class `ExtendedLeaf`, which is no longer necessary: the join
contexts are all managed by a single `Query` object, and the parsing
stack now only contains triples like (leaf, model, table_alias).
Also inline directly the SQL translation made in `_to_sql`, which
overall simplifies the management of SQL parameters in the domain
compilation. The algorithm is illustrated in the code itself.
Before this commit, the query to fetch records with/without relations would be
done in a separate query which results would be then passed to the main query.
This commit adds this query as a subquery of the main query to do it in the
same SQL query instead of separately.
task-2211013
closesodoo/odoo#47729
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
Signed-off-by: Romain Derie <rdeodoo@users.noreply.github.com>
Consider an x2many field `foo_ids` with `context={'active_test': False}`
in its definition, and a comodel with an active field. The value of the
field includes inactive records.
Now consider a search with a domain like `[('foo_ids.bar', op, value)]`.
The search should return all the records with corecords that satisfy the
domain `[('bar', op, value)]`, including inactive corecords, because the
field's context explicitly disables filtering on the active field.
closesodoo/odoo#43625
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
We optimize the search on domains like `[('line_ids', 'in', ids)]`.
The condition is rewritten `('id', 'in', ids1)` where `ids1` is the
result of
SELECT <many2one_field> FROM <comodel_table> WHERE id IN <ids>
The issue is that the latter potentially returns many duplicate values.
The fix consists in having as few duplicates as possible in `ids1`.
Note that domains like `[('line_ids.foo', '=', 42)]` implicitly benefit
from the optimization, as they are rewritten as the one above with
ids = comodel.search([('foo', '=', 42)]).ids
closesodoo/odoo#43574
X-original-commit: a13c05fa5df9c98504e8e1cdb09d8d4a8d13ea91
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Ensure that expression.OR and expression.AND and some other
expression.py methods do not propagate or rely on TRUE_DOMAIN and
FALSE_DOMAIN that may be muted on some instance.
For example, if we did:
self.search(expression.OR([]))
then in the search method we do something like:
received_domain.append(('res_field', '=', False))
before this commit, FALSE_DOMAIN would be altered for any succeeding code
that try to use it in `[(0, '=', 1), ('res_field', '=', 'False')]`.
Without the changeset, the added test would fail with:
[(1, '=', 1), ('id', '=', 1)] != [(1, '=', 1)]
[(0, '=', 1), ('id', '=', 1)] != [(0, '=', 1)]
[(0, '=', 1), ('id', '=', 1)] != [(0, '=', 1)]
[(1, '=', 1), ('id', '=', 1)] != [(1, '=', 1)]
[(1, '=', 1), ('id', '=', 1)] != [(1, '=', 1)]
note: another commit referenced in #41968 should make the TRUE_DOMAIN
and FALSE_DOMAIN immutable.
related to work on opw-2154448
closes#42107closesodoo/odoo#42222
X-original-commit: 303ce32e6ccd7432b96ffae545e4a6eced68ae84
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
`is_false` relies on no small part on equality tests between the input
triplets and either `TRUE_LEAF` or `FALSE_LEAF`. While these are
defined as tuples, RPC domains will always be lists (as neither
XML-RPC nor JSON have tuples, and their arrays deserialize to Python
lists).
This is an issue, because tuple and list never compare equal. As a
result, while the in / not in predicates can succeed, the TRUE_LEAF /
FALSE_LEAF never will, and thus domains which contain either and might
shortcut (avoid a query entirely) will always go through the entire
process.
Fix by having domain normalization also ensure all triplets are
tuples: that's the first thing `is_false` does, it should never cause
issues and could fix / improve / shortcut other routines.
Note: Also implements TRUE_LEAF and FALSE_LEAF handling in the
SSF's modifiers evaluator. And fixes the ValueError to work
correctly if it breaks on a tuple / dict.
closesodoo/odoo#39706
X-original-commit: a44f008b918ee66742f7e943b00e4ee3d7fe2b78
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
en_US may not be activated as it is possible to create a database in
another language using the database manager.
When trying to install a chart of account, the tax return entry tried
to format a date at the installation of the module, with no lang in
the context. The fallback was made on en_US but an error is raised if
that language is not activated.
As it is a very common scenario to retrieve a language from the
context, add a generic tool method to do it.
Replace and closesodoo/odoo#37629closesodoo/odoo#37568
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
As odoo makes little difference, one would expect `(field, !=, 'foo')`
or `(field, 'not ilike', 'foo')` to have the same behaviour when field is
empty, and provide the opposite result as the "positive" operators.
However because NULL <OP> <value> returns NULL which is generally
assimilated to false, the sections above would fail to match records
whose field is null (they would match records whose field is *empty*
though).
Fix this by wrapping the operation in a coalesce falling back
to (hopefully) the correct value if the result is null.
See also: odoo/odoo#22879closesodoo/odoo#33959
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Combining a list of elements containing only the monoid's unit should return the
unit.
Since the unit is skipped, it would return an empty domain instead of the unit,
meaning that (0, '=', 1) could not be used in a group rule.
To test this, create a record rule on a model M with domain (0, '=', 1)
that applies to group G.
Log in with a user in group G and access model M; you can read everything
(provided no other record rule applies) whereas you should see nothing.
Because of expression.AND(global_domains + [expression.OR(group_domains)])
we want the empty domain to still be left untouched, even if it violates
the OR's unit rules (i.e. OR(empty, OR(R)) = OR(R) <=> OR(empty) = unit).
Meaning that no group rule would reduce to having a (0, '=' 1) group rule.
So in the end we don't really care about all that algebraic nonsense.
closesodoo/odoo#28765
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
Before this commit, doing expression.OR() with only FALSE_LEAF would
yield [] which is equivalent to TRUE_LEAF and is therefore not correct.
The same happened (to a lesser extent) with expression.AND() within an
expression.OR(), since the former would return a [] which would be
ignored by expression.OR().
See tests for a clearer view of the use cases.
Fixes#30113, #26540closesodoo/odoo#31202
From this commit onwards, Date fields will return datetime.date objects and Datetime fields will return datetime.datetime objects, this implies a number of things that are clearly explained both in the ORM API for master.
This commit also introduces a number of helper functions for dates and datetimes that are exposed in tools.date_utils and fields.Date[time], explained in the documentation as well.
Task-ID: 47189
Before this rev:
* Take 2 domains that contain at least two leaves, and at least one of
them must use the implicit `&` operator
e.g.:
d1 = [('so_line', 'in', [91]), ('amount', '<=', 0.0)]
d2 = ['&', ('so_line', 'in', []), ('project_id', '!=', False)]
* Perform osv.expression.OR() between both domains
Expected result:
d3 = ['|', d1, d2]
Actual result (after normalization):
d3 = ['&', '|', d1, d2]
This is because, since the `&` is implicit for the first domain, when we
OR it, we give it an explicit `|` operator, so when we pass this domain
through the normalize_domain function, d1 no longer contains an implicit
`&` operator but instead the implicit operator is the one between d1 and
d2, therefore giving us a completely wrong domain.
The `combine` function states that it only accepts normalized domains,
however neither the OR nor AND functions do, this leads to a lot of
developers putting non-normalized domains into these functions, and
there's no error checking or anything that obviously indicates that the
domain is incorrect, so we might as well normalize all domains being
passed since it's already pretty optimized.
This replaces the former modified preorder tree traversal (MPTT) with the
fields `parent_left`/`parent_right`. Each record is associated to a string
`parent_path`, that represents the path from its root node to itself. The path
is made of the node ids suffixed with a slash:
a node | id | parent_path
/ \ a | 42 | 42/
... b b | 63 | 42/63/
/ \ c | 84 | 42/63/84/
c d d | 85 | 42/63/85/
This field provides an efficient implementation for parent_of/child_of queries:
the nodes in the subtree of record are the ones where `parent_path` starts with
the `parent_path` of record. It is also more efficient to maintain than the
MPTT fields, and less sensitive to concurrent updates, because the value of
`parent_path` does not depend on sibling nodes.
Rémy Voet (ryv)