The `distinct` keyword in query
```SELECT distinct res_id FROM mail_message WHERE model=%s```
is not strictly necessary for the correctness of the result, as
it will be inserted inside a sub-query that will be able to use
the main (model, res_id) index of mail_message.
For example, when the `has_message` field is used in a domain
by the website_livechat module[1], the complete query looks
like this:
```
SELECT "mail_channel".id FROM "mail_channel"
WHERE ("mail_channel"."active" = true) AND
("mail_channel"."livechat_visitor_id" = 112678387) AND
("mail_channel"."livechat_channel_id" = 1) AND
("mail_channel"."livechat_active" = true) AND
("mail_channel"."id" in (
SELECT distinct res_id FROM mail_message WHERE model='mail.channel'))
ORDER BY "mail_channel"."create_date" DESC LIMIT 1;
```
In this query, it's obvious that the `distinct` makes zero difference in
the results.
However, the presence of the DISTINCT keyword means that PostgreSQL will
factor the cost of applying that UNIQUE sort in the query planning, and
use MERGE JOIN strategy to avoid sorting multiple times (ok, it could
perhaps guess that distinct is useless here, but we asked for it.)
For a database with millions of mail messages, there can easily be
millions of hits for the generic "all res_ids for model" query, so
the cost of that part will be quite high.
The plan will then look like this (notice the "Unique" step and the cost,
730ms for the index scan + 200ms for the sort):
<details>
<summary>The query plan when `distinct` is used</summary>
```
QUERY PLAN
------------------------------------------------------------------------------
Limit (cost=151528.28..151528.29 rows=1 width=12) (actual time=1021.237..1021.239 rows=1 loops=1)
Output: mail_channel.id, mail_channel.create_date
Buffers: shared hit=767230
-> Sort (cost=151528.28..151528.29 rows=1 width=12) (actual time=998.799..998.801 rows=1 loops=1)
Output: mail_channel.id, mail_channel.create_date
Sort Key: mail_channel.create_date DESC
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=767230
-> Merge Join (cost=3.01..151528.27 rows=1 width=12) (actual time=998.786..998.790 rows=1 loops=1)
Output: mail_channel.id, mail_channel.create_date
Inner Unique: true
Merge Cond: (mail_channel.id = mail_message.res_id)
Buffers: shared hit=767230
-> Sort (cost=2.44..2.45 rows=1 width=12) (actual time=0.043..0.045 rows=1 loops=1)
Output: mail_channel.id, mail_channel.create_date
Sort Key: mail_channel.id
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=5
-> Index Scan using mail_channel_livechat_visitor_id_livechat_channel_id_idx on public.mail_channel (cost=0.41..2.43 rows=1 width=12) (actual time=0.035..0.039 rows=1 loops=1)
Output: mail_channel.id, mail_channel.create_date
Index Cond: ((mail_channel.livechat_visitor_id = 112678387) AND (mail_channel.livechat_channel_id = 1))
Filter: mail_channel.active
Buffers: shared hit=5
-> Unique (cost=0.57..143838.35 rows=614997 width=4) (actual time=0.033..993.210 rows=97187 loops=1)
Output: mail_message.res_id
Buffers: shared hit=767225
-> Index Only Scan using mail_message_model_res_id_idx on public.mail_message (cost=0.57..129884.36 rows=5581595 width=4) (actual time=0.032..730.233 rows=5586467 loops=1)
Output: mail_message.res_id
Index Cond: (mail_message.model = 'mail.channel'::text)
Heap Fetches: 17
Buffers: shared hit=767225
Planning Time: 0.471 ms
Execution Time: 1025.410 ms
(37 rows)
```
</details>
Now, if we remove the superfluous `distinct` clause, for the same
database, data volume, and result, the plan looks like this:
<details>
<summary>The query plan when `distinct` is not used</summary>
```
QUERY PLAN
-------------------------------------------------------------------------------------
Limit (cost=3.44..3.44 rows=1 width=12) (actual time=0.069..0.069 rows=0 loops=1)
Output: mail_channel.id, mail_channel.create_date
Buffers: shared hit=8
-> Sort (cost=3.44..3.44 rows=1 width=12) (actual time=0.068..0.068 rows=0 loops=1)
Output: mail_channel.id, mail_channel.create_date
Sort Key: mail_channel.create_date DESC
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=8
-> Nested Loop Semi Join (cost=0.98..3.43 rows=1 width=12) (actual time=0.061..0.061 rows=0 loops=1)
Output: mail_channel.id, mail_channel.create_date
Buffers: shared hit=8
-> Index Scan using mail_channel_livechat_visitor_id_livechat_channel_id_idx on public.mail_channel (cost=0.41..2.43 rows=1 width=12) (actual time=0.020..0.020 rows=1 loops=1)
Output: mail_channel.id, mail_channel.create_date
Index Cond: ((mail_channel.livechat_visitor_id = 117513256) AND (mail_channel.livechat_channel_id = 1))
Filter: mail_channel.active
Buffers: shared hit=4
-> Index Only Scan using mail_message_model_res_id_idx on public.mail_message (cost=0.57..2.77 rows=10 width=4) (actual time=0.040..0.040 rows=0 loops=1)
Output: mail_message.model, mail_message.res_id
Index Cond: ((mail_message.model = 'mail.channel'::text) AND (mail_message.res_id = mail_channel.id))
Heap Fetches: 0
Buffers: shared hit=4
Planning time: 0.761 ms
Execution time: 0.095 ms
(23 rows)
```
</details>
Total execution time goes from 1000 ms to 0.1ms.
Reference: introduced by 9a01a2953f,
coming from #69812, which was fixing a performance problem.
[1] https://github.com/odoo/odoo/blob/9b224f35f45876c86dc34934379bfa9e8b3e2160/addons/website_livechat/models/website.py#L40-L44closesodoo/odoo#78186
X-original-commit: 3ab87a848c96b284eac3dbbd6f6f2a9ea971876a
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Olivier Dony
63367b2519