[FIX] mail: speed up filtering out blocked emails

For large lists of blocked emails, the time taken to load the records in
cache becomes prohibitive. For instance on a sample blocklist of 600k
entries, the `search([])` to load them could take 80-90s to run!
And this toll is taken every time the mail composer processes an email
batch in mass-mailing mode. Technically, most of the time is spent
iterating on the list of ids many time, in order to prefetch fields
into cache by determining what's missing.

Loading the contents of the list in raw SQL takes a fraction of that
time (400ms vs 90s).

Subsequently using a set for lookups in that blocklist is also much
faster (average time complexity O(1) vs O(n)).

Example: looking up an item in a 600k-entry blocklist is easily 5
orders of magnitude faster!

```
In [1]: blocklist = set(x[0] for x in self._cr.fetchall())

In [2]: len(blocklist)

Out[2]: 634610

In [3]: self._cr.execute("SELECT email from mail_blacklist")

In [4]: blocklist = set(x[0] for x in self._cr.fetchall())

In [5]: %timeit "hello@hello.com" in blocklist
The slowest run took 35.29 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 31.6 ns per loop

In [6]: self._cr.execute("SELECT email from mail_blacklist")

In [7]: blocklist = [x[0] for x in self._cr.fetchall()]

In [8]: %timeit "hello@hello.com" in blocklist
100 loops, best of 3: 2.24 ms per loop
```

closes odoo/odoo#57129

X-original-commit: 3b9a85754f30f0bef508d0b576ff679c87ead5da
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
This commit is contained in:
Olivier Dony
2020-09-05 04:34:13 +00:00
parent d6e734d436
commit 564fedb2dc
+6 -5
View File
@@ -280,15 +280,16 @@ class MailComposer(models.TransientModel):
records = self.env[self.model].browse(res_ids)
reply_to_value = records._notify_get_reply_to(default=self.email_from)
blacklisted_rec_ids = []
blacklisted_rec_ids = set()
if mass_mail_mode and issubclass(type(self.env[self.model]), self.pool['mail.thread.blacklist']):
BL_sudo = self.env['mail.blacklist'].sudo()
blacklist = set(BL_sudo.search([]).mapped('email'))
self.env['mail.blacklist'].flush(['email'])
self._cr.execute("SELECT email FROM mail_blacklist")
blacklist = {x[0] for x in self._cr.fetchall()}
if blacklist:
targets = self.env[self.model].browse(res_ids).read(['email_normalized'])
# First extract email from recipient before comparing with blacklist
blacklisted_rec_ids.extend([target['id'] for target in targets
if target['email_normalized'] and target['email_normalized'] in blacklist])
blacklisted_rec_ids.update(target['id'] for target in targets
if target['email_normalized'] in blacklist)
for res_id in res_ids:
# static wizard (mail.message) values