[MERGE] (test_)mail/mass_mailing(_sms): improve mailing trace status management

PURPOSE

Purpose of this merge is to provide some cleaning in mail application notably
targeting mass mailing and failure management.

SPECIFICATIONS: FAILURE TYPES

Currently there are differences between mail and sms error management
especially when sending them in batch (mass mode). Moreover cancel
(ignored) and error (exception) states meaning is not clear. Finally
some failure types management between mail and sms can be cleaned.

Meaning of ignored / error we want to enforce now is

  * error: there was something wrong at sending and user has an action to
    perform, i.e. server failed -> check its logs;
  * canceled: invalid recipients due to contact information or mailing
    configuration (blacklist, opt out, void or invalid email or phone number).
    In indicates issues linked to records themselves;

In this task we also add failure information granularity on mailing traces
linked to email like what is done currently on SMS. This can be related to
mailing (blacklist, optout, duplicates) or related to recipient (no recipient,
incorrectly formatted).

We also correctly distinguish optout from blacklist when sending SMS.

Detect and store failure types directly on mail.mail: blacklist, optout,
duplicates, missing email, wrong email, ...

State computation is moved from mass mailing to mail. It is also improved as
currently everything is based on "first recipient found". This is globally
valid for mass mailing who uses default recipients (customer). However when
dealing with generic mailing this is not true anymore.

We choose to store and update status on mail.mail records only when having
a single recipient. When several recipients are defined on a given mail.mail
we cannot really decide a status prior to sending it and skip this computation.

Mass mailing override now uses information from mail.mail to update its linked
traces as computation is now delegated to base composer model.

SPECIFICATIONS: TRACE STATUS MANAGEMENT

In this merge we clean the way trace state is managed. Currently it is a
computed field based on several datetime fields. However this generates a
lot of noise in the table as well as unnecessary computation

  * there are several columns (one for each state) storing datetime at
    which status was reached. Generally only 2 or 3 contain relevant
    information;
  * state could be set in code directly to avoid a computed field based on
    many triggers;
  * recomputing it each time a date changes is not necessarily necessary;
  * state value can always be updated manually as this is main done through
    some automated server update (mailgateway, link clicks, ...);

As trace states and its triggers should not be updated manually it is better
to synchronize it in code flow. When there is an exception or update done
through sending or gateway status is updated as well accordingly. Various
datetime fields are also updated at the same time. In order to align with
notification model mail and sms trace status are updated to a classic field.
Only last status update is now kept as there is no need to store the entire
history of status change.

We keep only a datetime for relevant metrics: open, reply and click. Other
datetime bring no real value. Knowing when a trace was in error or bounced
is not necessary. Indeed exception generally indicates a server issue (at
sending), cancel indicates a data issue (at sending) and bounce depends on
customer email server.

Status update is removed as using write_date is sufficient. Once created
traces are updated only when an external event occurs (opened, replied, ...).
It allows to simplify trace model.

We also rename ignored field into canceled to match naming use through mail
and sms.

VARIOUS

Improve code readability by renaming some failure type keys, notably those
used in mail application.

Improve user experience by correctly contextualizing trace display when coming
from a mailing.

Improve user experience of report model build on mailing traces.

See sub commits for more details.

LINKS

Task ID-2377974

closes odoo/odoo#61467

Related: odoo/upgrade#1907
Related: odoo/enterprise#14633
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>

addons/account/wizard/account_invoice_send.py

@@ -78,7 +78,7 @@ class AccountInvoiceSend(models.TransientModel):
             if wizard.composer_id:
                 wizard.composer_id.template_id = wizard.template_id.id
                 wizard._compute_composition_mode()
-                wizard.composer_id.onchange_template_id_wrapper()
+                wizard.composer_id._onchange_template_id_wrapper()
 
     @api.onchange('is_email')
     def onchange_is_email(self):
@@ -94,7 +94,7 @@ class AccountInvoiceSend(models.TransientModel):
                 self.composer_id.template_id = self.template_id.id
                 self._compute_composition_mode()
             if not self.env.context.get("wizard_opened"):
-                self.composer_id.onchange_template_id_wrapper()
+                self.composer_id._onchange_template_id_wrapper()
 
     @api.onchange('is_email')
     def _compute_invoice_without_email(self):
@@ -117,7 +117,7 @@ class AccountInvoiceSend(models.TransientModel):
     def _send_email(self):
         if self.is_email:
             # with_context : we don't want to reimport the file we just exported.
-            self.composer_id.with_context(no_new_invoice=True, mail_notify_author=self.env.user.partner_id in self.composer_id.partner_ids).send_mail()
+            self.composer_id.with_context(no_new_invoice=True, mail_notify_author=self.env.user.partner_id in self.composer_id.partner_ids)._action_send_mail()
             if self.env.context.get('mark_invoice_as_sent'):
                 #Salesman send posted invoice, without the right to write
                 #but they should have the right to change this flag
@@ -154,7 +154,7 @@ class AccountInvoiceSend(models.TransientModel):
 
     def save_as_template(self):
         self.ensure_one()
-        self.composer_id.save_as_template()
+        self.composer_id.action_save_as_template()
         self.template_id = self.composer_id.template_id.id
         action = _reopen(self, self.id, self.model, context=self._context)
         action.update({'name': _('Send Invoice')})

addons/coupon/models/mail_compose_message.py

@@ -7,9 +7,9 @@ from odoo import models
 class MailComposeMessage(models.TransientModel):
     _inherit = 'mail.compose.message'
 
-    def send_mail(self, **kwargs):
+    def _action_send_mail(self, **kwargs):
         for wizard in self:
             if self._context.get('mark_coupon_as_sent') and wizard.model == 'coupon.coupon' and wizard.partner_ids:
                 # Mark coupon as sent in sudo, as helpdesk users don't have the right to write on coupons
                 self.env[wizard.model].sudo().browse(wizard.res_id).state = 'sent'
-        return super().send_mail(**kwargs)
+        return super()._action_send_mail(**kwargs)

addons/crm/tests/test_crm_lead_convert_mass.py

@@ -24,7 +24,7 @@ class TestLeadConvertMass(crm_common.TestLeadConvertMassCommon):
         with self.assertQueryCount(user_sales_manager=0):
             test_leads = self.env['crm.lead'].browse(test_leads.ids)
 
-        with self.assertQueryCount(user_sales_manager=255):  # often 252, sometimes +3 on runbot
+        with self.assertQueryCount(user_sales_manager=254):  # crm only: 251 - often 251, sometimes +3 on runbot
             test_leads._handle_salesmen_assignment(user_ids=user_ids, team_id=False)
 
         self.assertEqual(test_leads.team_id, self.sales_team_convert | self.sales_team_1)
@@ -42,7 +42,7 @@ class TestLeadConvertMass(crm_common.TestLeadConvertMassCommon):
         with self.assertQueryCount(user_sales_manager=0):
             test_leads = self.env['crm.lead'].browse(test_leads.ids)
 
-        with self.assertQueryCount(user_sales_manager=223):  # crm only: 218 - generally 220, sometimes +2/+3 on runbot
+        with self.assertQueryCount(user_sales_manager=222):  # crm only: 217 - generally 219, sometimes +2/+3 on runbot
             test_leads._handle_salesmen_assignment(user_ids=user_ids, team_id=team_id)
 
         self.assertEqual(test_leads.team_id, self.sales_team_convert)
@@ -166,7 +166,7 @@ class TestLeadConvertMass(crm_common.TestLeadConvertMassCommon):
         test_leads = self._create_leads_batch(count=50, user_ids=[False])
         user_ids = self.assign_users.ids
 
-        with self.assertQueryCount(user_sales_manager=1369):  # still some randomness (1366 spotted) - crm only: ??
+        with self.assertQueryCount(user_sales_manager=1336):  # still some randomness (1266 generally on runbot) - crm only: 1257
             mass_convert = self.env['crm.lead2opportunity.partner.mass'].with_context({
                 'active_model': 'crm.lead',
                 'active_ids': test_leads.ids,

addons/google_calendar/tests/test_sync_odoo2google.py

@@ -102,7 +102,7 @@ class TestSyncOdoo2Google(TestSyncGoogle):
         })
         partner_model = self.env.ref('base.model_res_partner')
         partner = self.env['res.partner'].search([], limit=1)
-        with self.assertQueryCount(__system__=3637):
+        with self.assertQueryCount(__system__=3635):
             event = self.env['calendar.event'].create({
                 'name': "Event",
                 'start': datetime(2020, 1, 15, 8, 0),

addons/hr_work_entry_holidays/tests/test_performance.py

@@ -31,7 +31,7 @@ class TestWorkEntryHolidaysPerformance(TestWorkEntryHolidaysBase):
         self.richard_emp.generate_work_entries(date(2018, 1, 1), date(2018, 1, 2))
         leave = self.create_leave(datetime(2018, 1, 1, 7, 0), datetime(2018, 1, 1, 18, 0))
 
-        with self.assertQueryCount(__system__=96, admin=100):
+        with self.assertQueryCount(__system__=96, admin=97):
             leave.action_validate()
         leave.action_refuse()
 
@@ -40,7 +40,7 @@ class TestWorkEntryHolidaysPerformance(TestWorkEntryHolidaysBase):
     def test_performance_leave_write(self):
         leave = self.create_leave(datetime(2018, 1, 1, 7, 0), datetime(2018, 1, 1, 18, 0))
 
-        with self.assertQueryCount(__system__=24, admin=33):
+        with self.assertQueryCount(__system__=21, admin=29):
             leave.date_to = datetime(2018, 1, 1, 19, 0)
         leave.action_refuse()
 

addons/mail/models/mail_mail.py

@@ -35,7 +35,7 @@ class MailMail(models.Model):
     headers = fields.Text('Headers', copy=False)
     # Auto-detected based on create() - if 'mail_message_id' was passed then this mail is a notification
     # and during unlink() we will not cascade delete the parent and its attachments
-    notification = fields.Boolean('Is Notification', help='Mail has been created to notify people of an existing mail.message')
+    is_notification = fields.Boolean('Notification Email', help='Mail has been created to notify people of an existing mail.message')
     # recipients: include inactive partners (they may have been archived after
     # the message was sent, but they should remain visible in the relation)
     email_to = fields.Text('To', help='Message recipients (emails)')
@@ -50,12 +50,24 @@ class MailMail(models.Model):
         ('exception', 'Delivery Failed'),
         ('cancel', 'Cancelled'),
     ], 'Status', readonly=True, copy=False, default='outgoing')
-    auto_delete = fields.Boolean(
-        'Auto Delete',
-        help="This option permanently removes any track of email after it's been sent, including from the Technical menu in the Settings, in order to preserve storage space of your Odoo database.")
+    failure_type = fields.Selection(selection=[
+        # generic
+        ("unknown", "Unknown error"),
+        # mail
+        ("mail_email_invalid", "Invalid email address"),
+        ("mail_email_missing", "Missing email"),
+        ("mail_smtp", "Connection failed (outgoing mail server problem)"),
+        # mass mode
+        ("mail_bl", "Blacklisted Address"),
+        ("mail_optout", "Opted Out"),
+        ("mail_dup", "Duplicated Email"),
+        ], string='Failure type')
     failure_reason = fields.Text(
         'Failure Reason', readonly=1, copy=False,
         help="Failure reason. This is usually the exception thrown by the email server, stored to ease the debugging of mailing issues.")
+    auto_delete = fields.Boolean(
+        'Auto Delete',
+        help="This option permanently removes any track of email after it's been sent, including from the Technical menu in the Settings, in order to preserve storage space of your Odoo database.")
     scheduled_date = fields.Char('Scheduled Send Date',
         help="If set, the queue manager will send the email after the date. If not set, the email will be send as soon as possible.")
 
@@ -63,8 +75,8 @@ class MailMail(models.Model):
     def create(self, values_list):
         # notification field: if not set, set if mail comes from an existing mail.message
         for values in values_list:
-            if 'notification' not in values and values.get('mail_message_id'):
-                values['notification'] = True
+            if 'is_notification' not in values and values.get('mail_message_id'):
+                values['is_notification'] = True
 
         new_mails = super(MailMail, self).create(values_list)
 
@@ -86,7 +98,7 @@ class MailMail(models.Model):
 
     def unlink(self):
         # cascade-delete the parent message for all mails that are not created for a notification
-        mail_msg_cascade_ids = [mail.mail_message_id.id for mail in self if not mail.notification]
+        mail_msg_cascade_ids = [mail.mail_message_id.id for mail in self if not mail.is_notification]
         res = super(MailMail, self).unlink()
         if mail_msg_cascade_ids:
             self.env['mail.message'].browse(mail_msg_cascade_ids).unlink()
@@ -156,11 +168,11 @@ class MailMail(models.Model):
 
         :return: True
         """
-        notif_mails_ids = [mail.id for mail in self if mail.notification]
+        notif_mails_ids = [mail.id for mail in self if mail.is_notification]
         if notif_mails_ids:
             notifications = self.env['mail.notification'].search([
                 ('notification_type', '=', 'email'),
-                ('mail_id', 'in', notif_mails_ids),
+                ('mail_mail_id', 'in', notif_mails_ids),
                 ('notification_status', 'not in', ('sent', 'canceled'))
             ])
             if notifications:
@@ -182,7 +194,7 @@ class MailMail(models.Model):
                     messages = notifications.mapped('mail_message_id').filtered(lambda m: m.is_thread_message())
                     # TDE TODO: could be great to notify message-based, not notifications-based, to lessen number of notifs
                     messages._notify_message_notification_update()  # notify user that we have a failure
-        if not failure_type or failure_type == 'RECIPIENT':  # if we have another error, we want to keep the mail.
+        if not failure_type or failure_type in ['mail_email_invalid', 'mail_email_missing']:  # if we have another error, we want to keep the mail.
             mail_to_delete_ids = [mail.id for mail in self if mail.auto_delete]
             self.browse(mail_to_delete_ids).sudo().unlink()
         return True
@@ -286,7 +298,7 @@ class MailMail(models.Model):
                 else:
                     batch = self.browse(batch_ids)
                     batch.write({'state': 'exception', 'failure_reason': exc})
-                    batch._postprocess_sent_message(success_pids=[], failure_type="SMTP")
+                    batch._postprocess_sent_message(success_pids=[], failure_type="mail_smtp")
             else:
                 self.browse(batch_ids)._send(
                     auto_commit=auto_commit,
@@ -360,14 +372,14 @@ class MailMail(models.Model):
                 # mail record.
                 notifs = self.env['mail.notification'].search([
                     ('notification_type', '=', 'email'),
-                    ('mail_id', 'in', mail.ids),
+                    ('mail_mail_id', 'in', mail.ids),
                     ('notification_status', 'not in', ('sent', 'canceled'))
                 ])
                 if notifs:
                     notif_msg = _('Error without exception. Probably due do concurrent access update of notification records. Please see with an administrator.')
                     notifs.sudo().write({
                         'notification_status': 'exception',
-                        'failure_type': 'UNKNOWN',
+                        'failure_type': 'unknown',
                         'failure_reason': notif_msg,
                     })
                     # `test_mail_bounce_during_send`, force immediate update to obtain the lock.
@@ -376,6 +388,8 @@ class MailMail(models.Model):
 
                 # build an RFC2822 email.message.Message object and send it without queuing
                 res = None
+                # TDE note: could be great to pre-detect missing to/cc and skip sending it
+                # to go directly to failed state update
                 for email in email_list:
                     msg = IrMailServer.build_email(
                         email_from=mail.email_from,
@@ -401,7 +415,11 @@ class MailMail(models.Model):
                         processing_pid = None
                     except AssertionError as error:
                         if str(error) == IrMailServer.NO_VALID_RECIPIENT:
-                            failure_type = "RECIPIENT"
+                            # if we have a list of void emails for email_list -> email missing, otherwise generic email failure
+                            if not email.get('email_to') and failure_type != "RECIPIENT":
+                                failure_type = "mail_email_missing"
+                            else:
+                                failure_type = "mail_email_invalid"
                             # No valid recipient found for this particular
                             # mail item -> ignore error to avoid blocking
                             # delivery to next recipients, if any. If this is
@@ -435,7 +453,7 @@ class MailMail(models.Model):
                 failure_reason = tools.ustr(e)
                 _logger.exception('failed sending mail (id: %s) due to %s', mail.id, failure_reason)
                 mail.write({'state': 'exception', 'failure_reason': failure_reason})
-                mail._postprocess_sent_message(success_pids=success_pids, failure_reason=failure_reason, failure_type='UNKNOWN')
+                mail._postprocess_sent_message(success_pids=success_pids, failure_reason=failure_reason, failure_type='unknown')
                 if raise_exception:
                     if isinstance(e, (AssertionError, UnicodeEncodeError)):
                         if isinstance(e, UnicodeEncodeError):

addons/mail/models/mail_message.py

@@ -27,37 +27,39 @@ class Message(models.Model):
       * notification_status
         'ready', 'sent', 'bounce', 'exception', 'canceled'
       * notification_type
-            'inbox', 'email', 'sms' (SMS addon), 'snail' (snailmail addon)
+        'inbox', 'email', 'sms' (SMS addon), 'snail' (snailmail addon)
       * failure_type
-            # mail
-            "SMTP", "RECIPIENT", "BOUNCE", "UNKNOWN"
-            # sms (SMS addon)
-            'sms_number_missing', 'sms_number_format', 'sms_credit',
-            'sms_server', 'sms_acc'
-            # snailmail (snailmail addon)
-            'sn_credit', 'sn_trial', 'sn_price', 'sn_fields',
-            'sn_format', 'sn_error'
+        # generic
+        unknown,
+        # mail
+        "mail_email_invalid", "mail_smtp", "mail_email_missing"
+        # sms (SMS addon)
+        'sms_number_missing', 'sms_number_format', 'sms_credit',
+        'sms_server', 'sms_acc'
+        # snailmail (snailmail addon)
+        'sn_credit', 'sn_trial', 'sn_price', 'sn_fields',
+        'sn_format', 'sn_error'
 
     * mail.mail
       * state
-            'outgoing', 'sent', 'received', 'exception', 'cancel'
+        'outgoing', 'sent', 'received', 'exception', 'cancel'
       * failure_reason: text
 
     * sms.sms (SMS addon)
       * state
-            'outgoing', 'sent', 'error', 'canceled'
+        'outgoing', 'sent', 'error', 'canceled'
       * error_code
-            'sms_number_missing', 'sms_number_format', 'sms_credit',
-            'sms_server', 'sms_acc',
-            # mass mode specific codes
-            'sms_blacklist', 'sms_duplicate'
+        'sms_number_missing', 'sms_number_format', 'sms_credit',
+        'sms_server', 'sms_acc',
+        # mass mode specific codes
+        'sms_blacklist', 'sms_duplicate'
 
     * snailmail.letter (snailmail addon)
       * state
-            'pending', 'sent', 'error', 'canceled'
+        'pending', 'sent', 'error', 'canceled'
       * error_code
-            'CREDIT_ERROR', 'TRIAL_ERROR', 'NO_PRICE_AVAILABLE', 'FORMAT_ERROR',
-            'UNKNOWN_ERROR',
+        'CREDIT_ERROR', 'TRIAL_ERROR', 'NO_PRICE_AVAILABLE', 'FORMAT_ERROR',
+        'UNKNOWN_ERROR',
 
     See ``mailing.trace`` model in mass_mailing application for mailing trace
     information.

addons/mail/models/mail_notification.py

@@ -17,7 +17,7 @@ class MailNotification(models.Model):
 
     # origin
     mail_message_id = fields.Many2one('mail.message', 'Message', index=True, ondelete='cascade', required=True)
-    mail_id = fields.Many2one('mail.mail', 'Mail', index=True, help='Optional mail_mail ID. Used mainly to optimize searches.')
+    mail_mail_id = fields.Many2one('mail.mail', 'Mail', index=True, help='Optional mail_mail ID. Used mainly to optimize searches.')
     # recipient
     res_partner_id = fields.Many2one('res.partner', 'Recipient', index=True, ondelete='cascade')
     # status
@@ -34,10 +34,12 @@ class MailNotification(models.Model):
     is_read = fields.Boolean('Is Read', index=True)
     read_date = fields.Datetime('Read Date', copy=False)
     failure_type = fields.Selection(selection=[
-        ("SMTP", "Connection failed (outgoing mail server problem)"),
-        ("RECIPIENT", "Invalid email address"),
-        ("BOUNCE", "Email address rejected by destination"),
-        ("UNKNOWN", "Unknown error"),
+        # generic
+        ("unknown", "Unknown error"),
+        # mail
+        ("mail_email_invalid", "Invalid email address"),
+        ("mail_email_missing", "Missing email addresss"),
+        ("mail_smtp", "Connection failed (outgoing mail server problem)"),
         ], string='Failure type')
     failure_reason = fields.Text('Failure reason', copy=False)
 
@@ -73,7 +75,7 @@ class MailNotification(models.Model):
 
     def format_failure_reason(self):
         self.ensure_one()
-        if self.failure_type != 'UNKNOWN':
+        if self.failure_type != 'unknown':
             return dict(type(self).failure_type.selection).get(self.failure_type, _('No Error'))
         else:
             return _("Unknown error") + ": %s" % (self.failure_reason or '')

addons/mail/models/mail_thread.py

@@ -1942,9 +1942,9 @@ class MailThread(models.AbstractModel):
         # Simulate the onchange (like trigger in form the view) only
         # when having a template in single-email mode
         if template_id:
-            update_values = composer.onchange_template_id(template_id, kwargs['composition_mode'], self._name, res_id)['value']
+            update_values = composer._onchange_template_id(template_id, kwargs['composition_mode'], self._name, res_id)['value']
             composer.write(update_values)
-        return composer.send_mail(auto_commit=auto_commit)
+        return composer._action_send_mail(auto_commit=auto_commit)
 
     def message_notify(self, *,
                        partner_ids=False, parent_id=False, model=False, res_id=False,
@@ -2254,13 +2254,13 @@ class MailThread(models.AbstractModel):
                             tocreate_recipient_ids = [rid for rid in recipient_ids if rid not in existing_notifications.mapped('res_partner_id.id')]
                             existing_notifications.write({
                                 'notification_status': 'ready',
-                                'mail_id': email.id,
+                                'mail_mail_id': email.id,
                             })
                     notif_create_values += [{
                         'mail_message_id': message.id,
                         'res_partner_id': recipient_id,
                         'notification_type': 'email',
-                        'mail_id': email.id,
+                        'mail_mail_id': email.id,
                         'is_read': True,  # discard Inbox notification
                         'notification_status': 'ready',
                     } for recipient_id in tocreate_recipient_ids]

addons/mail/tests/__init__.py

@@ -5,5 +5,6 @@ from . import test_mail_channel
 from . import test_mail_channel_partner
 from . import test_mail_full_composer
 from . import test_mail_render
+from . import test_mail_tools
 from . import test_res_partner
 from . import test_update_notification

addons/mail/tests/test_mail_tools.py

@@ -0,0 +1,193 @@
+# -*- coding: utf-8 -*-
+# Part of Odoo. See LICENSE file for full copyright and licensing details.
+
+from odoo.addons.mail.tests.common import MailCommon
+from odoo.tests import tagged, users
+from odoo import tools
+
+
+@tagged('mail_tools')
+class TestMailTools(MailCommon):
+
+    @classmethod
+    def setUpClass(cls):
+        super(TestMailTools, cls).setUpClass()
+
+        cls._test_email = 'alfredoastaire@test.example.com'
+        cls.test_partner = cls.env['res.partner'].create({
+            'country_id': cls.env.ref('base.be').id,
+            'email': cls._test_email,
+            'mobile': '0456001122',
+            'name': 'Alfred Astaire',
+            'phone': '0456334455',
+        })
+
+        cls.sources = [
+            # single email
+            'alfred.astaire@test.example.com',
+            ' alfred.astaire@test.example.com ',
+            'Fredo The Great <alfred.astaire@test.example.com>',
+            '"Fredo The Great" <alfred.astaire@test.example.com>',
+            'Fredo "The Great" <alfred.astaire@test.example.com>',
+            # multiple emails
+            'alfred.astaire@test.example.com, evelyne.gargouillis@test.example.com',
+            'Fredo The Great <alfred.astaire@test.example.com>, Evelyne The Goat <evelyne.gargouillis@test.example.com>',
+            '"Fredo The Great" <alfred.astaire@test.example.com>, evelyne.gargouillis@test.example.com',
+            '"Fredo The Great" <alfred.astaire@test.example.com>, <evelyne.gargouillis@test.example.com>',
+            # text containing email
+            'Hello alfred.astaire@test.example.com how are you ?',
+            '<p>Hello alfred.astaire@test.example.com</p>',
+            # text containing emails
+            'Hello "Fredo" <alfred.astaire@test.example.com>, evelyne.gargouillis@test.example.com',
+            'Hello "Fredo" <alfred.astaire@test.example.com> and evelyne.gargouillis@test.example.com',
+            # falsy
+            '<p>Hello Fredo</p>',
+            'j\'adore écrire des @gmail.com ou "@gmail.com" a bit randomly',
+            '',
+        ]
+
+    @users('employee')
+    def test_find_partner_from_emails(self):
+        Partner = self.env['res.partner']
+        test_partner = Partner.browse(self.test_partner.ids)
+        self.assertEqual(test_partner.email, self._test_email)
+
+        # test direct match
+        found = Partner._mail_find_partner_from_emails([self._test_email])
+        self.assertEqual(found, [test_partner])
+
+        # test encapsulated email
+        found = Partner._mail_find_partner_from_emails(['"Norbert Poiluchette" <%s>' % self._test_email])
+        self.assertEqual(found, [test_partner])
+
+        # test with wildcard "_"
+        found = Partner._mail_find_partner_from_emails(['alfred_astaire@test.example.com'])
+        self.assertEqual(found, [self.env['res.partner']])
+
+        # sub-check: this search does not consider _ as a wildcard
+        found = Partner._mail_search_on_partner(['alfred_astaire@test.example.com'])
+        self.assertEqual(found, self.env['res.partner'])
+
+        # test partners with encapsulated emails
+        # ------------------------------------------------------------
+        test_partner.sudo().write({'email': '"Alfred Mighty Power Astaire" <%s>' % self._test_email})
+
+        # test direct match
+        found = Partner._mail_find_partner_from_emails([self._test_email])
+        self.assertEqual(found, [test_partner])
+
+        # test encapsulated email
+        found = Partner._mail_find_partner_from_emails(['"Norbert Poiluchette" <%s>' % self._test_email])
+        self.assertEqual(found, [test_partner])
+
+        # test with wildcard "_"
+        found = Partner._mail_find_partner_from_emails(['alfred_astaire@test.example.com'])
+        self.assertEqual(found, [self.env['res.partner']])
+
+        # sub-check: this search does not consider _ as a wildcard
+        found = Partner._mail_search_on_partner(['alfred_astaire@test.example.com'])
+        self.assertEqual(found, self.env['res.partner'])
+
+        # test partners with look-alike emails
+        # ------------------------------------------------------------
+        for email_lookalike in [
+                'alfred.astaire@test.example.com',
+                'alfredoastaire@example.com',
+                'aalfredoastaire@test.example.com',
+                'alfredoastaire@test.example.comm']:
+            test_partner.sudo().write({'email': '"Alfred Astaire" <%s>' % email_lookalike})
+
+            # test direct match
+            found = Partner._mail_find_partner_from_emails([self._test_email])
+            self.assertEqual(found, [self.env['res.partner']])
+            # test encapsulated email
+            found = Partner._mail_find_partner_from_emails(['"Norbert Poiluchette" <%s>' % self._test_email])
+            self.assertEqual(found, [self.env['res.partner']])
+            # test with wildcard "_"
+            found = Partner._mail_find_partner_from_emails(['alfred_astaire@test.example.com'])
+            self.assertEqual(found, [self.env['res.partner']])
+
+    @users('employee')
+    def test_tools_email_re(self):
+        expected = [
+            # single email
+            ['alfred.astaire@test.example.com'],
+            ['alfred.astaire@test.example.com'],
+            ['alfred.astaire@test.example.com'],
+            ['alfred.astaire@test.example.com'],
+            ['alfred.astaire@test.example.com'],
+            # multiple emails
+            ['alfred.astaire@test.example.com', 'evelyne.gargouillis@test.example.com'],
+            ['alfred.astaire@test.example.com', 'evelyne.gargouillis@test.example.com'],
+            ['alfred.astaire@test.example.com', 'evelyne.gargouillis@test.example.com'],
+            ['alfred.astaire@test.example.com', 'evelyne.gargouillis@test.example.com'],
+            # text containing email
+            ['alfred.astaire@test.example.com'],
+            ['alfred.astaire@test.example.com'],
+            # text containing emails
+            ['alfred.astaire@test.example.com', 'evelyne.gargouillis@test.example.com'],
+            ['alfred.astaire@test.example.com', 'evelyne.gargouillis@test.example.com'],
+            # falsy
+            [], [], [],
+        ]
+
+        for src, exp in zip(self.sources, expected):
+            res = tools.email_re.findall(src)
+            self.assertEqual(
+                res, exp,
+                'Seems email_re is broken with %s (expected %r, received %r)' % (src, exp, res)
+            )
+
+    @users('employee')
+    def test_tools_email_split_tuples(self):
+        expected = [
+            # single email
+            [('', 'alfred.astaire@test.example.com')],
+            [('', 'alfred.astaire@test.example.com')],
+            [('Fredo The Great', 'alfred.astaire@test.example.com')],
+            [('Fredo The Great', 'alfred.astaire@test.example.com')],
+            [('Fredo The Great', 'alfred.astaire@test.example.com')],
+            # multiple emails
+            [('', 'alfred.astaire@test.example.com'), ('', 'evelyne.gargouillis@test.example.com')],
+            [('Fredo The Great', 'alfred.astaire@test.example.com'), ('Evelyne The Goat', 'evelyne.gargouillis@test.example.com')],
+            [('Fredo The Great', 'alfred.astaire@test.example.com'), ('', 'evelyne.gargouillis@test.example.com')],
+            [('Fredo The Great', 'alfred.astaire@test.example.com'), ('', 'evelyne.gargouillis@test.example.com')],
+            # text containing email -> probably not designed for that
+            [('', 'Hello alfred.astaire@test.example.comhowareyou?')],
+            [('', 'Hello alfred.astaire@test.example.com')],
+            # text containing emails -> probably not designed for that
+            [('Hello Fredo', 'alfred.astaire@test.example.com'), ('', 'evelyne.gargouillis@test.example.com')],
+            [('Hello Fredo', 'alfred.astaire@test.example.com'), ('', 'and evelyne.gargouillis@test.example.com')],
+            # falsy -> probably not designed for that
+            [], [('', "j'adore écrire des@gmail.comou"), ('', '@gmail.com')], [],
+        ]
+
+        for src, exp in zip(self.sources, expected):
+            res = tools.email_split_tuples(src)
+            self.assertEqual(
+                res, exp,
+                'Seems email_split_tuples is broken with %s (expected %r, received %r)' % (src, exp, res)
+            )
+
+    @users('employee')
+    def test_tools_single_email_re(self):
+        expected = [
+            # single email
+            ['alfred.astaire@test.example.com'],
+            [], [], [], [], # formatting issue for single email re
+            # multiple emails -> couic
+            [], [], [], [],
+            # text containing email -> couic
+            [], [],
+            # text containing emails -> couic
+            [], [],
+            # falsy
+            [], [], [],
+        ]
+
+        for src, exp in zip(self.sources, expected):
+            res = tools.single_email_re.findall(src)
+            self.assertEqual(
+                res, exp,
+                'Seems single_email_re is broken with %s (expected %r, received %r)' % (src, exp, res)
+            )

addons/mail/views/mail_mail_views.xml

@@ -37,7 +37,7 @@
                                 <group>
                                     <group string="Status">
                                         <field name="auto_delete"/>
-                                        <field name="notification"/>
+                                        <field name="is_notification"/>
                                         <field name="message_type"/>
                                         <field name="mail_server_id"/>
                                         <field name="model"/>

addons/mail/views/mail_notification_views.xml

@@ -27,7 +27,7 @@
                         <group string="Source">
                             <field name="mail_message_id"/>
                             <field name="notification_type"/>
-                            <field name="mail_id"/>
+                            <field name="mail_mail_id"/>
                             <field name="res_partner_id"/>
                         </group>
                         <group string="Status">

addons/mail/wizard/mail_compose_message.py

@@ -7,10 +7,7 @@ import re
 
 from odoo import _, api, fields, models, tools, Command
 from odoo.exceptions import UserError
-
-
-# main mako-like expression pattern
-EXPRESSION_PATTERN = re.compile('(\$\{.+?\})')
+from odoo.tools import email_re
 
 
 def _reopen(self, res_id, model, context=None):
@@ -161,6 +158,15 @@ class MailComposer(models.TransientModel):
         for composer in self:
             composer.render_model = composer.model
 
+    # Onchanges
+
+    @api.onchange('template_id')
+    def _onchange_template_id_wrapper(self):
+        self.ensure_one()
+        values = self._onchange_template_id(self.template_id.id, self.composition_mode, self.model, self.res_id)['value']
+        for fname, value in values.items():
+            setattr(self, fname, value)
+
     @api.model
     def get_record_data(self, values):
         """ Returns a defaults-like dict with initial values for the composition
@@ -190,16 +196,37 @@ class MailComposer(models.TransientModel):
 
         return result
 
+    # ------------------------------------------------------------
+    # CRUD / ORM
+    # ------------------------------------------------------------
+
+    @api.autovacuum
+    def _gc_lost_attachments(self):
+        """ Garbage collect lost mail attachments. Those are attachments
+            - linked to res_model 'mail.compose.message', the composer wizard
+            - with res_id 0, because they were created outside of an existing
+                wizard (typically user input through Chatter or reports
+                created on-the-fly by the templates)
+            - unused since at least one day (create_date and write_date)
+        """
+        limit_date = fields.Datetime.subtract(fields.Datetime.now(), days=1)
+        self.env['ir.attachment'].search([
+            ('res_model', '=', self._name),
+            ('res_id', '=', 0),
+            ('create_date', '<', limit_date),
+            ('write_date', '<', limit_date)]
+        ).unlink()
+
     # ------------------------------------------------------------
     # ACTIONS
     # ------------------------------------------------------------
-    # action buttons call with positionnal arguments only, so we need an intermediary function
-    # to ensure the context is passed correctly
+
     def action_send_mail(self):
-        self.send_mail()
+        """ Used for action button that do not accept arguments. """
+        self._action_send_mail(auto_commit=False)
         return {'type': 'ir.actions.act_window_close'}
 
-    def send_mail(self, auto_commit=False):
+    def _action_send_mail(self, auto_commit=False):
         """ Process the wizard content and proceed with sending the related
             email(s), rendering any template patterns on the fly if needed. """
         notif_layout = self._context.get('custom_layout')
@@ -283,6 +310,30 @@ class MailComposer(models.TransientModel):
                 if wizard.composition_mode == 'mass_mail':
                     batch_mails_sudo.send(auto_commit=auto_commit)
 
+    def action_save_as_template(self):
+        """ hit save as template button: current form value will be a new
+            template attached to the current document. """
+        for record in self:
+            model = self.env['ir.model']._get(record.model or 'mail.message')
+            model_name = model.name or ''
+            template_name = "%s: %s" % (model_name, tools.ustr(record.subject))
+            values = {
+                'name': template_name,
+                'subject': record.subject or False,
+                'body_html': record.body or False,
+                'model_id': model.id or False,
+                'attachment_ids': [Command.set(record.attachment_ids.ids)],
+            }
+            template = self.env['mail.template'].create(values)
+            # generate the saved template
+            record.write({'template_id': template.id})
+            record._onchange_template_id_wrapper()
+            return _reopen(self, record.id, record.model, context=self._context)
+
+    # ------------------------------------------------------------
+    # RENDERING / VALUES GENERATION
+    # ------------------------------------------------------------
+
     def get_mail_values(self, res_ids):
         """Generate the values that will be used by send_mail to create mail_messages
         or mail_mails. """
@@ -300,17 +351,6 @@ class MailComposer(models.TransientModel):
             records = self.env[self.model].browse(res_ids)
             reply_to_value = records._notify_get_reply_to(default=self.email_from)
 
-        blacklisted_rec_ids = set()
-        if mass_mail_mode and issubclass(type(self.env[self.model]), self.pool['mail.thread.blacklist']):
-            self.env['mail.blacklist'].flush(['email'])
-            self._cr.execute("SELECT email FROM mail_blacklist WHERE active=true")
-            blacklist = {x[0] for x in self._cr.fetchall()}
-            if blacklist:
-                targets = self.env[self.model].browse(res_ids).read(['email_normalized'])
-                # First extract email from recipient before comparing with blacklist
-                blacklisted_rec_ids.update(target['id'] for target in targets
-                                           if target['email_normalized'] in blacklist)
-
         for res_id in res_ids:
             # static wizard (mail.message) values
             mail_values = {
@@ -332,7 +372,7 @@ class MailComposer(models.TransientModel):
                 record = self.env[self.model].browse(res_id)
                 mail_values['headers'] = record._notify_email_headers()
                 # keep a copy unless specifically requested, reset record name (avoid browsing records)
-                mail_values.update(notification=not self.auto_delete_message, model=self.model, res_id=res_id, record_name=False)
+                mail_values.update(is_notification=not self.auto_delete_message, model=self.model, res_id=res_id, record_name=False)
                 # auto deletion of mail_mail
                 if self.auto_delete or self.template_id.auto_delete:
                     mail_values['auto_delete'] = True
@@ -362,27 +402,114 @@ class MailComposer(models.TransientModel):
                     attachment_ids,
                     {'model': 'mail.message', 'res_id': 0}
                 )['attachment_ids']
-                # Filter out the blacklisted records by setting the mail state to cancel -> Used for Mass Mailing stats
-                if res_id in blacklisted_rec_ids:
-                    mail_values['state'] = 'cancel'
-                    # Do not post the mail into the recipient's chatter
-                    mail_values['notification'] = False
 
             results[res_id] = mail_values
+
+        results = self._process_state(results)
         return results
 
-    # ------------------------------------------------------------
-    # TEMPLATES
-    # ------------------------------------------------------------
+    def _process_recipient_values(self, mail_values_dict):
+        # Preprocess res.partners to batch-fetch from db if recipient_ids is present
+        # it means they are partners (the only object to fill get_default_recipient this way)
+        recipient_pids = [
+            recipient_command[1]
+            for mail_values in mail_values_dict.values()
+            # recipient_ids is a list of x2m command tuples at this point
+            for recipient_command in mail_values.get('recipient_ids') or []
+            if recipient_command[1]
+        ]
+        recipient_emails = {
+            p.id: p.email
+            for p in self.env['res.partner'].browse(set(recipient_pids))
+        } if recipient_pids else {}
 
-    @api.onchange('template_id')
-    def onchange_template_id_wrapper(self):
-        self.ensure_one()
-        values = self.onchange_template_id(self.template_id.id, self.composition_mode, self.model, self.res_id)['value']
-        for fname, value in values.items():
-            setattr(self, fname, value)
+        recipients_info = {}
+        for record_id, mail_values in mail_values_dict.items():
+            mail_to = []
+            if mail_values.get('email_to'):
+                mail_to += email_re.findall(mail_values['email_to'])
+                # if unrecognized email in email_to -> keep it as used for further processing
+                if not mail_to:
+                    mail_to.append(mail_values['email_to'])
+            # add email from recipients (res.partner)
+            mail_to += [
+                recipient_emails[recipient_command[1]]
+                for recipient_command in mail_values.get('recipient_ids') or []
+                if recipient_command[1]
+            ]
+            mail_to = list(set(mail_to))
+            recipients_info[record_id] = {
+                'mail_to': mail_to,
+                'mail_to_normalized': [
+                    tools.email_normalize(mail)
+                    for mail in mail_to
+                    if tools.email_normalize(mail)
+                ]
+            }
+        return recipients_info
 
-    def onchange_template_id(self, template_id, composition_mode, model, res_id):
+    def _process_state(self, mail_values_dict):
+        recipients_info = self._process_recipient_values(mail_values_dict)
+        blacklist_ids = self._get_blacklist_record_ids(mail_values_dict)
+        optout_emails = self._get_optout_emails(mail_values_dict)
+        done_emails = self._get_done_emails(mail_values_dict)
+
+        for record_id, mail_values in mail_values_dict.items():
+            recipients = recipients_info[record_id]
+            # when having more than 1 recipient: we cannot really decide when a single
+            # email is linked to several to -> skip that part. Mass mailing should
+            # anyway always have a single recipient per record as this is default behavior.
+            if len(recipients['mail_to']) > 1:
+                continue
+
+            mail_to = recipients['mail_to'][0] if recipients['mail_to'] else ''
+            mail_to_normalized = recipients['mail_to_normalized'][0] if recipients['mail_to_normalized'] else ''
+
+            # prevent sending to blocked addresses that were included by mistake
+            # blacklisted or optout or duplicate -> cancel
+            if record_id in blacklist_ids:
+                mail_values['state'] = 'cancel'
+                mail_values['failure_type'] = 'mail_bl'
+                # Do not post the mail into the recipient's chatter
+                mail_values['is_notification'] = False
+            elif optout_emails and mail_to in optout_emails:
+                mail_values['state'] = 'cancel'
+                mail_values['failure_type'] = 'mail_optout'
+            elif done_emails and mail_to in done_emails:
+                mail_values['state'] = 'cancel'
+                mail_values['failure_type'] = 'mail_dup'
+            # void of falsy values -> error
+            elif not mail_to:
+                mail_values['state'] = 'cancel'
+                mail_values['failure_type'] = 'mail_email_missing'
+            elif not mail_to_normalized or not email_re.findall(mail_to):
+                mail_values['state'] = 'cancel'
+                mail_values['failure_type'] = 'mail_email_invalid'
+            elif done_emails is not None:
+                done_emails.append(mail_to)
+
+        return mail_values_dict
+
+    def _get_blacklist_record_ids(self, mail_values_dict):
+        blacklisted_rec_ids = set()
+        if self.composition_mode == 'mass_mail' and issubclass(type(self.env[self.model]), self.pool['mail.thread.blacklist']):
+            self.env['mail.blacklist'].flush(['email'])
+            self._cr.execute("SELECT email FROM mail_blacklist WHERE active=true")
+            blacklist = {x[0] for x in self._cr.fetchall()}
+            if blacklist:
+                targets = self.env[self.model].browse(mail_values_dict.keys()).read(['email_normalized'])
+                # First extract email from recipient before comparing with blacklist
+                blacklisted_rec_ids.update(target['id'] for target in targets
+                                           if target['email_normalized'] in blacklist)
+        return blacklisted_rec_ids
+
+    def _get_done_emails(self, mail_values_dict):
+        return []
+
+    def _get_optout_emails(self, mail_values_dict):
+        return []
+
+    def _onchange_template_id(self, template_id, composition_mode, model, res_id):
         """ - mass_mailing: we cannot render, so return the template values
             - normal mode: return rendered values
             /!\ for x2many field, this onchange return command instead of ids
@@ -427,30 +554,6 @@ class MailComposer(models.TransientModel):
 
         return {'value': values}
 
-    def save_as_template(self):
-        """ hit save as template button: current form value will be a new
-            template attached to the current document. """
-        for record in self:
-            model = self.env['ir.model']._get(record.model or 'mail.message')
-            model_name = model.name or ''
-            template_name = "%s: %s" % (model_name, tools.ustr(record.subject))
-            values = {
-                'name': template_name,
-                'subject': record.subject or False,
-                'body_html': record.body or False,
-                'model_id': model.id or False,
-                'attachment_ids': [Command.set([att.id for att in record.attachment_ids])],
-            }
-            template = self.env['mail.template'].create(values)
-            # generate the saved template
-            record.write({'template_id': template.id})
-            record.onchange_template_id_wrapper()
-            return _reopen(self, record.id, record.model, context=self._context)
-
-    # ------------------------------------------------------------
-    # RENDERING
-    # ------------------------------------------------------------
-
     def render_message(self, res_ids):
         """Generate template-based values of wizard, for the document records given
         by res_ids. This method is meant to be inherited by email_template that
@@ -537,20 +640,3 @@ class MailComposer(models.TransientModel):
             values[res_id] = res_id_values
 
         return multi_mode and values or values[res_ids[0]]
-
-    @api.autovacuum
-    def _gc_lost_attachments(self):
-        """ Garbage collect lost mail attachments. Those are attachments
-            - linked to res_model 'mail.compose.message', the composer wizard
-            - with res_id 0, because they were created outside of an existing
-                wizard (typically user input through Chatter or reports
-                created on-the-fly by the templates)
-            - unused since at least one day (create_date and write_date)
-        """
-        limit_date = fields.Datetime.subtract(fields.Datetime.now(), days=1)
-        self.env['ir.attachment'].search([
-            ('res_model', '=', self._name),
-            ('res_id', '=', 0),
-            ('create_date', '<', limit_date),
-            ('write_date', '<', limit_date)]
-        ).unlink()

addons/mail/wizard/mail_compose_message_view.xml

@@ -79,7 +79,8 @@
                         <button string="Log" attrs="{'invisible': [('is_log', '=', False)]}" name="action_send_mail" type="object" class="btn-primary" data-hotkey="q"/>
                         <button string="Cancel" class="btn-secondary" special="cancel" data-hotkey="z" />
 
-                        <button icon="fa-lg fa-save" type="object" name="save_as_template" string="Save as new template"
+                        <button icon="fa-lg fa-save" type="object"
+                                name="action_save_as_template" string="Save as new template"
                                 class="float-right btn-secondary" help="Save as a new template"/>
                     </footer>
                 </form>

addons/mass_mailing/controllers/main.py

@@ -98,7 +98,7 @@ class MassMailController(http.Controller):
         if not consteq(token, tools.hmac(request.env(su=True), 'mass_mailing-mail_mail-open', mail_id)):
             raise BadRequest()
 
-        request.env['mailing.trace'].sudo().set_opened(mail_mail_ids=[mail_id])
+        request.env['mailing.trace'].sudo().set_opened(domain=[('mail_mail_id_int', 'in', [mail_id])])
         response = werkzeug.wrappers.Response()
         response.mimetype = 'image/gif'
         response.data = base64.b64decode(b'R0lGODlhAQABAIAAANvf7wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==')

addons/mass_mailing/data/mass_mailing_demo.xml

@@ -184,9 +184,11 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_7"/>
             <field name="email">billy.fox45@example.com</field>
-            <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
-            <field name="opened" eval="(DateTime.today() - relativedelta(days=2)).strftime('%Y-%m-%d %H:%M:%S')"/>
-            <field name="replied" eval="(DateTime.today() - relativedelta(days=1)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">reply</field>
+            <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="open_datetime" eval="(DateTime.today() - relativedelta(days=4)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="reply_datetime" eval="(DateTime.today() - relativedelta(days=3)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="write_date" eval="(DateTime.today() - relativedelta(days=3)).strftime('%Y-%m-%d %H:%M:%S')"/>
         </record>
         <record id="mass_mail_1_stat_1" model="mailing.trace">
             <field name="mass_mailing_id" ref="mass_mail_1"/>
@@ -194,9 +196,11 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_13"/>
             <field name="email">kim.snyder96@example.com</field>
-            <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
-            <field name="opened" eval="(DateTime.today() - relativedelta(days=2)).strftime('%Y-%m-%d %H:%M:%S')"/>
-            <field name="replied" eval="(DateTime.today() - relativedelta(days=0)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">reply</field>
+            <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="open_datetime" eval="(DateTime.today() - relativedelta(days=2)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="reply_datetime" eval="(DateTime.today() - relativedelta(days=0)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="write_date" eval="(DateTime.today() - relativedelta(days=0)).strftime('%Y-%m-%d %H:%M:%S')"/>
         </record>
         <record id="mass_mail_1_stat_2" model="mailing.trace">
             <field name="mass_mailing_id" ref="mass_mail_1"/>
@@ -204,8 +208,10 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_14"/>
             <field name="email">edith.sanchez68@example.com</field>
-            <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
-            <field name="opened" eval="(DateTime.today() - relativedelta(days=2)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">open</field>
+            <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="open_datetime" eval="(DateTime.today() - relativedelta(days=2)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="write_date" eval="(DateTime.today() - relativedelta(days=2)).strftime('%Y-%m-%d %H:%M:%S')"/>
         </record>
         <record id="mass_mail_1_stat_3" model="mailing.trace">
             <field name="mass_mailing_id" ref="mass_mail_1"/>
@@ -213,8 +219,10 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_24"/>
             <field name="email">theodore.gardner36@example.com</field>
-            <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
-            <field name="opened" eval="(DateTime.today() - relativedelta(days=1)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">open</field>
+            <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="open_datetime" eval="(DateTime.today() - relativedelta(days=1)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="write_date" eval="(DateTime.today() - relativedelta(days=1)).strftime('%Y-%m-%d %H:%M:%S')"/>
         </record>
         <record id="mass_mail_1_stat_4" model="mailing.trace">
             <field name="mass_mailing_id" ref="mass_mail_1"/>
@@ -222,7 +230,9 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_32"/>
             <field name="email">sandra.neal80@example.com</field>
-            <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">sent</field>
+            <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="write_date" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
         </record>
         <record id="mass_mail_1_stat_5" model="mailing.trace">
             <field name="mass_mailing_id" ref="mass_mail_1"/>
@@ -230,8 +240,8 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_33"/>
             <field name="email">julie.richards84@example.com</field>
-            <field name="sent" eval="False"/>
-            <field name="exception" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">error</field>
+            <field name="sent_datetime" eval="False"/>
         </record>
         <record id="mass_mail_1_stat_6" model="mailing.trace">
             <field name="mass_mailing_id" ref="mass_mail_1"/>
@@ -239,8 +249,9 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_34"/>
             <field name="email">travis.mendoza24@example.com</field>
-            <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
-            <field name="bounced" eval="(DateTime.today() - relativedelta(days=3)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">bounce</field>
+            <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="write_date" eval="(DateTime.today() - relativedelta(days=3)).strftime('%Y-%m-%d %H:%M:%S')"/>
         </record>
         <record id="mass_mail_1_stat_7" model="mailing.trace">
             <field name="mass_mailing_id" ref="mass_mail_1"/>
@@ -248,8 +259,8 @@
             <field name="model">res.partner</field>
             <field name="res_id" ref="base.res_partner_address_34"/>
             <field name="email">travis.mendoza24@example.com</field>
-            <field name="sent" eval="False"/>
-            <field name="ignored" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+            <field name="trace_status">bounce</field>
+            <field name="sent_datetime" eval="False"/>
         </record>
 
         <!-- Generate some clicks -->

addons/mass_mailing/models/mail_mail.py

@@ -81,7 +81,7 @@ class MailMail(models.Model):
         for mail in self:
             if mail.mailing_id:
                 if mail_sent is True and mail.mailing_trace_ids:
-                    mail.mailing_trace_ids.write({'sent': fields.Datetime.now(), 'exception': False})
+                    mail.mailing_trace_ids.set_sent()
                 elif mail_sent is False and mail.mailing_trace_ids:
-                    mail.mailing_trace_ids.write({'exception': fields.Datetime.now(), 'failure_type': failure_type})
+                    mail.mailing_trace_ids.set_failed(failure_type=failure_type)
         return super(MailMail, self)._postprocess_sent_message(success_pids, failure_reason=failure_reason, failure_type=failure_type)

addons/mass_mailing/models/mail_thread.py

@@ -23,8 +23,8 @@ class MailThread(models.AbstractModel):
             thread_references = message_dict['references'] or message_dict['in_reply_to']
             msg_references = tools.mail_header_msgid_re.findall(thread_references)
             if msg_references:
-                self.env['mailing.trace'].set_opened(mail_message_ids=msg_references)
-                self.env['mailing.trace'].set_replied(mail_message_ids=msg_references)
+                self.env['mailing.trace'].set_opened(domain=[('message_id', 'in', msg_references)])
+                self.env['mailing.trace'].set_replied(domain=[('message_id', 'in', msg_references)])
         return super(MailThread, self)._message_route_process(message, message_dict, routes)
 
     def message_post_with_template(self, template_id, **kwargs):
@@ -53,10 +53,10 @@ class MailThread(models.AbstractModel):
         bounced_partner = message_dict['bounced_partner']
 
         if bounced_msg_id:
-            self.env['mailing.trace'].set_bounced(mail_message_ids=bounced_msg_id)
+            self.env['mailing.trace'].set_bounced(domain=[('message_id', 'in', bounced_msg_id)])
         if bounced_email:
             three_months_ago = fields.Datetime.to_string(datetime.datetime.now() - datetime.timedelta(weeks=13))
-            stats = self.env['mailing.trace'].search(['&', ('bounced', '>', three_months_ago), ('email', '=ilike', bounced_email)]).mapped('bounced')
+            stats = self.env['mailing.trace'].search(['&', '&', ('trace_status', '=', 'bounce'), ('write_date', '>', three_months_ago), ('email', '=ilike', bounced_email)]).mapped('write_date')
             if len(stats) >= BLACKLIST_MAX_BOUNCED_LIMIT and (not bounced_partner or any(p.message_bounce >= BLACKLIST_MAX_BOUNCED_LIMIT for p in bounced_partner)):
                 if max(stats) > min(stats) + datetime.timedelta(weeks=1):
                     blacklist_rec = self.env['mail.blacklist'].sudo()._add(bounced_email)

addons/mass_mailing/models/mailing.py

@@ -143,7 +143,7 @@ class MassMailing(models.Model):
     total = fields.Integer(compute="_compute_total")
     scheduled = fields.Integer(compute="_compute_statistics")
     expected = fields.Integer(compute="_compute_statistics")
-    ignored = fields.Integer(compute="_compute_statistics")
+    canceled = fields.Integer(compute="_compute_statistics")
     sent = fields.Integer(compute="_compute_statistics")
     delivered = fields.Integer(compute="_compute_statistics")
     opened = fields.Integer(compute="_compute_statistics")
@@ -203,7 +203,7 @@ class MassMailing(models.Model):
     def _compute_statistics(self):
         """ Compute statistics of the mass mailing """
         for key in (
-            'scheduled', 'expected', 'ignored', 'sent', 'delivered', 'opened',
+            'scheduled', 'expected', 'canceled', 'sent', 'delivered', 'opened',
             'clicked', 'replied', 'bounced', 'failed', 'received_ratio',
             'opened_ratio', 'replied_ratio', 'bounced_ratio',
         ):
@@ -216,15 +216,15 @@ class MassMailing(models.Model):
             SELECT
                 m.id as mailing_id,
                 COUNT(s.id) AS expected,
-                COUNT(CASE WHEN s.sent is not null THEN 1 ELSE null END) AS sent,
-                COUNT(CASE WHEN s.scheduled is not null AND s.sent is null AND s.exception is null AND s.ignored is null AND s.bounced is null THEN 1 ELSE null END) AS scheduled,
-                COUNT(CASE WHEN s.scheduled is not null AND s.sent is null AND s.exception is null AND s.ignored is not null THEN 1 ELSE null END) AS ignored,
-                COUNT(CASE WHEN s.sent is not null AND s.exception is null AND s.bounced is null THEN 1 ELSE null END) AS delivered,
-                COUNT(CASE WHEN s.opened is not null THEN 1 ELSE null END) AS opened,
-                COUNT(CASE WHEN s.clicked is not null THEN 1 ELSE null END) AS clicked,
-                COUNT(CASE WHEN s.replied is not null THEN 1 ELSE null END) AS replied,
-                COUNT(CASE WHEN s.bounced is not null THEN 1 ELSE null END) AS bounced,
-                COUNT(CASE WHEN s.exception is not null THEN 1 ELSE null END) AS failed
+                COUNT(s.sent_datetime) AS sent,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'outgoing') AS scheduled,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'cancel') AS canceled,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status in ('sent', 'open', 'reply')) AS delivered,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status in ('open', 'reply')) AS opened,
+                COUNT(s.links_click_datetime) AS clicked,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'reply') AS replied,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'bounce') AS bounced,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'error') AS failed
             FROM
                 mailing_trace s
             RIGHT JOIN
@@ -236,7 +236,7 @@ class MassMailing(models.Model):
                 m.id
         """, (tuple(self.ids), ))
         for row in self.env.cr.dictfetchall():
-            total = (row['expected'] - row['ignored']) or 1
+            total = (row['expected'] - row['canceled']) or 1
             row['received_ratio'] = 100.0 * row['delivered'] / total
             row['opened_ratio'] = 100.0 * row['opened'] / total
             row['replied_ratio'] = 100.0 * row['replied'] / total
@@ -431,8 +431,8 @@ class MassMailing(models.Model):
     def action_view_traces_scheduled(self):
         return self._action_view_traces_filtered('scheduled')
 
-    def action_view_traces_ignored(self):
-        return self._action_view_traces_filtered('ignored')
+    def action_view_traces_canceled(self):
+        return self._action_view_traces_filtered('canceled')
 
     def action_view_traces_failed(self):
         return self._action_view_traces_filtered('failed')
@@ -442,10 +442,14 @@ class MassMailing(models.Model):
 
     def _action_view_traces_filtered(self, view_filter):
         action = self.env["ir.actions.actions"]._for_xml_id("mass_mailing.mailing_trace_action")
-        action['name'] = _('%s Traces') % (self.name)
+        action['name'] = _('Sent Mailings')
         action['context'] = {'search_default_mass_mailing_id': self.id,}
         filter_key = 'search_default_filter_%s' % (view_filter)
         action['context'][filter_key] = True
+        action['views'] = [
+            (self.env.ref('mass_mailing.mailing_trace_view_tree_mail').id, 'tree'),
+            (self.env.ref('mass_mailing.mailing_trace_view_form').id, 'form')
+        ]
         return action
 
     def action_view_clicked(self):
@@ -460,25 +464,31 @@ class MassMailing(models.Model):
         }
 
     def action_view_opened(self):
-        return self._action_view_documents_filtered('opened')
+        return self._action_view_documents_filtered('open')
 
     def action_view_replied(self):
-        return self._action_view_documents_filtered('replied')
+        return self._action_view_documents_filtered('reply')
 
     def action_view_bounced(self):
-        return self._action_view_documents_filtered('bounced')
+        return self._action_view_documents_filtered('bounce')
 
     def action_view_delivered(self):
         return self._action_view_documents_filtered('delivered')
 
     def _action_view_documents_filtered(self, view_filter):
-        if view_filter in ('opened', 'replied', 'bounced'):
-            opened_stats = self.mailing_trace_ids.filtered(lambda stat: stat[view_filter])
-        elif view_filter == ('delivered'):
-            opened_stats = self.mailing_trace_ids.filtered(lambda stat: stat.sent and not stat.bounced)
+        if view_filter in ('reply', 'bounce'):
+            found_traces = self.mailing_trace_ids.filtered(lambda trace: trace.trace_status == view_filter)
+        elif view_filter == 'open':
+            found_traces = self.mailing_trace_ids.filtered(lambda trace: trace.trace_status in ('open', 'reply'))
+        elif view_filter == 'click':
+            found_traces = self.mailing_trace_ids.filtered(lambda trace: trace.links_click_datetime)
+        elif view_filter == 'delivered':
+            found_traces = self.mailing_trace_ids.filtered(lambda trace: trace.trace_status in ('sent', 'open', 'reply'))
+        elif view_filter == 'sent':
+            found_traces = self.mailing_trace_ids.filtered(lambda trace: trace.sent_datetime)
         else:
-            opened_stats = self.env['mailing.trace']
-        res_ids = opened_stats.mapped('res_id')
+            found_traces = self.env['mailing.trace']
+        res_ids = found_traces.mapped('res_id')
         model_name = self.env['ir.model']._get(self.mailing_model_real).display_name
         return {
             'name': model_name,
@@ -597,8 +607,6 @@ class MassMailing(models.Model):
     def _get_mass_mailing_context(self):
         """Returns extra context items with pre-filled blacklist and seen list for massmailing"""
         return {
-            'mass_mailing_opt_out_list': self._get_opt_out_list(),
-            'mass_mailing_seen_list': self._get_seen_list(),
             'post_convert_links': self._get_link_tracker_values(),
         }
 
@@ -687,7 +695,7 @@ class MassMailing(models.Model):
             composer = composer.with_context(active_ids=res_ids, **extra_context)
             # auto-commit except in testing mode
             auto_commit = not getattr(threading.currentThread(), 'testing', False)
-            composer.send_mail(auto_commit=auto_commit)
+            composer._action_send_mail(auto_commit=auto_commit)
             mailing.write({
                 'state': 'done',
                 'sent_date': fields.Datetime.now(),

addons/mass_mailing/models/mailing_trace.py

@@ -12,31 +12,32 @@ class MailingTrace(models.Model):
 
     Note:: State management / Error codes / Failure types summary
 
-      * state
+      * trace_status
         'outgoing', 'sent', 'opened', 'replied',
-        'exception', 'bounced', 'ignored'
+        'error', 'bouce', 'cancel'
       * failure_type
+        # generic
+        'unknown',
         # mass_mailing
-        "SMTP", "RECIPIENT", "BOUNCE", "UNKNOWN"
+        "mail_email_invalid", "mail_smtp", "mail_email_missing"
+        # mass mailing mass mode specific codes
+        "mail_bl", "mail_optout", "mail_dup"
         # mass_mailing_sms
         'sms_number_missing', 'sms_number_format', 'sms_credit',
         'sms_server', 'sms_acc'
         # mass_mailing_sms mass mode specific codes
-        'sms_blacklist', 'sms_duplicate'
-      * ignored:
+        'sms_blacklist', 'sms_duplicate', 'sms_optout',
+      * cancel:
         * mail: set in get_mail_values in composer, if email is blacklisted
           (mail) or in opt_out / seen list (mass_mailing) or email_to is void
           or incorrectly formatted (mass_mailing) - based on mail cancel state
         * sms: set in _prepare_mass_sms_trace_values in composer if sms is
           in cancel state; either blacklisted (sms) or in opt_out / seen list
           (sms);
-        * difference: void mail -> cancel -> ignore, void sms -> error
-          sms_number_missing -> exception
-        * difference: invalid mail -> cancel -> ignore, invalid sms -> error
-          sms_number_format -> sent + bounce;
+        * void mail / void sms number -> error (mail_missing, sms_number_missing)
+        * invalid mail / invalid sms number -> error (RECIPIENT, sms_number_format)
       * exception: set in  _postprocess_sent_message (_postprocess_iap_sent_sms)
-        if mail (sms) not sent with failure type, reset if sent; also set for
-        sms in _prepare_mass_sms_trace_values if void number
+        if mail (sms) not sent with failure type, reset if sent;
       * sent: set in _postprocess_sent_message (_postprocess_iap_sent_sms) if
         mail (sms) sent
       * clicked: triggered by add_click
@@ -48,9 +49,9 @@ class MailingTrace(models.Model):
     _name = 'mailing.trace'
     _description = 'Mailing Statistics'
     _rec_name = 'id'
-    _order = 'scheduled DESC'
+    _order = 'create_date DESC'
 
-    trace_type = fields.Selection([('mail', 'Mail')], string='Type', default='mail', required=True)
+    trace_type = fields.Selection([('mail', 'Email')], string='Type', default='mail', required=True)
     display_name = fields.Char(compute='_compute_display_name')
     # mail data
     mail_mail_id = fields.Many2one('mail.mail', string='Mail', index=True)
@@ -62,73 +63,51 @@ class MailingTrace(models.Model):
         index=True,
     )
     email = fields.Char(string="Email", help="Normalized email address")
-    message_id = fields.Char(string='Message-ID')
+    message_id = fields.Char(string='Message-ID', help="Technical field for the email Message-ID (RFC 2392)")
     medium_id = fields.Many2one(related='mass_mailing_id.medium_id')
     source_id = fields.Many2one(related='mass_mailing_id.source_id')
     # document
     model = fields.Char(string='Document model', required=True)
     res_id = fields.Many2oneReference(string='Document ID', model_field='model', required=True)
-    # campaign / wave data
+    # campaign data
     mass_mailing_id = fields.Many2one('mailing.mailing', string='Mailing', index=True, ondelete='cascade')
     campaign_id = fields.Many2one(
         related='mass_mailing_id.campaign_id',
         string='Campaign',
         store=True, readonly=True, index=True)
-    # Bounce and tracking
-    ignored = fields.Datetime(help='Date when the email has been invalidated. '
-                                   'Invalid emails are blacklisted, opted-out or invalid email format')
-    scheduled = fields.Datetime(help='Date when the email has been created', default=fields.Datetime.now)
-    sent = fields.Datetime(help='Date when the email has been sent')
-    exception = fields.Datetime(help='Date of technical error leading to the email not being sent')
-    opened = fields.Datetime(help='Date when the email has been opened the first time')
-    replied = fields.Datetime(help='Date when this email has been replied for the first time.')
-    bounced = fields.Datetime(help='Date when this email has bounced.')
+    # Status
+    sent_datetime = fields.Datetime('Sent On')
+    open_datetime = fields.Datetime('Opened On')
+    reply_datetime = fields.Datetime('Replied On')
+    trace_status = fields.Selection(selection=[
+        ('outgoing', 'Outgoing'),
+        ('sent', 'Sent'),
+        ('open', 'Opened'),
+        ('reply', 'Replied'),
+        ('bounce', 'Bounced'),
+        ('error', 'Exception'),
+        ('cancel', 'Canceled')], string='Status', default='outgoing')
+    failure_type = fields.Selection(selection=[
+        # generic
+        ("unknown", "Unknown error"),
+        # mail
+        ("mail_email_invalid", "Invalid email address"),
+        ("mail_email_missing", "Missing email address"),
+        ("mail_smtp", "Connection failed (outgoing mail server problem)"),
+        # mass mode
+        ("mail_bl", "Blacklisted Address"),
+        ("mail_optout", "Opted Out"),
+        ("mail_dup", "Duplicated Email"),
+    ], string='Failure type')
     # Link tracking
     links_click_ids = fields.One2many('link.tracker.click', 'mailing_trace_id', string='Links click')
-    clicked = fields.Datetime(help='Date when customer clicked on at least one tracked link')
-    # Status
-    state = fields.Selection(compute="_compute_state",
-                             selection=[('outgoing', 'Outgoing'),
-                                        ('exception', 'Exception'),
-                                        ('sent', 'Sent'),
-                                        ('opened', 'Opened'),
-                                        ('replied', 'Replied'),
-                                        ('bounced', 'Bounced'),
-                                        ('ignored', 'Ignored')], store=True)
-    failure_type = fields.Selection(selection=[
-        ("SMTP", "Connection failed (outgoing mail server problem)"),
-        ("RECIPIENT", "Invalid email address"),
-        ("BOUNCE", "Email address rejected by destination"),
-        ("UNKNOWN", "Unknown error"),
-    ], string='Failure type')
-    state_update = fields.Datetime(compute="_compute_state", string='State Update',
-                                   help='Last state update of the mail',
-                                   store=True)
+    links_click_datetime = fields.Datetime('Clicked On', help='Stores last click datetime in case of multi clicks.')
 
     @api.depends('trace_type', 'mass_mailing_id')
     def _compute_display_name(self):
         for trace in self:
             trace.display_name = '%s: %s (%s)' % (trace.trace_type, trace.mass_mailing_id.name, trace.id)
 
-    @api.depends('sent', 'opened', 'clicked', 'replied', 'bounced', 'exception', 'ignored')
-    def _compute_state(self):
-        self.update({'state_update': fields.Datetime.now()})
-        for stat in self:
-            if stat.ignored:
-                stat.state = 'ignored'
-            elif stat.exception:
-                stat.state = 'exception'
-            elif stat.replied:
-                stat.state = 'replied'
-            elif stat.opened or stat.clicked:
-                stat.state = 'opened'
-            elif stat.bounced:
-                stat.state = 'bounced'
-            elif stat.sent:
-                stat.state = 'sent'
-            else:
-                stat.state = 'outgoing'
-
     @api.model_create_multi
     def create(self, values_list):
         for values in values_list:
@@ -146,33 +125,37 @@ class MailingTrace(models.Model):
             'res_id': self.res_id
         }
 
-    def _get_records(self, mail_mail_ids=None, mail_message_ids=None, domain=None):
-        if not self.ids and mail_mail_ids:
-            base_domain = [('mail_mail_id_int', 'in', mail_mail_ids)]
-        elif not self.ids and mail_message_ids:
-            base_domain = [('message_id', 'in', mail_message_ids)]
-        else:
-            base_domain = [('id', 'in', self.ids)]
-        if domain:
-            base_domain = ['&'] + domain + base_domain
-        return self.search(base_domain)
-
-    def set_opened(self, mail_mail_ids=None, mail_message_ids=None):
-        traces = self._get_records(mail_mail_ids, mail_message_ids, [('opened', '=', False)])
-        traces.write({'opened': fields.Datetime.now(), 'bounced': False})
+    def set_sent(self, domain=None):
+        traces = self + (self.search(domain) if domain else self.env['mailing.trace'])
+        traces.write({'trace_status': 'sent', 'sent_datetime': fields.Datetime.now(), 'failure_type': False})
         return traces
 
-    def set_clicked(self, mail_mail_ids=None, mail_message_ids=None):
-        traces = self._get_records(mail_mail_ids, mail_message_ids, [('clicked', '=', False)])
-        traces.write({'clicked': fields.Datetime.now()})
+    def set_opened(self, domain=None):
+        traces = self + (self.search(domain) if domain else self.env['mailing.trace'])
+        traces.write({'trace_status': 'open', 'open_datetime': fields.Datetime.now()})
         return traces
 
-    def set_replied(self, mail_mail_ids=None, mail_message_ids=None):
-        traces = self._get_records(mail_mail_ids, mail_message_ids, [('replied', '=', False)])
-        traces.write({'replied': fields.Datetime.now()})
+    def set_clicked(self, domain=None):
+        traces = self + (self.search(domain) if domain else self.env['mailing.trace'])
+        traces.write({'links_click_datetime': fields.Datetime.now()})
         return traces
 
-    def set_bounced(self, mail_mail_ids=None, mail_message_ids=None):
-        traces = self._get_records(mail_mail_ids, mail_message_ids, [('bounced', '=', False), ('opened', '=', False)])
-        traces.write({'bounced': fields.Datetime.now()})
+    def set_replied(self, domain=None):
+        traces = self + (self.search(domain) if domain else self.env['mailing.trace'])
+        traces.write({'trace_status': 'reply', 'reply_datetime': fields.Datetime.now()})
+        return traces
+
+    def set_bounced(self, domain=None):
+        traces = self + (self.search(domain) if domain else self.env['mailing.trace'])
+        traces.write({'trace_status': 'bounce'})
+        return traces
+
+    def set_failed(self, domain=None, failure_type=False):
+        traces = self + (self.search(domain) if domain else self.env['mailing.trace'])
+        traces.write({'trace_status': 'error', 'failure_type': failure_type})
+        return traces
+
+    def set_canceled(self, domain=None):
+        traces = self + (self.search(domain) if domain else self.env['mailing.trace'])
+        traces.write({'trace_status': 'cancel'})
         return traces

addons/mass_mailing/models/utm.py

@@ -47,12 +47,12 @@ class UtmCampaign(models.Model):
             SELECT
                 c.id as campaign_id,
                 COUNT(s.id) AS expected,
-                COUNT(CASE WHEN s.sent is not null THEN 1 ELSE null END) AS sent,
-                COUNT(CASE WHEN s.scheduled is not null AND s.sent is null AND s.exception is null AND s.ignored is not null THEN 1 ELSE null END) AS ignored,
-                COUNT(CASE WHEN s.id is not null AND s.bounced is null THEN 1 ELSE null END) AS delivered,
-                COUNT(CASE WHEN s.opened is not null THEN 1 ELSE null END) AS opened,
-                COUNT(CASE WHEN s.replied is not null THEN 1 ELSE null END) AS replied,
-                COUNT(CASE WHEN s.bounced is not null THEN 1 ELSE null END) AS bounced
+                COUNT(s.sent_datetime) AS sent,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status in ('sent', 'open', 'reply')) AS delivered,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status in ('open', 'reply')) AS open,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'reply') AS reply,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'bounce') AS bounce,
+                COUNT(s.trace_status) FILTER (WHERE s.trace_status = 'cancel') AS cancel
             FROM
                 mailing_trace s
             RIGHT JOIN
@@ -75,13 +75,13 @@ class UtmCampaign(models.Model):
             if not stats:
                 vals = default_vals
             else:
-                total = (stats['expected'] - stats['ignored']) or 1
-                delivered = stats['sent'] - stats['bounced']
+                total = (stats['expected'] - stats['cancel']) or 1
+                delivered = stats['sent'] - stats['bounce']
                 vals = {
                     'received_ratio': 100.0 * delivered / total,
-                    'opened_ratio': 100.0 * stats['opened'] / total,
-                    'replied_ratio': 100.0 * stats['replied'] / total,
-                    'bounced_ratio': 100.0 * stats['bounced'] / total
+                    'opened_ratio': 100.0 * stats['open'] / total,
+                    'replied_ratio': 100.0 * stats['reply'] / total,
+                    'bounced_ratio': 100.0 * stats['bounce'] / total
                 }
 
             campaign.update(vals)

addons/mass_mailing/report/mailing_trace_report.py

@@ -19,12 +19,15 @@ class MailingTraceReport(models.Model):
         string='Status', readonly=True)
     email_from = fields.Char('From', readonly=True)
     # traces
+    scheduled = fields.Integer(readonly=True)
     sent = fields.Integer(readonly=True)
     delivered = fields.Integer(readonly=True)
+    error = fields.Integer(readonly=True)
     opened = fields.Integer(readonly=True)
     replied = fields.Integer(readonly=True)
-    clicked = fields.Integer(readonly=True)
     bounced = fields.Integer(readonly=True)
+    canceled = fields.Integer(readonly=True)
+    clicked = fields.Integer(readonly=True)
 
     def init(self):
         """Mass Mail Statistical Report: based on mailing.trace that models the various
@@ -52,23 +55,26 @@ class MailingTraceReport(models.Model):
             'utm_source.name as name',
             'mailing.mailing_type',
             'utm_campaign.name as campaign',
-            'trace.scheduled as scheduled_date',
+            'trace.create_date as scheduled_date',
             'mailing.state',
             'mailing.email_from',
-            'count(trace.sent) as sent',
-            '(count(trace.sent) - count(trace.bounced)) as delivered',
-            'count(trace.opened) as opened',
-            'count(trace.replied) as replied',
-            'count(trace.clicked) as clicked',
-            'count(trace.bounced) as bounced'
+            "COUNT(trace.id) as scheduled",
+            'COUNT(trace.sent_datetime) as sent',
+            "(COUNT(trace.id) - COUNT(trace.trace_status) FILTER (WHERE trace.trace_status IN ('error', 'bounce', 'cancel'))) as delivered",
+            "COUNT(trace.trace_status) FILTER (WHERE trace.trace_status = 'error') as error",
+            "COUNT(trace.trace_status) FILTER (WHERE trace.trace_status = 'bounce') as bounced",
+            "COUNT(trace.trace_status) FILTER (WHERE trace.trace_status = 'cancel') as canceled",
+            "COUNT(trace.trace_status) FILTER (WHERE trace.trace_status = 'open') as opened",
+            "COUNT(trace.trace_status) FILTER (WHERE trace.trace_status = 'reply') as replied",
+            "COUNT(trace.links_click_datetime) as clicked",
         ]
 
     def _report_get_request_from_items(self):
         return [
             'mailing_trace as trace',
-            'left join mailing_mailing as mailing ON (trace.mass_mailing_id=mailing.id)',
-            'left join utm_campaign as utm_campaign ON (mailing.campaign_id = utm_campaign.id)',
-            'left join utm_source as utm_source ON (mailing.source_id = utm_source.id)'
+            'LEFT JOIN mailing_mailing as mailing ON (trace.mass_mailing_id=mailing.id)',
+            'LEFT JOIN utm_campaign as utm_campaign ON (mailing.campaign_id = utm_campaign.id)',
+            'LEFT JOIN utm_source as utm_source ON (mailing.source_id = utm_source.id)'
         ]
 
     def _report_get_request_where_items(self):
@@ -76,7 +82,7 @@ class MailingTraceReport(models.Model):
 
     def _report_get_request_group_by_items(self):
         return [
-            'trace.scheduled',
+            'trace.create_date',
             'utm_source.name',
             'utm_campaign.name',
             'mailing.mailing_type',

addons/mass_mailing/report/mailing_trace_report_views.xml

@@ -1,17 +1,42 @@
 <?xml version="1.0" encoding="utf-8"?>
 <odoo>
+        <record id="mailing_trace_report_view_tree" model="ir.ui.view">
+            <field name="name">mailing.trace.report.view.tree</field>
+            <field name="model">mailing.trace.report</field>
+            <field name="arch" type="xml">
+                <tree string="Mass Mailing Statistics" sample="1">
+                    <field name="name"/>
+                    <field name="campaign" groups="mass_mailing.group_mass_mailing_campaign"/>
+                    <field name="mailing_type" invisible="1"/>
+                    <field name="state"/>
+                    <field name="scheduled"/>
+                    <field name="sent"/>
+                    <field name="delivered"/>
+                    <field name="opened"/>
+                    <field name="replied"/>
+                    <field name="clicked"/>
+                    <field name="canceled" optional="hide"/>
+                    <field name="error" optional="hide"/>
+                    <field name="bounced" optional="hide"/>
+                </tree>
+            </field>
+        </record>
+
         <record id="mailing_trace_report_view_pivot" model="ir.ui.view">
             <field name="name">mailing.trace.report.view.pivot</field>
             <field name="model">mailing.trace.report</field>
             <field name="arch" type="xml">
-                <pivot string="Mass Mailing Statistics" disable_linking="True" sample="1">
+                <pivot string="Mass Mailing Statistics" disable_linking="1" sample="1">
                     <field name="name" type="row"/>
                     <field name="sent" type="measure"/>
+                    <field name="scheduled" type="measure"/>
                     <field name="delivered" type="measure"/>
                     <field name="opened" type="measure"/>
-                    <field name="bounced" type="measure"/>
-                    <field name="replied" type="measure" invisible="0"/>
+                    <field name="replied" type="measure"/>
                     <field name="clicked" type="measure"/>
+                    <field name="canceled"/>
+                    <field name="error"/>
+                    <field name="bounced"/>
                 </pivot>
             </field>
         </record>
@@ -20,10 +45,11 @@
             <field name="name">mailing.trace.report.view.graph</field>
             <field name="model">mailing.trace.report</field>
             <field name="arch" type="xml">
-                <graph string="Mass Mailing Statistics" sample="1" disable_linking="1">
+                <graph string="Mass Mailing Statistics" disable_linking="1" sample="1">
                     <field name="name"/>
                     <field name="sent" type="measure"/>
-                    <field name="replied" invisible="0"/>
+                    <field name="replied"/>
+                    <field name="clicked"/>
                 </graph>
             </field>
         </record>
@@ -35,9 +61,9 @@
                 <search string="Mass Mailing Statistics">
                     <field name="name" string="Mailing"/>
                     <field name="campaign" string="Campaign" groups="mass_mailing.group_mass_mailing_campaign"/>
-                    <filter name="filter_scheduled_date" date="scheduled_date"/>
+                    <filter name="filter_scheduled_date" date="scheduled"/>
                     <group expand="0" string="Extended Filters...">
-                        <field name="scheduled_date"/>
+                        <field name="scheduled"/>
                     </group>
                     <group expand="1" string="Group By...">
                         <filter string="Mass Mailing Campaign" domain="[]" name="mass_mailing_campaign"
@@ -47,8 +73,8 @@
                         <filter string="Sent By" domain="[]" name="sent_by"
                             context="{'group_by':'email_from'}"/>
                         <separator/>
-                        <filter string="Scheduled Period" name="scheduled_date"
-                            domain="[]" context="{'group_by':'scheduled_date'}"/>
+                        <filter string="Scheduled Period" name="scheduled"
+                            domain="[]" context="{'group_by':'scheduled'}"/>
                     </group>
                 </search>
             </field>
@@ -59,8 +85,14 @@
            <field name="name">Mass Mailing Analysis</field>
            <field name="res_model">mailing.trace.report</field>
            <field name="domain">[('mailing_type', '=', 'mail')]</field>
-           <field name="view_mode">graph,pivot</field>
-           <field name="help" type="html"><p>Mass Mailing Statistics allows you to check different mailing related information like number of bounced mails, opened mails, replied mails. You can sort out your analysis by different groups to get accurate grained analysis.</p></field>
+           <field name="view_mode">graph,pivot,tree</field>
+           <field name="help" type="html">
+<p class="o_view_nocontent_smiling_face">
+    Mass Mailing Statistics allows you to check different mailing related information
+    like number of bounced mails, opened mails, replied mails. You can sort out
+    your analysis by different groups to get accurate grained analysis.
+</p>
+</field>
        </record>
 
        <menuitem name="Reporting" id="menu_mass_mailing_report" sequence="99"

addons/mass_mailing/tests/common.py

@@ -6,9 +6,13 @@ import random
 import re
 import werkzeug
 
+from unittest.mock import patch
+
+from odoo import tools
 from odoo.addons.link_tracker.tests.common import MockLinkTracker
 from odoo.addons.mail.tests.common import MailCase, MailCommon, mail_new_test_user
-from odoo import tools
+from odoo.sql_db import Cursor
+
 
 class MassMailCase(MailCase, MockLinkTracker):
 
@@ -25,7 +29,7 @@ class MassMailCase(MailCase, MockLinkTracker):
             kwargs['delivered'] = len(mailing.mailing_trace_ids)
         for fname in ['scheduled', 'expected', 'sent', 'delivered',
                       'opened', 'replied', 'clicked',
-                      'ignored', 'failed', 'bounced']:
+                      'canceled', 'failed', 'bounced']:
             self.assertEqual(
                 mailing[fname], kwargs.get(fname, 0),
                 'Mailing %s statistics failed: got %s instead of %s' % (fname, mailing[fname], kwargs.get(fname, 0))
@@ -44,7 +48,7 @@ class MassMailCase(MailCase, MockLinkTracker):
             # TRACE
             'partner': res.partner record (may be empty),
             'email': email used when sending email (may be empty, computed based on partner),
-            'state': outgoing / sent / ignored / bounced / exception / opened (sent by default),
+            'trace_status': outgoing / sent / open / reply / bounce / error / cancel (sent by default),
             'record: linked record,
             # MAIL.MAIL
             'content': optional content that should be present in mail.mail body_html;
@@ -67,12 +71,11 @@ class MassMailCase(MailCase, MockLinkTracker):
         # map trace state to email state
         state_mapping = {
             'sent': 'sent',
-            'opened': 'sent',  # opened implies something has been sent
-            'replied': 'sent',  # replied implies something has been sent
-            'ignored': 'cancel',
-            'exception': 'exception',
-            'canceled': 'cancel',
-            'bounced': 'cancel',
+            'open': 'sent',  # opened implies something has been sent
+            'reply': 'sent',  # replied implies something has been sent
+            'error': 'exception',
+            'cancel': 'cancel',
+            'bounce': 'cancel',
         }
 
         traces = self.env['mailing.trace'].search([
@@ -90,40 +93,43 @@ class MassMailCase(MailCase, MockLinkTracker):
         for recipient_info, link_info, record in zip(recipients_info, mail_links_info, records):
             partner = recipient_info.get('partner', self.env['res.partner'])
             email = recipient_info.get('email')
-            state = recipient_info.get('state', 'sent')
+            status = recipient_info.get('trace_status', 'sent')
             record = record or recipient_info.get('record')
             content = recipient_info.get('content')
             if email is None and partner:
                 email = partner.email_normalized
 
             recipient_trace = traces.filtered(
-                lambda t: t.email == email and t.state == state and (t.res_id == record.id if record else True)
+                lambda t: (t.email == email or (not email and not t.email)) and \
+                          t.trace_status == status and \
+                          (t.res_id == record.id if record else True)
             )
             self.assertTrue(
                 len(recipient_trace) == 1,
-                'MailTrace: email %s (recipient %s, state: %s, record: %s): found %s records (1 expected)' % (email, partner, state, record, len(recipient_trace))
+                'MailTrace: email %s (recipient %s, status: %s, record: %s): found %s records (1 expected)' % (email, partner, status, record, len(recipient_trace))
             )
             self.assertTrue(bool(recipient_trace.mail_mail_id_int))
+            if 'failure_type' in recipient_info or status in ('error', 'cancel', 'bounce'):
+                self.assertEqual(recipient_trace.failure_type, recipient_info['failure_type'])
 
             if check_mail:
                 if author is None:
                     author = self.env.user.partner_id
 
+                # mail.mail specific values to check
                 fields_values = {'mailing_id': mailing}
-                if 'failure_type' in recipient_info:
-                    fields_values['failure_type'] = recipient_info['failure_type']
 
                 # specific for partner: email_formatted is used
                 if partner:
-                    if state == 'sent' and sent_unlink:
+                    if status == 'sent' and sent_unlink:
                         self.assertSentEmail(author, [partner])
                     else:
-                        self.assertMailMail(partner, state_mapping[state], author=author, content=content, fields_values=fields_values)
+                        self.assertMailMail(partner, state_mapping[status], author=author, content=content, fields_values=fields_values)
                 # specific if email is False -> could have troubles finding it if several falsy traces
-                elif not email and state in ('ignored', 'canceled', 'bounced'):
-                    self.assertMailMailWId(recipient_trace.mail_mail_id_int, state_mapping[state], content=content, fields_values=fields_values)
+                elif not email and status in ('cancel', 'bounce'):
+                    self.assertMailMailWId(recipient_trace.mail_mail_id_int, state_mapping[status], content=content, fields_values=fields_values)
                 else:
-                    self.assertMailMailWEmails([email], state_mapping[state], author=author, content=content, fields_values=fields_values)
+                    self.assertMailMailWEmails([email], state_mapping[status], author=author, content=content, fields_values=fields_values)
 
             if link_info:
                 trace_mail = self._find_mail_mail_wrecord(record)
@@ -199,15 +205,18 @@ class MassMailCase(MailCase, MockLinkTracker):
         if dt is None:
             dt = datetime.datetime.now() - datetime.timedelta(days=1)
         randomized = random.random()
-        trace = cls.env['mailing.trace'].create({
-            'mass_mailing_id': mailing.id,
-            'model': record._name,
-            'res_id': record.id,
-            'bounced': dt,
-            # TDE FIXME: improve this with a mail-enabled heuristics
-            'email': trace_email,
-            'message_id': '<%5f@gilbert.boitempomils>' % randomized,
-        })
+        # Cursor.now() uses transaction's timestamp and not datetime lib -> freeze_time
+        # is not sufficient
+        with patch.object(Cursor, 'now', lambda *args, **kwargs: dt):
+            trace = cls.env['mailing.trace'].sudo().create({
+                'mass_mailing_id': mailing.id,
+                'model': record._name,
+                'res_id': record.id,
+                'trace_status': 'bounce',
+                # TDE FIXME: improve this with a mail-enabled heuristics
+                'email': trace_email,
+                'message_id': '<%5f@gilbert.boitempomils>' % randomized,
+            })
         return trace
 
 

addons/mass_mailing/tests/test_mailing_internals.py

@@ -276,7 +276,7 @@ class TestMassMailFeatures(MassMailCommon, CronMixinCase):
 
         self.assertMailTraces(
             [{'partner': partner_a},
-             {'partner': partner_b, 'state': 'ignored'}],
+             {'partner': partner_b, 'trace_status': 'cancel', 'failure_type': 'mail_bl'}],
             mailing, partner_a + partner_b, check_mail=True
         )
 

addons/mass_mailing/views/mailing_mailing_views.xml

@@ -66,14 +66,14 @@
                         <field name="state" readonly="1" widget="statusbar"/>
                     </header>
                     <div class="alert alert-info text-center" role="alert"
-                        attrs="{'invisible': ['&amp;','&amp;','&amp;','&amp;','&amp;',('state', '!=', 'in_queue'),('sent', '=', 0),('ignored', '=', 0),('scheduled', '=', 0),('failed', '=', 0),('warning_message', '=', False)]}">
-                        <div class="o_mails_ignored" attrs="{'invisible': [('ignored', '=', 0)]}">
+                        attrs="{'invisible': ['&amp;','&amp;','&amp;','&amp;','&amp;',('state', '!=', 'in_queue'),('sent', '=', 0),('canceled', '=', 0),('scheduled', '=', 0),('failed', '=', 0),('warning_message', '=', False)]}">
+                        <div class="o_mails_canceled" attrs="{'invisible': [('canceled', '=', 0)]}">
                             <button class="btn-link py-0"
-                                    name="action_view_traces_ignored"
+                                    name="action_view_traces_canceled"
                                     type="object">
                                 <strong>
-                                    <field name="ignored" class="oe_inline mr-2"/>
-                                    <span name="ignored_text">emails have been ignored and will not be sent.</span>
+                                    <field name="canceled" class="oe_inline mr-2"/>
+                                    <span name="canceled_text">emails have been canceled and will not be sent.</span>
                                 </strong>
                             </button>
                         </div>
@@ -339,7 +339,7 @@
                 <xpath expr="//div[hasclass('alert-info')]" position="attributes">
                     <attribute name="attrs">{'invisible': ['|',('state', '=', 'draft'),'&amp;','&amp;',('state', '!=', 'in_queue'),('scheduled', '=', 0),('failed', '=', 0)]}</attribute>
                 </xpath>
-                <xpath expr="//div[hasclass('alert-info')]/div[hasclass('o_mails_ignored')]" position="replace"/>
+                <xpath expr="//div[hasclass('alert-info')]/div[hasclass('o_mails_canceled')]" position="replace"/>
                 <xpath expr="//div[hasclass('alert-info')]/div[hasclass('o_mails_sent')]" position="replace"/>
                 <xpath expr="//div[hasclass('alert-info')]/div[hasclass('o_mails_failed')]//span[@name='failed_text']" position="replace">
                     <span name="failed_text">email(s) not sent.</span>
@@ -374,10 +374,10 @@
                         type="object" class="oe_stat_button" icon="fa-paper-plane">
                         <field name="sent" widget="statinfo" string="Sent"/>
                     </button>
-                    <button name="action_view_traces_ignored"
+                    <button name="action_view_traces_canceled"
                         attrs="{'invisible': [('state', 'in', ('draft','test'))]}"
                         type="object" class="oe_stat_button" icon="fa-paper-plane-o">
-                        <field name="ignored" widget="statinfo" string="Ignored"/>
+                        <field name="canceled" widget="statinfo" string="Ignored"/>
                     </button>
                 </xpath>
                 <xpath expr="//notebook" position="inside">

addons/mass_mailing/views/mailing_trace_views.xml

@@ -2,7 +2,7 @@
 <odoo>
     <!--  MAILING TRACE !-->
     <record model="ir.ui.view" id="mailing_trace_view_search">
-        <field name="name">mailing.trace.search</field>
+        <field name="name">mailing.trace.view.search</field>
         <field name="model">mailing.trace</field>
         <field name="arch" type="xml">
            <search string="Mail Statistics">
@@ -10,56 +10,72 @@
                 <field name="message_id"/>
                 <field name="email"/>
                 <field name="mass_mailing_id"/>
-                <filter string="Scheduled" name="filter_scheduled" domain="[('scheduled', '!=', False), ('sent', '=', False), ('exception', '=', False), ('ignored', '=', False), ('bounced', '=', False)]"/>
-                <filter string="Ignored" name="filter_ignored" domain="[('scheduled', '!=', False), ('sent', '=', False), ('exception', '=', False), ('ignored', '!=', False)]"/>
-                <filter string="Sent" name="filter_sent" domain="[('sent', '!=', False)]"/>
-                <filter string="Delivered" name="filter_delivered" domain="[('sent', '!=', False), ('exception', '=', False), ('bounced', '=', False)]"/>
-                <separator/>
-                <filter string="Opened" name="filter_opened" domain="[('opened', '!=', False)]"/>
-                <filter string="Clicked" name="filter_clicked" domain="[('clicked', '!=', False)]"/>
-                <filter string="Replied" name="filter_replied" domain="[('replied', '!=', False)]"/>
-                <filter string="Bounced" name="filter_bounced" domain="[('bounced', '!=', False)]"/>
-                <filter string="Failed" name="filter_failed" domain="[('exception', '!=', False)]"/>
+                <filter string="Scheduled" name="filter_scheduled" domain="[('trace_status', '=', 'outgoing')]"/>
+                <filter string="Canceled" name="filter_canceled" domain="[('trace_status', '=', 'cancel')]"/>
+                <filter string="Sent" name="filter_sent" domain="[('sent_datetime', '!=', False)]"/>
+                <filter string="Clicked" name="filter_clicked" domain="[('links_click_datetime', '!=', False)]"/>
+                <filter string="Delivered" name="filter_delivered" domain="[('sent_datetime', '!=', False), ('trace_status', 'not in', ['error', 'cancel'])]"/>
+                <filter string="Opened" name="filter_opened" domain="[('trace_status', 'in', ['open', 'reply'])]"/>
+                <filter string="Replied" name="filter_replied" domain="[('trace_status', '=', 'reply')]"/>
+                <filter string="Bounced" name="filter_bounced" domain="[('trace_status', '=', 'bounce')]"/>
+                <filter string="Failed" name="filter_failed" domain="[('trace_status', '=', 'error')]"/>
                 <group expand="0" string="Group By">
-                    <filter string="State" name="state" domain="[]" context="{'group_by':'state'}"/>
-                    <filter string="Open Date" name="group_open_date" context="{'group_by': 'opened:day'}"/>
-                    <filter string="Reply Date" name="group_reply_date" context="{'group_by': 'replied:day'}"/>
-                    <filter string="Last State Update" name="state_update" domain="[]" context="{'group_by':'state_update'}"/>
-                    <filter string="Mass Mailing"  name="mass_mailing" domain="[]" context="{'group_by':'mass_mailing_id'}"/>
+                    <filter string="State" name="state" domain="[]" context="{'group_by': 'trace_status'}"/>
+                    <filter string="Open Date" name="group_open_date" domain="[('trace_status', 'in', ['open', 'reply'])]" context="{'group_by': 'open_datetime:day'}"/>
+                    <filter string="Reply Date" name="group_reply_date" domain="[('trace_status', '=', 'reply')]" context="{'group_by': 'reply_datetime:day'}"/>
+                    <filter string="Last State Update" name="state_update" domain="[]" context="{'group_by': 'write_date'}"/>
+                    <filter string="Mass Mailing" name="mass_mailing" domain="[]" context="{'group_by': 'mass_mailing_id'}"/>
                 </group>
             </search>
         </field>
     </record>
 
-    <record model="ir.ui.view" id="mailing_trace_view_tree">
-        <field name="name">mailing.trace.tree</field>
+    <record id="mailing_trace_view_tree" model="ir.ui.view">
+        <field name="name">mailing.trace.view.tree</field>
         <field name="model">mailing.trace</field>
         <field name="arch" type="xml">
-            <tree string="Mail Statistics" create="0">
+            <tree string="Mailing Traces" create="0">
                 <field name="mass_mailing_id"/>
                 <field name="email"/>
-                <field name="state"/>
                 <field name="message_id"/>
-                <field name="scheduled"/>
-                <field name="sent"/>
-                <field name="exception"/>
-                <field name="opened"/>
-                <field name="clicked"/>
-                <field name="replied"/>
-                <field name="bounced"/>
-                <field name="ignored"/>
+                <field name="sent_datetime"/>
+                <field name="links_click_datetime"/>
+                <field name="trace_status" widget="badge"/>
+                <field name="failure_type" optional="show"/>
+                <field name="open_datetime" optional="hide"/>
+                <field name="reply_datetime" optional="hide"/>
+                <button name="action_view_contact" type="object"
+                        string="Open Recipient" icon="fa-user"/>
             </tree>
         </field>
     </record>
 
-    <record model="ir.ui.view" id="mailing_trace_view_form">
-        <field name="name">mailing.trace.form</field>
+    <record id="mailing_trace_view_tree_mail" model="ir.ui.view">
+        <field name="name">mailing.trace.view.tree.mail</field>
+        <field name="model">mailing.trace</field>
+        <field name="priority">20</field>
+        <field name="arch" type="xml">
+            <tree string="Mail Traces" create="0">
+                <field name="mass_mailing_id"/>
+                <field name="email"/>
+                <field name="message_id" optional="hide"/>
+                <field name="sent_datetime"/>
+                <field name="links_click_datetime"/>
+                <field name="trace_status" widget="badge"/>
+                <field name="failure_type" optional="show"/>
+                <field name="open_datetime" optional="hide"/>
+                <field name="reply_datetime" optional="hide"/>
+                <button name="action_view_contact" type="object"
+                        string="Open Recipient" icon="fa-user"/>
+            </tree>
+        </field>
+    </record>
+
+    <record id="mailing_trace_view_form" model="ir.ui.view">
+        <field name="name">mailing.trace.view.form</field>
         <field name="model">mailing.trace</field>
         <field name="arch" type="xml">
             <form string="Mail Statistics" create="0" edit="0">
-                <header>
-                    <field name="state" widget="statusbar"/>
-                </header>
                 <sheet>
                     <div class="oe_button_box" name="button_box">
                         <button name="action_view_contact"
@@ -68,29 +84,25 @@
                         </button>
                     </div>
                     <group>
+                        <group string="Status">
+                            <field name="trace_status"/>
+                            <field name="failure_type" attrs="{'invisible' : [('failure_type', '=', False)]}"/>
+                            <field name="sent_datetime" attrs="{'invisible' : [('sent_datetime', '=', False)]}"/>
+                            <field name="links_click_datetime" attrs="{'invisible' : [('links_click_datetime', '=', False)]}"/>
+                            <field name="open_datetime" attrs="{'invisible' : [('open_datetime', '=', False)]}"/>
+                            <field name="reply_datetime" attrs="{'invisible' : [('reply_datetime', '=', False)]}"/>
+                        </group>
                         <group string="Mailing">
                             <field name="trace_type" invisible="1"/>
-                            <field name="email" string="Recipient Address" readonly="1"/>
-                            <field name="sent" attrs="{'invisible' : [('sent', '=', False)]}" readonly="1"/>
-                            <field name="opened" attrs="{'invisible' : [('opened', '=', False)]}" readonly="1"/>
-                            <field name="replied" attrs="{'invisible' : [('replied', '=', False)]}" readonly="1"/>
-                            <field name="bounced" attrs="{'invisible' : [('bounced', '=', False)]}" readonly="1"/>
-                            <field name="ignored" attrs="{'invisible' : [('ignored', '=', False)]}" readonly="1"/>
-                            <field name="failure_type" attrs="{'invisible' : [('failure_type', '=', False)]}" readonly="1"/>
+                            <field name="email" string="Recipient Address"/>
+                            <field name="mass_mailing_id"/>
+                            <field name="mail_mail_id_int" string="Mail ID" groups="base.group_no_one"/>
+                            <field name="message_id" groups="base.group_no_one"/>
                         </group>
-
                         <group string="Marketing">
-                            <field name="campaign_id" readonly="1"/>
-                            <field name="medium_id" readonly="1"/>
-                            <field name="source_id" readonly="1"/>
-                            <field name="mass_mailing_id" readonly="1"/>
-                        </group>
-
-                        <group string="Technical">
-                            <field name="exception" readonly="1"/>
-                            <field name="state_update" readonly="1"/>
-                            <field name="mail_mail_id_int" string="Mail ID" readonly="1"/>
-                            <field name="message_id" readonly="1"/>
+                            <field name="campaign_id" groups="mass_mailing.group_mass_mailing_campaign"/>
+                            <field name="medium_id"/>
+                            <field name="source_id"/>
                         </group>
                     </group>
                 </sheet>
@@ -103,8 +115,8 @@
         <field name="model">mailing.trace</field>
         <field name="arch" type="xml">
             <graph string="Mail Statistics" type="bar" stacked="True" sample="1">
-                <field name="state_update" type="row" interval="day"/>
-                <field name="state" type="row"/>
+                <field name="write_date" type="row" interval="day"/>
+                <field name="trace_status" type="row"/>
             </graph>
         </field>
     </record>

addons/mass_mailing/wizard/mail_compose_message.py

@@ -1,8 +1,7 @@
 # -*- coding: utf-8 -*-
 # Part of Odoo. See LICENSE file for full copyright and licensing details.
 
-from odoo import api, fields, models, tools
-from odoo.tools import email_re
+from odoo import fields, models
 
 
 class MailComposeMessage(models.TransientModel):
@@ -17,6 +16,7 @@ class MailComposeMessage(models.TransientModel):
         """ Override method that generated the mail content by creating the
         mailing.trace values in the o2m of mail_mail, when doing pure
         email mass mailing. """
+        now = fields.Datetime.now()
         self.ensure_one()
         res = super(MailComposeMessage, self).get_mail_values(res_ids)
         # use only for allowed models in mass mailing
@@ -32,60 +32,53 @@ class MailComposeMessage(models.TransientModel):
                     'state': 'done',
                     'reply_to_mode': self.reply_to_mode,
                     'reply_to': self.reply_to if self.reply_to_mode == 'new' else False,
-                    'sent_date': fields.Datetime.now(),
+                    'sent_date': now,
                     'body_html': self.body,
                     'mailing_model_id': self.env['ir.model']._get(self.model).id,
                     'mailing_domain': self.active_domain,
                     'attachment_ids': [(6, 0, self.attachment_ids.ids)],
                 })
+                self.mass_mailing_id = mass_mailing.id
 
-            # Preprocess res.partners to batch-fetch from db
-            # if recipient_ids is present, it means they are partners
-            # (the only object to fill get_default_recipient this way)
-            recipient_partners_ids = []
-            read_partners = {}
-            for res_id in res_ids:
-                mail_values = res[res_id]
-                if mail_values.get('recipient_ids'):
-                    # recipient_ids is a list of x2m command tuples at this point
-                    recipient_partners_ids.append(mail_values.get('recipient_ids')[0][1])
-            read_partners = self.env['res.partner'].browse(recipient_partners_ids)
-
-            partners_email = {p.id: p.email for p in read_partners}
-
-            opt_out_list = self._context.get('mass_mailing_opt_out_list')
-            seen_list = self._context.get('mass_mailing_seen_list')
+            recipients_info = self._process_recipient_values(res)
             mass_mail_layout = self.env.ref('mass_mailing.mass_mailing_mail_layout', raise_if_not_found=False)
             for res_id in res_ids:
                 mail_values = res[res_id]
-                if mail_values.get('email_to'):
-                    mail_to = tools.email_normalize(mail_values['email_to'])
-                else:
-                    partner_id = (mail_values.get('recipient_ids') or [(False, '')])[0][1]
-                    mail_to = tools.email_normalize(partners_email.get(partner_id))
-                if (opt_out_list and mail_to in opt_out_list) or (seen_list and mail_to in seen_list) \
-                        or (not mail_to or not email_re.findall(mail_to)):
-                    # prevent sending to blocked addresses that were included by mistake
-                    mail_values['state'] = 'cancel'
-                elif seen_list is not None:
-                    seen_list.add(mail_to)
+                if mail_values.get('body_html') and mass_mail_layout:
+                    mail_values['body_html'] = mass_mail_layout._render({'body': mail_values['body_html']}, engine='ir.qweb', minimal_qcontext=True)
+
                 trace_vals = {
                     'model': self.model,
                     'res_id': res_id,
                     'mass_mailing_id': mass_mailing.id,
                     # if mail_to is void, keep falsy values to allow searching / debugging traces
-                    'email': mail_to or mail_values.get('email_to'),
+                    'email': recipients_info[res_id]['mail_to'][0] if recipients_info[res_id]['mail_to'] else '',
                 }
-                if mail_values.get('body_html') and mass_mail_layout:
-                    mail_values['body_html'] = mass_mail_layout._render({'body': mail_values['body_html']}, engine='ir.qweb', minimal_qcontext=True)
-                # propagate ignored state to trace when still-born
+                # propagate failed states to trace when still-born
                 if mail_values.get('state') == 'cancel':
-                    trace_vals['ignored'] = fields.Datetime.now()
+                    trace_vals['trace_status'] = 'cancel'
+                elif mail_values.get('state') == 'exception':
+                    trace_vals['trace_status'] = 'error'
+                if mail_values.get('failure_type'):
+                    trace_vals['failure_type'] = mail_values['failure_type']
+
                 mail_values.update({
                     'mailing_id': mass_mailing.id,
                     'mailing_trace_ids': [(0, 0, trace_vals)],
                     # email-mode: keep original message for routing
-                    'notification': mass_mailing.reply_to_mode == 'update',
+                    'is_notification': mass_mailing.reply_to_mode == 'update',
                     'auto_delete': not mass_mailing.keep_archives,
                 })
         return res
+
+    def _get_done_emails(self, mail_values_dict):
+        seen_list = super(MailComposeMessage, self)._get_done_emails(mail_values_dict)
+        if self.mass_mailing_id:
+            seen_list += self.mass_mailing_id._get_seen_list()
+        return seen_list
+
+    def _get_optout_emails(self, mail_values_dict):
+        opt_out_list = super(MailComposeMessage, self)._get_optout_emails(mail_values_dict)
+        if self.mass_mailing_id:
+            opt_out_list += self.mass_mailing_id._get_opt_out_list()
+        return opt_out_list

addons/mass_mailing/wizard/mailing_mailing_test.py

@@ -56,7 +56,7 @@ class TestMassMailing(models.TransientModel):
                 'email_to': valid_email,
                 'subject': subject,
                 'body_html': mass_mail_layout._render({'body': full_body}, engine='ir.qweb', minimal_qcontext=True),
-                'notification': True,
+                'is_notification': True,
                 'mailing_id': mailing.id,
                 'attachment_ids': [(4, attachment.id) for attachment in mailing.attachment_ids],
                 'auto_delete': False,  # they are manually deleted after notifying the document

addons/mass_mailing_sms/data/mailing_demo.xml

@@ -21,7 +21,9 @@
         <field name="sms_number">+32465000000</field>
         <field name="model">res.partner</field>
         <field name="res_id" ref="base.res_partner_address_7"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">sent</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="write_date" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_0_trace_1" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_0"/>
@@ -29,7 +31,9 @@
         <field name="sms_number">+32465000001</field>
         <field name="model">res.partner</field>
         <field name="res_id" ref="base.res_partner_address_13"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">sent</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="write_date" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_0_trace_2" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_0"/>
@@ -37,7 +41,9 @@
         <field name="sms_number">+32465000002</field>
         <field name="model">res.partner</field>
         <field name="res_id" ref="base.res_partner_address_14"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">sent</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="write_date" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_0_trace_3" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_0"/>
@@ -45,7 +51,9 @@
         <field name="sms_number">+32465000003</field>
         <field name="model">res.partner</field>
         <field name="res_id" ref="base.res_partner_address_24"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">sent</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="write_date" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_0_trace_4" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_0"/>
@@ -53,8 +61,10 @@
         <field name="sms_number">+32465000004</field>
         <field name="model">res.partner</field>
         <field name="res_id" ref="base.res_partner_address_33"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
-        <field name="exception" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">error</field>
+        <field name="failure_type">sms_server</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="write_date" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_0_trace_5" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_0"/>
@@ -62,8 +72,9 @@
         <field name="sms_number">123123</field>
         <field name="model">res.partner</field>
         <field name="res_id" ref="base.res_partner_address_33"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
-        <field name="bounced" eval="(DateTime.today() - relativedelta(days=1)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">bounce</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="write_date" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_0_trace_6" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_0"/>
@@ -71,8 +82,9 @@
         <field name="sms_number"></field>
         <field name="model">res.partner</field>
         <field name="res_id" ref="base.res_partner_address_34"/>
-        <field name="sent" eval="False"/>
-        <field name="exception" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">error</field>
+        <field name="failure_type">sms_number_missing</field>
+        <field name="sent_datetime" eval="False"/>
     </record>
 
     <!-- Generate some clicks -->
@@ -128,7 +140,8 @@
         <field name="sms_number">+32456001100</field>
         <field name="model">mailing.contact</field>
         <field name="res_id" ref="mass_mailing_sms.mailing_contact_0_0"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">sent</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_1_trace_1" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_1"/>
@@ -136,7 +149,8 @@
         <field name="sms_number">+32456001111</field>
         <field name="model">mailing.contact</field>
         <field name="res_id" ref="mass_mailing_sms.mailing_contact_0_1"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">sent</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_1_trace_2" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_1"/>
@@ -144,7 +158,8 @@
         <field name="sms_number">+32456001122</field>
         <field name="model">mailing.contact</field>
         <field name="res_id" ref="mass_mailing_sms.mailing_contact_0_2"/>
-        <field name="sent" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">sent</field>
+        <field name="sent_datetime" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
     </record>
     <record id="mailing_sms_1_trace_3" model="mailing.trace">
         <field name="mass_mailing_id" ref="mailing_sms_1"/>
@@ -152,7 +167,7 @@
         <field name="sms_number">+32456001133</field>
         <field name="model">mailing.contact</field>
         <field name="res_id" ref="mass_mailing_sms.mailing_contact_0_3"/>
-        <field name="exception" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">error</field>
         <field name="failure_type">sms_credit</field>
     </record>
     <record id="mailing_sms_1_trace_4" model="mailing.trace">
@@ -161,7 +176,7 @@
         <field name="sms_number">dummy</field>
         <field name="model">mailing.contact</field>
         <field name="res_id" ref="mass_mailing_sms.mailing_contact_0_4"/>
-        <field name="exception" eval="(DateTime.today() - relativedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')"/>
+        <field name="trace_status">error</field>
         <field name="failure_type">sms_number_format</field>
     </record>
 

addons/mass_mailing_sms/models/mailing_trace.py

@@ -36,6 +36,7 @@ class MailingTrace(models.Model):
         # mass mode specific codes
         ('sms_blacklist', 'Blacklisted'),
         ('sms_duplicate', 'Duplicate'),
+        ('sms_optout', 'Opted Out'),
     ])
 
     @api.model_create_multi
@@ -52,36 +53,3 @@ class MailingTrace(models.Model):
         as it serves as obfuscation when unsubscribing. A valid trio
         code / mailing_id / number will be requested. """
         return ''.join(random.choice(string.ascii_letters + string.digits) for dummy in range(self.CODE_SIZE))
-
-    def _get_records_from_sms(self, sms_sms_ids=None, additional_domain=None):
-        if not self.ids and sms_sms_ids:
-            domain = [('sms_sms_id_int', 'in', sms_sms_ids)]
-        else:
-            domain = [('id', 'in', self.ids)]
-        if additional_domain:
-            domain = expression.AND([domain, additional_domain])
-        return self.search(domain)
-
-    def set_failed(self, failure_type):
-        for trace in self:
-            trace.write({'exception': fields.Datetime.now(), 'failure_type': failure_type})
-
-    def set_sms_sent(self, sms_sms_ids=None):
-        statistics = self._get_records_from_sms(sms_sms_ids, [('sent', '=', False)])
-        statistics.write({'sent': fields.Datetime.now()})
-        return statistics
-
-    def set_sms_clicked(self, sms_sms_ids=None):
-        statistics = self._get_records_from_sms(sms_sms_ids, [('clicked', '=', False)])
-        statistics.write({'clicked': fields.Datetime.now()})
-        return statistics
-
-    def set_sms_ignored(self, sms_sms_ids=None):
-        statistics = self._get_records_from_sms(sms_sms_ids, [('ignored', '=', False)])
-        statistics.write({'ignored': fields.Datetime.now()})
-        return statistics
-
-    def set_sms_exception(self, sms_sms_ids=None):
-        statistics = self._get_records_from_sms(sms_sms_ids, [('exception', '=', False)])
-        statistics.write({'exception': fields.Datetime.now()})
-        return statistics

addons/mass_mailing_sms/models/sms_sms.py

@@ -37,7 +37,7 @@ class SmsSms(models.Model):
                     ('sms_sms_id_int', 'in', sms_ids)
                 ])
                 if traces and state == 'success':
-                    traces.write({'sent': fields.Datetime.now(), 'exception': False})
+                    traces.set_sent()
                 elif traces:
                     traces.set_failed(failure_type=self.IAP_TO_SMS_STATE[state])
         return super(SmsSms, self)._postprocess_iap_sent_sms(iap_results, failure_reason=failure_reason, delete_all=delete_all)

addons/mass_mailing_sms/report/mailing_trace_report_views.xml

@@ -1,5 +1,22 @@
 <?xml version="1.0" encoding="utf-8"?>
 <odoo>
+    <record id="mailing_trace_report_sms_view_tree" model="ir.ui.view">
+        <field name="name">mailing.sms.trace.report.view.tree</field>
+        <field name="model">mailing.trace.report</field>
+        <field name="inherit_id" ref="mass_mailing.mailing_trace_report_view_tree"/>
+        <field name="priority" eval="50"/>
+        <field name="mode">primary</field>
+        <field name="type">tree</field>
+        <field name="arch" type="xml">
+            <xpath expr="//field[@name='opened']" position="attributes">
+                <attribute name="invisible">1</attribute>
+            </xpath>
+            <xpath expr="//field[@name='replied']" position="attributes">
+                <attribute name="invisible">1</attribute>
+            </xpath>
+        </field>
+    </record>
+
     <record id="mailing_trace_report_sms_view_pivot" model="ir.ui.view">
         <field name="name">mailing.sms.trace.report.view.pivot</field>
         <field name="model">mailing.trace.report</field>
@@ -8,6 +25,9 @@
         <field name="mode">primary</field>
         <field name="type">pivot</field>
         <field name="arch" type="xml">
+            <xpath expr="//field[@name='opened']" position="attributes">
+                <attribute name="invisible">1</attribute>
+            </xpath>
             <xpath expr="//field[@name='replied']" position="attributes">
                 <attribute name="invisible">1</attribute>
             </xpath>
@@ -32,7 +52,7 @@
     <record id="mailing_trace_report_action_sms" model="ir.actions.act_window">
         <field name="name">SMS Marketing Analysis</field>
         <field name="res_model">mailing.trace.report</field>
-        <field name="view_mode">graph,pivot</field>
+        <field name="view_mode">graph,pivot,tree</field>
         <field name="domain">[('mailing_type', '=', 'sms')]</field>
         <field name="help" type="html">
             <p class="o_view_nocontent_smiling_face">
@@ -56,6 +76,13 @@
         <field name="view_id" ref="mailing_trace_report_sms_view_pivot"/>
     </record>
 
+    <record id="mailing_trace_report_action_sms_view_tree" model="ir.actions.act_window.view">
+        <field name="sequence">2</field>
+        <field name="view_mode">tree</field>
+        <field name="act_window_id" ref="mailing_trace_report_action_sms"/>
+        <field name="view_id" ref="mailing_trace_report_sms_view_tree"/>
+    </record>
+
     <!-- SMS Marketing / Reporting -->
     <record id="mass_mailing_sms_menu_reporting" model="ir.ui.menu">
         <field name="action" ref="mailing_trace_report_action_sms"/>

addons/mass_mailing_sms/tests/common.py

@@ -31,7 +31,7 @@ class MassSMSCase(SMSCase):
           # TRACE
           'partner': res.partner record (may be empty),
           'number': number used for notification (may be empty, computed based on partner),
-          'state': outgoing / sent / ignored / bounced / exception / opened (outgoing by default),
+          'trace_status': outgoing / sent / cancel / bounce / error / opened (sent by default),
           'record: linked record,
           # SMS.SMS
           'content': optional: if set, check content of sent SMS;
@@ -55,9 +55,9 @@ class MassSMSCase(SMSCase):
         state_mapping = {
             'sent': 'sent',
             'outgoing': 'outgoing',
-            'exception': 'error',
-            'ignored': 'canceled',
-            'bounced': 'error',
+            'error': 'error',
+            'cancel': 'canceled',
+            'bounce': 'error',
         }
         traces = self.env['mailing.trace'].search([
             ('mass_mailing_id', 'in', mailing.ids),
@@ -74,28 +74,28 @@ class MassSMSCase(SMSCase):
         for recipient_info, link_info, record in zip(recipients_info, sms_links_info, records):
             partner = recipient_info.get('partner', self.env['res.partner'])
             number = recipient_info.get('number')
-            state = recipient_info.get('state', 'outgoing')
+            status = recipient_info.get('trace_status', 'outgoing')
             content = recipient_info.get('content', None)
             if number is None and partner:
                 number = partner._sms_get_recipients_info()[partner.id]['sanitized']
 
             trace = traces.filtered(
-                lambda t: t.sms_number == number and t.state == state and (t.res_id == record.id if record else True)
+                lambda t: t.sms_number == number and t.trace_status == status and (t.res_id == record.id if record else True)
             )
             self.assertTrue(len(trace) == 1,
-                            'SMS: found %s notification for number %s, (state: %s) (1 expected)' % (len(trace), number, state))
+                            'SMS: found %s notification for number %s, (status: %s) (1 expected)' % (len(trace), number, status))
             self.assertTrue(bool(trace.sms_sms_id_int))
 
             if check_sms:
-                if state == 'sent':
+                if status == 'sent':
                     if sent_unlink:
                         self.assertSMSIapSent([number], content=content)
                     else:
                         self.assertSMS(partner, number, 'sent', content=content)
-                elif state in state_mapping:
-                    sms_state = state_mapping[state]
-                    error_code = recipient_info['failure_type'] if state in ('exception', 'ignored', 'bounced') else None
-                    self.assertSMS(partner, number, sms_state, error_code=error_code, content=content)
+                elif status in state_mapping:
+                    sms_state = state_mapping[status]
+                    failure_type = recipient_info['failure_type'] if status in ('error', 'cancel', 'bounce') else None
+                    self.assertSMS(partner, number, sms_state, failure_type=failure_type, content=content)
                 else:
                     raise NotImplementedError()
 
@@ -103,7 +103,7 @@ class MassSMSCase(SMSCase):
                 # shortened links are directly included in sms.sms record as well as
                 # in sent sms (not like mails who are post-processed)
                 sms_sent = self._find_sms_sent(partner, number)
-                sms_sms = self._find_sms_sms(partner, number, state_mapping[state])
+                sms_sms = self._find_sms_sms(partner, number, state_mapping[status])
                 for (url, is_shortened, add_link_params) in link_info:
                     if url == 'unsubscribe':
                         url = '%s/sms/%d/%s' % (mailing.get_base_url(), mailing.id, trace.sms_code)

addons/mass_mailing_sms/views/mailing_mailing_views.xml

@@ -61,11 +61,11 @@
                     </button>
                 </div>
             </xpath>
-            <xpath expr="//span[@name='ignored_text']" position="attributes">
+            <xpath expr="//span[@name='canceled_text']" position="attributes">
                 <attribute name="attrs">{'invisible': [('mailing_type', '!=', 'mail')]}</attribute>
             </xpath>
-            <xpath expr="//span[@name='ignored_text']" position="after">
-                <span name="ignored_text_sms" attrs="{'invisible': [('mailing_type', '!=', 'sms')]}">SMS Text Message have been ignored and will not be sent.</span>
+            <xpath expr="//span[@name='canceled_text']" position="after">
+                <span name="canceled_text_sms" attrs="{'invisible': [('mailing_type', '!=', 'sms')]}">SMS Text Message have been canceled and will not be sent.</span>
             </xpath>
             <xpath expr="//span[@name='scheduled_text']" position="attributes">
                 <attribute name="attrs">{'invisible': [('mailing_type', '!=', 'mail')]}</attribute>

addons/mass_mailing_sms/views/mailing_trace_views.xml

@@ -33,15 +33,16 @@
         <field name="priority">20</field>
         <field name="arch" type="xml">
             <tree string="SMS Traces" create="0">
+                <field name="mass_mailing_id"/>
                 <field name="sms_number"/>
-                <field name="state"/>
-                <field name="scheduled"/>
-                <field name="sent"/>
-                <field name="clicked"/>
-                <field name="bounced"/>
-                <field name="exception"/>
-                <field name="ignored"/>
-                <field name="failure_type"/>
+                <field name="sent_datetime"/>
+                <field name="links_click_datetime"/>
+                <field name="trace_status" widget="badge"/>
+                <field name="failure_type" optional="show"/>
+                <field name="open_datetime" optional="hide"/>
+                <field name="reply_datetime" optional="hide"/>
+                <button name="action_view_contact" type="object"
+                        string="Open Recipient" icon="fa-user"/>
             </tree>
         </field>
     </record>
@@ -60,22 +61,15 @@
             <xpath expr="//field[@name='message_id']" position="attributes">
                 <attribute name="attrs">{'invisible': [('trace_type', '!=', 'mail')]}</attribute>
             </xpath>
-            <xpath expr="//field[@name='opened']" position="attributes">
-                <attribute name="attrs">
-                    {'invisible': ['|', ('opened', '=', False), ('trace_type', '=', 'sms')]}
-                </attribute>
-            </xpath>
-            <xpath expr="//field[@name='replied']" position="attributes">
-                <attribute name="attrs">
-                    {'invisible': ['|', ('replied', '=', False), ('trace_type', '=', 'sms')]}
-                </attribute>
-            </xpath>
             <xpath expr="//field[@name='email']" position="after">
                 <field name="sms_number" attrs="{'invisible': [('trace_type', '!=', 'sms')]}"/>
             </xpath>
-            <xpath expr="//field[@name='mail_mail_id_int']" position="after">
-                <field name="sms_sms_id_int" string="SMS ID" attrs="{'invisible': [('trace_type', '!=', 'sms')]}"/>
-                <field name="sms_code" attrs="{'invisible': [('trace_type', '!=', 'sms')]}"/>
+            <xpath expr="//field[@name='message_id']" position="after">
+                <field name="sms_sms_id_int" string="SMS ID"
+                    attrs="{'invisible': [('trace_type', '!=', 'sms')]}"
+                    groups="base.group_no_one"/>
+                <field name="sms_code" attrs="{'invisible': [('trace_type', '!=', 'sms')]}"
+                    groups="base.group_no_one"/>
             </xpath>
         </field>
     </record>
@@ -85,34 +79,41 @@
         <field name="model">mailing.trace</field>
         <field name="priority">20</field>
         <field name="arch" type="xml">
-            <form string="SMS Trace" create="0">
-                <header>
-                    <field name="state" widget="statusbar"/>
-                </header>
+            <form string="SMS Trace" create="0" edit="0">
                 <sheet>
-                    <div class="alert alert-info text-center" attrs="{'invisible': [('exception', '=', False)]}" role="alert">
+                    <div class="oe_button_box" name="button_box">
+                        <button name="action_view_contact"
+                                type="object" icon="fa-user" class="oe_stat_button">
+                            <span widget="statinfo">Open Recipient</span>
+                        </button>
+                    </div>
+                    <div class="alert alert-info text-center" attrs="{'invisible': [('trace_status', '!=', 'error')]}" role="alert">
                         <strong>This SMS could not be sent.</strong>
                     </div>
-                    <div class="alert alert-info text-center" attrs="{'invisible': [('bounced', '=', False)]}" role="alert">
+                    <div class="alert alert-info text-center" attrs="{'invisible': [('trace_status', '!=', 'bounce')]}" role="alert">
                         <strong>This number appears to be invalid.</strong>
                     </div>
                     <group>
-                        <group>
+                        <group string="Status">
+                            <field name="trace_status"/>
+                            <field name="failure_type" attrs="{'invisible' : [('failure_type', '=', False)]}"/>
+                            <field name="sent_datetime" attrs="{'invisible' : [('sent_datetime', '=', False)]}"/>
+                            <field name="links_click_datetime" attrs="{'invisible' : [('links_click_datetime', '=', False)]}"/>
+                            <field name="open_datetime" attrs="{'invisible' : [('open_datetime', '=', False)]}"/>
+                            <field name="reply_datetime" attrs="{'invisible' : [('reply_datetime', '=', False)]}"/>
+                        </group>
+                        <group string="Mailing">
+                            <field name="trace_type" invisible="1"/>
                             <field name="sms_number"/>
                             <field name="mass_mailing_id"/>
-                            <field name="campaign_id" groups="mass_mailing.group_mass_mailing_campaign"/>
-                            <field name="sms_sms_id_int" groups="base.group_no_one"/>
-                            <field name="model" groups="base.group_no_one"/>
-                            <field name="res_id" groups="base.group_no_one"/>
+                            <field name="sms_sms_id_int" string="SMS ID" groups="base.group_no_one"/>
+                            <field name="sms_code" groups="base.group_no_one"/>
                         </group>
-                        <group>
-                            <field name="scheduled"/>
-                            <field name="sent"/>
-                            <field name="clicked"/>
-                            <field name="bounced"/>
-                            <field name="exception"/>
-                            <field name="failure_type" attrs="{'invisible': [('exception', '=', False)]}"/>
-                            <field name="ignored"/>
+                        <group string="Marketing">
+                            <field name="campaign_id" groups="mass_mailing.group_mass_mailing_campaign"/>
+                            <field name="medium_id"/>
+                            <field name="source_id"/>
+                            <field name="sms_sms_id_int" groups="base.group_no_one"/>
                         </group>
                     </group>
                 </sheet>

addons/mass_mailing_sms/wizard/sms_composer.py

@@ -35,22 +35,19 @@ class SMSComposer(models.TransientModel):
             'sms_code': trace_code,
         }
         if sms_values['state'] == 'error':
-            if sms_values['error_code'] == 'sms_number_format':
-                trace_values['sent'] = fields.Datetime.now()
-                trace_values['bounced'] = fields.Datetime.now()
-            else:
-                trace_values['exception'] = fields.Datetime.now()
+            trace_values['failure_type'] = sms_values['failure_type']
+            trace_values['trace_status'] = 'error'
         elif sms_values['state'] == 'canceled':
-            trace_values['ignored'] = fields.Datetime.now()
+            trace_values['failure_type'] = sms_values['failure_type']
+            trace_values['trace_status'] = 'cancel'
         else:
             if self.mass_sms_allow_unsubscribe:
                 sms_values['body'] = '%s\n%s' % (sms_values['body'] or '', _('STOP SMS : %s', self._get_unsubscribe_url(record.id, trace_code, sms_values['number'])))
         return trace_values
 
-    def _get_blacklist_record_ids(self, records, recipients_info):
-        """ Consider opt-outed contact as being blacklisted for that specific
-        mailing. """
-        res = super(SMSComposer, self)._get_blacklist_record_ids(records, recipients_info)
+    def _get_optout_record_ids(self, records, recipients_info):
+        """ Fetch opt-out records based on mailing. """
+        res = super(SMSComposer, self)._get_optout_record_ids(records, recipients_info)
         if self.mailing_id:
             optout_res_ids = self.mailing_id._get_opt_out_list_sms()
             res += optout_res_ids

addons/purchase/models/mail_compose_message.py

@@ -7,7 +7,7 @@ from odoo import models
 class MailComposeMessage(models.TransientModel):
     _inherit = 'mail.compose.message'
 
-    def send_mail(self, auto_commit=False):
+    def _action_send_mail(self, auto_commit=False):
         if self.env.context.get('mark_rfq_as_sent') and self.model == 'purchase.order':
             self = self.with_context(mail_notify_author=self.env.user.partner_id in self.partner_ids)
-        return super(MailComposeMessage, self).send_mail(auto_commit=auto_commit)
+        return super(MailComposeMessage, self)._action_send_mail(auto_commit=auto_commit)

addons/sale/models/res_company.py

@@ -91,10 +91,10 @@ class ResCompany(models.Model):
             'composition_mode': 'comment'})
 
         # Simulate the onchange (like trigger in form the view)
-        update_values = message_composer.onchange_template_id(template.id, 'comment', 'sale.order', sample_sales_order.id)['value']
+        update_values = message_composer._onchange_template_id(template.id, 'comment', 'sale.order', sample_sales_order.id)['value']
         message_composer.write(update_values)
 
-        message_composer.send_mail()
+        message_composer._action_send_mail()
 
         self.set_onboarding_step_done('sale_onboarding_sample_quotation_state')
 

addons/sale/wizard/mail_compose_message.py

@@ -7,7 +7,7 @@ from odoo import models
 class MailComposeMessage(models.TransientModel):
     _inherit = 'mail.compose.message'
 
-    def send_mail(self, auto_commit=False):
+    def _action_send_mail(self, auto_commit=False):
         if self.env.context.get('mark_so_as_sent') and self.model == 'sale.order':
             self = self.with_context(mail_notify_author=self.env.user.partner_id in self.partner_ids)
-        return super(MailComposeMessage, self).send_mail(auto_commit=auto_commit)
+        return super(MailComposeMessage, self)._action_send_mail(auto_commit=auto_commit)

addons/sms/__manifest__.py

@@ -3,7 +3,7 @@
 
 {
     'name': 'SMS gateway',
-    'version': '2.1',
+    'version': '2.2',
     'category': 'Hidden/Tools',
     'summary': 'SMS Text Messaging',
     'description': """

addons/sms/data/mail_demo.xml

@@ -23,7 +23,7 @@
         <field name="res_partner_id" ref="base.res_partner_address_28"/>
         <field name="notification_type">email</field>
         <field name="notification_status">exception</field>
-        <field name="failure_type">SMTP</field>
+        <field name="failure_type">mail_smtp</field>
     </record>
 
     <record id="message_demo_partner_1_2" model="mail.message">

addons/sms/models/mail_thread.py

@@ -296,7 +296,7 @@ class MailThread(models.AbstractModel):
                 partner_id=False,
                 number=n,
                 state='outgoing' if n else 'error',
-                error_code='' if n else 'sms_number_missing',
+                failure_type='' if n else 'sms_number_missing',
             ) for n in tocreate_numbers]
 
         # create sms and notification
@@ -325,7 +325,7 @@ class MailThread(models.AbstractModel):
                 'sms_id': sms.id,
                 'is_read': True,  # discard Inbox notification
                 'notification_status': 'ready' if sms.state == 'outgoing' else 'exception',
-                'failure_type': '' if sms.state == 'outgoing' else sms.error_code,
+                'failure_type': '' if sms.state == 'outgoing' else sms.failure_type,
             } for sms in sms_all if (sms.partner_id and sms.partner_id.id not in existing_pids) or (not sms.partner_id and sms.number not in existing_numbers)]
             if notif_create_values:
                 self.env['mail.notification'].sudo().create(notif_create_values)

addons/sms/models/sms_sms.py

@@ -33,7 +33,7 @@ class SmsSms(models.Model):
         ('error', 'Error'),
         ('canceled', 'Canceled')
     ], 'SMS Status', readonly=True, copy=False, default='outgoing', required=True)
-    error_code = fields.Selection([
+    failure_type = fields.Selection([
         ('sms_number_missing', 'Missing Number'),
         ('sms_number_format', 'Wrong Number Format'),
         ('sms_credit', 'Insufficient Credit'),
@@ -42,6 +42,7 @@ class SmsSms(models.Model):
         # mass mode specific codes
         ('sms_blacklist', 'Blacklisted'),
         ('sms_duplicate', 'Duplicate'),
+        ('sms_optout', 'Opted Out'),
     ], copy=False)
 
     def send(self, delete_all=False, auto_commit=False, raise_exception=False):
@@ -153,7 +154,7 @@ class SmsSms(models.Model):
                 if state != 'success' and not delete_all:
                     self.env['sms.sms'].sudo().browse(sms_ids).write({
                         'state': 'error',
-                        'error_code': self.IAP_TO_SMS_STATE[state],
+                        'failure_type': self.IAP_TO_SMS_STATE[state],
                     })
                 notifications = self.env['mail.notification'].sudo().search([
                     ('notification_type', '=', 'sms'),

addons/sms/tests/common.py

@@ -135,7 +135,7 @@ class SMSCase(MockSMS):
         """ Deprecated. Remove in 14.4 """
         return self.assertSMSIapSent(numbers, content=content)
 
-    def assertSMS(self, partner, number, status, error_code=None,
+    def assertSMS(self, partner, number, status, failure_type=None,
                   content=None, fields_values=None):
         """ Find a ``sms.sms`` record, based on given partner, number and status.
 
@@ -143,14 +143,14 @@ class SMSCase(MockSMS):
           if not given;
         :param number: optional number, used to find a ``sms.sms``, notably if
           partner is not given;
-        :param error_code: check error code if SMS is not sent or outgoing;
+        :param failure_type: check failure type if SMS is not sent or outgoing;
         :param content: if given, should be contained in sms body;
         :param fields_values: optional values allowing to check directly some
           values on ``sms.sms`` record;
         """
         sms_sms = self._find_sms_sms(partner, number, status)
-        if error_code:
-            self.assertEqual(sms_sms.error_code, error_code)
+        if failure_type:
+            self.assertEqual(sms_sms.failure_type, failure_type)
         if content is not None:
             self.assertIn(content, sms_sms.body)
         for fname, fvalue in (fields_values or {}).items():
@@ -160,15 +160,15 @@ class SMSCase(MockSMS):
         if status == 'sent':
             self.assertSMSIapSent([sms_sms.number], content=content)
 
-    def assertSMSCanceled(self, partner, number, error_code, content=None, fields_values=None):
+    def assertSMSCanceled(self, partner, number, failure_type, content=None, fields_values=None):
         """ Check canceled SMS. Search is done for a pair partner / number where
         partner can be an empty recordset. """
-        self.assertSMS(partner, number, 'canceled', error_code=error_code, content=content, fields_values=fields_values)
+        self.assertSMS(partner, number, 'canceled', failure_type=failure_type, content=content, fields_values=fields_values)
 
-    def assertSMSFailed(self, partner, number, error_code, content=None, fields_values=None):
+    def assertSMSFailed(self, partner, number, failure_type, content=None, fields_values=None):
         """ Check failed SMS. Search is done for a pair partner / number where
         partner can be an empty recordset. """
-        self.assertSMS(partner, number, 'error', error_code=error_code, content=content, fields_values=fields_values)
+        self.assertSMS(partner, number, 'error', failure_type=failure_type, content=content, fields_values=fields_values)
 
     def assertSMSOutgoing(self, partner, number, content=None, fields_values=None):
         """ Check outgoing SMS. Search is done for a pair partner / number where
@@ -229,9 +229,9 @@ class SMSCase(MockSMS):
                 elif state == 'ready':
                     self.assertSMS(partner, number, 'outgoing', content=content)
                 elif state == 'exception':
-                    self.assertSMS(partner, number, 'error', error_code=recipient_info['failure_type'], content=content)
+                    self.assertSMS(partner, number, 'error', failure_type=recipient_info['failure_type'], content=content)
                 elif state == 'canceled':
-                    self.assertSMS(partner, number, 'canceled', error_code=recipient_info['failure_type'], content=content)
+                    self.assertSMS(partner, number, 'canceled', failure_type=recipient_info['failure_type'], content=content)
                 else:
                     raise NotImplementedError('Not implemented')
 

addons/sms/views/mail_notification_views.xml

@@ -19,7 +19,7 @@
             <xpath expr="//field[@name='res_partner_id']" position="after">
                 <field name="sms_number"/>
             </xpath>
-            <xpath expr="//field[@name='mail_id']" position="after">
+            <xpath expr="//field[@name='mail_mail_id']" position="after">
                 <field name="sms_id"/>
             </xpath>
         </field>

addons/sms/views/sms_sms_views.xml

@@ -19,7 +19,7 @@
                         </group>
                         <group>
                             <field name="number" required="1"/>
-                            <field name="error_code" readonly="1" attrs="{'invisible': [('error_code', '=', False)]}"/>
+                            <field name="failure_type" readonly="1" attrs="{'invisible': [('failure_type', '=', False)]}"/>
                         </group>
                     </group>
                     <group>
@@ -38,7 +38,7 @@
                 <field name="number"/>
                 <field name="partner_id"/>
                 <field name="state"/>
-                <field name="error_code"/>
+                <field name="failure_type"/>
             </tree>
         </field>
     </record>

addons/sms/wizard/sms_composer.py

@@ -273,6 +273,11 @@ class SendSMS(models.TransientModel):
             return [r.id for r in records if recipients_info[r.id]['sanitized'] in bl_numbers]
         return []
 
+    def _get_optout_record_ids(self, records, recipients_info):
+        """ Compute opt-outed contacts, not necessarily blacklisted. Void by default
+        as no opt-out mechanism exist in SMS, see SMS Marketing. """
+        return []
+
     def _get_done_record_ids(self, records, recipients_info):
         """ Get a list of already-done records. Order of record set is used to
         spot duplicates so pay attention to it if necessary. """
@@ -300,6 +305,7 @@ class SendSMS(models.TransientModel):
         all_bodies = self._prepare_body_values(records)
         all_recipients = self._prepare_recipient_values(records)
         blacklist_ids = self._get_blacklist_record_ids(records, all_recipients)
+        optout_ids = self._get_optout_record_ids(records, all_recipients)
         done_ids = self._get_done_record_ids(records, all_recipients)
 
         result = {}
@@ -308,23 +314,26 @@ class SendSMS(models.TransientModel):
             sanitized = recipients['sanitized']
             if sanitized and record.id in blacklist_ids:
                 state = 'canceled'
-                error_code = 'sms_blacklist'
+                failure_type = 'sms_blacklist'
+            elif sanitized and record.id in optout_ids:
+                state = 'canceled'
+                failure_type = 'sms_optout'
             elif sanitized and record.id in done_ids:
                 state = 'canceled'
-                error_code = 'sms_duplicate'
+                failure_type = 'sms_duplicate'
             elif not sanitized:
-                state = 'error'
-                error_code = 'sms_number_format' if recipients['number'] else 'sms_number_missing'
+                state = 'canceled'
+                failure_type = 'sms_number_format' if recipients['number'] else 'sms_number_missing'
             else:
                 state = 'outgoing'
-                error_code = ''
+                failure_type = ''
 
             result[record.id] = {
                 'body': all_bodies[record.id],
                 'partner_id': recipients['partner'].id,
                 'number': sanitized if sanitized else recipients['number'],
                 'state': state,
-                'error_code': error_code,
+                'failure_type': failure_type,
             }
         return result
 

addons/test_mail/tests/__init__.py

@@ -11,7 +11,6 @@ from . import test_mail_mail
 from . import test_mail_gateway
 from . import test_mail_thread_internals
 from . import test_mail_template
-from . import test_mail_tools
 from . import test_message_management
 from . import test_message_post
 from . import test_message_track

addons/test_mail/tests/test_mail_composer.py

@@ -216,7 +216,7 @@ class TestComposerInternals(TestMailComposer):
             'body': '<p>Test Body</p>',
         })
         # currently onchange necessary
-        composer.onchange_template_id_wrapper()
+        composer._onchange_template_id_wrapper()
 
         # values coming from template
         self.assertEqual(len(composer.attachment_ids), 4)
@@ -255,7 +255,7 @@ class TestComposerInternals(TestMailComposer):
             # changing template should update its email_from
             composer.write({'template_id': self.template.id, 'author_id': self.env.user.partner_id})
             # currently onchange necessary
-            composer.onchange_template_id_wrapper()
+            composer._onchange_template_id_wrapper()
             self.assertEqual(composer.author_id, self.env.user.partner_id,
                              'MailComposer: should take value given by user')
             if composition_mode == 'comment':
@@ -294,7 +294,7 @@ class TestComposerInternals(TestMailComposer):
             # changing template should update its content
             composer.write({'template_id': self.template.id})
             # currently onchange necessary
-            composer.onchange_template_id_wrapper()
+            composer._onchange_template_id_wrapper()
 
             # values come from template
             if composition_mode == 'comment':
@@ -313,7 +313,7 @@ class TestComposerInternals(TestMailComposer):
             # reset template should reset values
             composer.write({'template_id': False})
             # currently onchange necessary
-            composer.onchange_template_id_wrapper()
+            composer._onchange_template_id_wrapper()
 
             # values are reset
             if composition_mode == 'comment':
@@ -336,7 +336,7 @@ class TestComposerInternals(TestMailComposer):
                 'template_id': self.template.id,
             })
             # currently onchange necessary
-            composer.onchange_template_id_wrapper()
+            composer._onchange_template_id_wrapper()
 
             # values come from template
             if composition_mode == 'comment':
@@ -354,7 +354,7 @@ class TestComposerInternals(TestMailComposer):
                 'template_id': self.template.id,
             })
             # currently onchange necessary
-            composer.onchange_template_id_wrapper()
+            composer._onchange_template_id_wrapper()
 
             # values come from template
             if composition_mode == 'comment':
@@ -407,7 +407,7 @@ class TestComposerInternals(TestMailComposer):
                 'subject': 'Subject',
                 'body': '<p>Body text</p>',
                 'partner_ids': []
-            }).send_mail()
+            })._action_send_mail()
 
             self.assertEqual(self.test_record.message_ids[0].body, '<p>Body text</p>')
             self.assertEqual(self.test_record.message_ids[0].author_id, portal_user.partner_id)
@@ -418,7 +418,7 @@ class TestComposerInternals(TestMailComposer):
             }).create({
                 'subject': 'Subject',
                 'body': '<p>Body text 2</p>'
-            }).send_mail()
+            })._action_send_mail()
 
             self.assertEqual(self.test_record.message_ids[0].body, '<p>Body text 2</p>')
             self.assertEqual(self.test_record.message_ids[0].author_id, portal_user.partner_id)
@@ -430,7 +430,7 @@ class TestComposerInternals(TestMailComposer):
         ).create({
             'subject': 'Template Subject',
             'body': '<p>Template Body</p>',
-        }).save_as_template()
+        }).action_save_as_template()
 
         # Test: email_template subject, body_html, model
         template = self.env['mail.template'].search([
@@ -457,7 +457,7 @@ class TestComposerResultsComment(TestMailComposer):
             'partner_ids': [(4, self.partner_1.id), (4, self.partner_2.id)]
         })
         with self.mock_mail_gateway(mail_unlink_sent=True):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # notifications
         message = self.test_record.message_ids[0]
@@ -477,7 +477,7 @@ class TestComposerResultsComment(TestMailComposer):
             'partner_ids': [(4, self.partner_1.id), (4, self.partner_2.id)]
         })
         with self.mock_mail_gateway(mail_unlink_sent=True):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # notifications
         message = self.test_record.message_ids[0]
@@ -498,7 +498,7 @@ class TestComposerResultsComment(TestMailComposer):
             'body': '<p>Test Body</p>',
             'partner_ids': [(4, self.partner_1.id), (4, self.partner_2.id)]
         })
-        composer.send_mail()
+        composer._action_send_mail()
 
         message = self.test_record.message_ids[0]
         self.assertEqual(message.body, '<p>Test Body</p>')
@@ -543,7 +543,7 @@ class TestComposerResultsComment(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=False), self.mock_mail_app():
-            composer.send_mail()
+            composer._action_send_mail()
 
         # check new partners have been created based on emails given
         new_partners = self.env['res.partner'].search([
@@ -626,7 +626,7 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=True):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # global outgoing
         self.assertEqual(len(self._new_mails), 2, 'Should have created 1 mail.mail per record')
@@ -682,7 +682,7 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=False):
-            composer.send_mail()
+            composer._action_send_mail()
 
         new_partners = self.env['res.partner'].search([
             ('email', 'in', [email_to_1, email_to_2, email_to_3, email_cc_1])
@@ -731,7 +731,7 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=True):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # global outgoing
         self.assertEqual(len(self._new_mails), 2, 'Should have created 1 mail.mail per record')
@@ -764,7 +764,7 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=True):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # global outgoing
         self.assertEqual(len(self._new_mails), 2, 'Should have created 1 mail.mail per record')
@@ -793,7 +793,7 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=True):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # global outgoing
         self.assertEqual(len(self._new_mails), 2, 'Should have created 1 mail.mail per record')
@@ -812,7 +812,7 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=True):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # global outgoing
         self.assertEqual(len(self._new_mails), 2, 'Should have created 1 mail.mail per record')
@@ -826,7 +826,7 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=False):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # global outgoing
         self.assertEqual(len(self._new_mails), 1, 'Should have created 1 mail.mail per record')
@@ -839,4 +839,4 @@ class TestComposerResultsMass(TestMailComposer):
         ))
         composer = composer_form.save()
         with self.mock_mail_gateway(mail_unlink_sent=False), self.assertRaises(ValueError):
-            composer.send_mail()
+            composer._action_send_mail()

addons/test_mail/tests/test_mail_followers.py

@@ -406,8 +406,8 @@ class AdvancedResponsibleNotifiedTest(TestMailCommon):
         # verify that a mail.mail is attached to it with the correct state ('outgoing')
         mail_notification = mail_message.notification_ids
         self.assertEqual(1, len(mail_notification))
-        self.assertTrue(bool(mail_notification.mail_id))
-        self.assertEqual('outgoing', mail_notification.mail_id.state)
+        self.assertTrue(bool(mail_notification.mail_mail_id))
+        self.assertEqual(mail_notification.mail_mail_id.state, 'outgoing')
 
 
 @tagged('post_install', '-at_install')

addons/test_mail/tests/test_mail_mail.py

@@ -143,7 +143,7 @@ class TestMailMailRace(common.TransactionCase):
         # we need to simulate a mail sent by the cron task, first create mail, message and notification by hand
         mail = self.env['mail.mail'].sudo().create({
             'body_html': '<p>Test</p>',
-            'notification': True,
+            'is_notification': True,
             'state': 'outgoing',
             'recipient_ids': [(4, self.partner.id)]
         })
@@ -153,7 +153,7 @@ class TestMailMailRace(common.TransactionCase):
             'subtype_id': self.ref('mail.mt_comment'),
             'notification_ids': [(0, 0, {
                 'res_partner_id': self.partner.id,
-                'mail_id': mail.id,
+                'mail_mail_id': mail.id,
                 'notification_type': 'email',
                 'is_read': True,
                 'notification_status': 'ready',

addons/test_mail/tests/test_mail_tools.py

@@ -1,83 +0,0 @@
-# -*- coding: utf-8 -*-
-# Part of Odoo. See LICENSE file for full copyright and licensing details.
-
-from odoo.addons.test_mail.tests.common import TestMailCommon, TestRecipients
-from odoo.tests import tagged, users
-
-
-@tagged('mail_tools')
-class TestMailTools(TestMailCommon, TestRecipients):
-
-    @classmethod
-    def setUpClass(cls):
-        super(TestMailTools, cls).setUpClass()
-
-        cls._test_email = 'alfredoastaire@test.example.com'
-        cls.test_partner = cls.env['res.partner'].create({
-            'country_id': cls.env.ref('base.be').id,
-            'email': cls._test_email,
-            'mobile': '0456001122',
-            'name': 'Alfred Astaire',
-            'phone': '0456334455',
-        })
-
-    @users('employee')
-    def test_find_partner_from_emails(self):
-        Partner = self.env['res.partner']
-        test_partner = Partner.browse(self.test_partner.ids)
-        self.assertEqual(test_partner.email, self._test_email)
-
-        # test direct match
-        found = Partner._mail_find_partner_from_emails([self._test_email])
-        self.assertEqual(found, [test_partner])
-
-        # test encapsulated email
-        found = Partner._mail_find_partner_from_emails(['"Norbert Poiluchette" <%s>' % self._test_email])
-        self.assertEqual(found, [test_partner])
-
-        # test with wildcard "_"
-        found = Partner._mail_find_partner_from_emails(['alfred_astaire@test.example.com'])
-        self.assertEqual(found, [self.env['res.partner']])
-
-        # sub-check: this search does not consider _ as a wildcard
-        found = Partner._mail_search_on_partner(['alfred_astaire@test.example.com'])
-        self.assertEqual(found, self.env['res.partner'])
-
-        # test partners with encapsulated emails
-        # ------------------------------------------------------------
-        test_partner.sudo().write({'email': '"Alfred Mighty Power Astaire" <%s>' % self._test_email})
-
-        # test direct match
-        found = Partner._mail_find_partner_from_emails([self._test_email])
-        self.assertEqual(found, [test_partner])
-
-        # test encapsulated email
-        found = Partner._mail_find_partner_from_emails(['"Norbert Poiluchette" <%s>' % self._test_email])
-        self.assertEqual(found, [test_partner])
-
-        # test with wildcard "_"
-        found = Partner._mail_find_partner_from_emails(['alfred_astaire@test.example.com'])
-        self.assertEqual(found, [self.env['res.partner']])
-
-        # sub-check: this search does not consider _ as a wildcard
-        found = Partner._mail_search_on_partner(['alfred_astaire@test.example.com'])
-        self.assertEqual(found, self.env['res.partner'])
-
-        # test partners with look-alike emails
-        # ------------------------------------------------------------
-        for email_lookalike in [
-                'alfred.astaire@test.example.com',
-                'alfredoastaire@example.com',
-                'aalfredoastaire@test.example.com',
-                'alfredoastaire@test.example.comm']:
-            test_partner.sudo().write({'email': '"Alfred Astaire" <%s>' % email_lookalike})
-
-            # test direct match
-            found = Partner._mail_find_partner_from_emails([self._test_email])
-            self.assertEqual(found, [self.env['res.partner']])
-            # test encapsulated email
-            found = Partner._mail_find_partner_from_emails(['"Norbert Poiluchette" <%s>' % self._test_email])
-            self.assertEqual(found, [self.env['res.partner']])
-            # test with wildcard "_"
-            found = Partner._mail_find_partner_from_emails(['alfred_astaire@test.example.com'])
-            self.assertEqual(found, [self.env['res.partner']])

addons/test_mail/tests/test_performance.py

@@ -272,7 +272,7 @@ class TestMailAPIPerformance(BaseMailPerformance):
             })
 
         with self.assertQueryCount(__system__=32, emp=38):
-            composer.send_mail()
+            composer._action_send_mail()
 
     @users('__system__', 'emp')
     @warmup
@@ -293,7 +293,7 @@ class TestMailAPIPerformance(BaseMailPerformance):
             })
 
         with self.assertQueryCount(__system__=25, emp=31):
-            composer.send_mail()
+            composer._action_send_mail()
 
     @users('__system__', 'emp')
     @warmup
@@ -309,10 +309,10 @@ class TestMailAPIPerformance(BaseMailPerformance):
                 'default_res_id': test_record.id,
                 'default_template_id': test_template.id,
             }).create({})
-            composer.onchange_template_id_wrapper()
+            composer._onchange_template_id_wrapper()
 
         with self.assertQueryCount(__system__=33, emp=39):
-            composer.send_mail()
+            composer._action_send_mail()
 
         # remove created partner to ensure tests are the same each run
         self.env['res.partner'].sudo().search([('email', '=', 'nopartner.test@example.com')]).unlink()

addons/test_mail_full/models/mailing_mailing.py

@@ -1,29 +0,0 @@
-# -*- coding: utf-8 -*-
-# Part of Odoo. See LICENSE file for full copyright and licensing details.
-
-import logging
-
-from odoo import models
-
-_logger = logging.getLogger(__name__)
-
-
-class Mailing(models.Model):
-    _inherit = 'mailing.mailing'
-
-    def _get_opt_out_list_sms(self):
-        """Returns a set of emails opted-out in target model"""
-        self.ensure_one()
-        if self.mailing_model_real in ('mail.test.sms.bl.optout',
-                                       'mail.test.sms.partner',
-                                       'mail.test.sms.partner.2many'):
-            res_ids = self._get_recipients()
-            opt_out_contacts = set(self.env[self.mailing_model_real].search([
-                ('id', 'in', res_ids),
-                ('opt_out', '=', True)
-            ]).ids)
-            _logger.info(
-                "Mass-mailing %s targets %s, optout: %s emails",
-                self, self.mailing_model_real, len(opt_out_contacts))
-            return opt_out_contacts
-        return super(Mailing, self)._get_opt_out_list_sms()

addons/test_mail_full/tests/test_mass_mailing.py

@@ -11,6 +11,7 @@ from odoo.tests import tagged
 
 @tagged('mass_mailing')
 class TestMassMailing(TestMailFullCommon):
+
     @users('user_marketing')
     @mute_logger('odoo.addons.mail.models.mail_mail')
     def test_mailing_w_blacklist_opt_out(self):
@@ -44,21 +45,26 @@ class TestMassMailing(TestMailFullCommon):
             recipient_info = {
                 'email': recipient.email_normalized,
                 'content': 'Hello %s' % recipient.name}
-            # opt-out: ignored (cancel mail)
+            # opt-out: cancel (cancel mail)
             if recipient in recipients[1] | recipients[2]:
-                recipient_info['state'] = 'ignored'
-            # blacklisted: ignored (cancel mail)
+                recipient_info['trace_status'] = "cancel"
+                recipient_info['failure_type'] = "mail_optout"
+            # blacklisted: cancel (cancel mail)
             elif recipient in recipients[3] | recipients[4]:
-                recipient_info['state'] = 'ignored'
-            # duplicates: ignored (cancel mail)
+                recipient_info['trace_status'] = "cancel"
+                recipient_info['failure_type'] = "mail_bl"
+            # duplicates: cancel (cancel mail)
             elif recipient == recipient_dup_1:
-                recipient_info['state'] = 'ignored'
-            # void: ignored (cancel mail)
+                recipient_info['trace_status'] = "cancel"
+                recipient_info['failure_type'] = "mail_dup"
+            # void: error (failed mail)
             elif recipient == recipient_void_1:
-                recipient_info['state'] = 'ignored'
-            # falsy: ignored (cancel mail)
+                recipient_info['trace_status'] = 'cancel'
+                recipient_info['failure_type'] = "mail_email_missing"
+            # falsy: error (failed mail)
             elif recipient == recipient_falsy_1:
-                recipient_info['state'] = 'ignored'
+                recipient_info['trace_status'] = "cancel"
+                recipient_info['failure_type'] = "mail_email_invalid"
                 recipient_info['email'] = recipient.email_from  # normalized is False but email should be falsymail
             else:
                 email = self._find_sent_mail_wemail(recipient.email_normalized)
@@ -111,4 +117,5 @@ class TestMassMailing(TestMailFullCommon):
                 check_mail=True,)
 
         # sent: 13, 2 bl, 2 opt-out, 3 invalid -> 6 remaining
-        self.assertMailingStatistics(mailing, expected=13, delivered=6, sent=6, ignored=7)
+        # ignored: 2 bl + 2 optout + 2 invalid + 1 duplicate; failed: 0
+        self.assertMailingStatistics(mailing, expected=13, delivered=6, sent=6, canceled=7, failed=0)

addons/test_mail_full/tests/test_mass_sms.py

@@ -169,28 +169,28 @@ class TestMassSMSInternals(TestMassSMSCommon):
         # duplicates
         self.assertSMSTraces(
             [{'partner': new_record_2.customer_id, 'number': self.records_numbers[0],
-              'content': 'Dear %s this is a mass SMS.' % new_record_2.display_name, 'state': 'ignored',
+              'content': 'Dear %s this is a mass SMS.' % new_record_2.display_name, 'trace_status': 'cancel',
               'failure_type': 'sms_duplicate'}],
             self.mailing, new_record_2,
         )
         # blacklist
         self.assertSMSTraces(
             [{'partner': self.env['res.partner'], 'number': phone_validation.phone_format(bl_record_1.phone_nbr, 'BE', '32', force_format='E164'),
-              'content': 'Dear %s this is a mass SMS.' % bl_record_1.display_name, 'state': 'ignored',
+              'content': 'Dear %s this is a mass SMS.' % bl_record_1.display_name, 'trace_status': 'cancel',
               'failure_type': 'sms_blacklist'}],
             self.mailing, bl_record_1,
         )
         # missing number
         self.assertSMSTraces(
             [{'partner': self.env['res.partner'], 'number': False,
-              'content': 'Dear %s this is a mass SMS.' % void_record.display_name, 'state': 'exception',
+              'content': 'Dear %s this is a mass SMS.' % void_record.display_name, 'trace_status': 'cancel',
               'failure_type': 'sms_number_missing'}],
             self.mailing, void_record,
         )
         # wrong values
         self.assertSMSTraces(
             [{'partner': self.env['res.partner'], 'number': record.phone_nbr,
-              'content': 'Dear %s this is a mass SMS.' % record.display_name, 'state': 'bounced',
+              'content': 'Dear %s this is a mass SMS.' % record.display_name, 'trace_status': 'cancel',
               'failure_type': 'sms_number_format'}
              for record in falsy_record_1 + falsy_record_2],
             self.mailing, falsy_record_1 + falsy_record_2,
@@ -220,7 +220,7 @@ class TestMassSMSInternals(TestMassSMSCommon):
         # new failed traces generated for duplicates
         self.assertSMSTraces(
             [{'partner': record.customer_id, 'number': self.records_numbers[i],
-              'content': 'Dear %s this is a mass SMS.' % record.display_name, 'state': 'ignored',
+              'content': 'Dear %s this is a mass SMS.' % record.display_name, 'trace_status': 'cancel',
               'failure_type': 'sms_duplicate'}
              for i, record in enumerate(self.records[:5])],
             self.mailing, self.records[:5],
@@ -282,7 +282,7 @@ class TestMassSMS(TestMassSMSCommon):
         self.assertSMSTraces(
             [{'partner': record.customer_id,
               'number': self.records_numbers[i],
-              'state': 'sent',
+              'trace_status': 'sent',
               'content': 'Dear %s this is a mass SMS with two links' % record.display_name
              } for i, record in enumerate(self.records)],
             mailing, self.records,
@@ -317,7 +317,7 @@ class TestMassSMS(TestMassSMSCommon):
         self.assertSMSTraces(
             [{'partner': record.customer_id,
               'number': record.customer_id.phone_sanitized,
-              'state': 'sent',
+              'trace_status': 'sent',
               'content': 'Dear %s this is a mass SMS with two links' % record.display_name
              } for record in records],
             mailing, records,
@@ -342,7 +342,7 @@ class TestMassSMS(TestMassSMSCommon):
         self.assertSMSTraces(
             [{'partner': new_record.customer_id,
               'number': new_record.customer_id.phone_sanitized,
-              'state': 'sent',
+              'trace_status': 'sent',
               'content': 'Dear %s this is a mass SMS with two links' % new_record.display_name
              }],
             mailing, new_record,
@@ -395,11 +395,11 @@ class TestMassSMS(TestMassSMSCommon):
             mailing.action_send_sms()
 
         self.assertSMSTraces(
-            [{'number': '+32456000000', 'state': 'ignored', 'failure_type': 'sms_blacklist'},  # TDE FIXME: should be opt_out
-             {'number': '+32456000101', 'state': 'ignored', 'failure_type': 'sms_blacklist'},  # TDE FIXME: should be opt_out
-             {'number': '+32456000202', 'state': 'sent'},
-             {'number': '+32456000303', 'state': 'sent'},
-             {'number': '+32456000404', 'state': 'ignored', 'failure_type': 'sms_blacklist'}],
+            [{'number': '+32456000000', 'trace_status': 'cancel', 'failure_type': 'sms_optout'},
+             {'number': '+32456000101', 'trace_status': 'cancel', 'failure_type': 'sms_optout'},
+             {'number': '+32456000202', 'trace_status': 'sent'},
+             {'number': '+32456000303', 'trace_status': 'sent'},
+             {'number': '+32456000404', 'trace_status': 'cancel', 'failure_type': 'sms_blacklist'}],
             mailing, recipients
         )
-        self.assertEqual(mailing.ignored, 3)
+        self.assertEqual(mailing.canceled, 3)

addons/test_mail_full/tests/test_sms_composer.py

@@ -368,7 +368,7 @@ class TestSMSComposerMass(TestMailFullCommon):
         for partner in self.partners[5:]:
             self.assertSMSOutgoing(partner, partner.phone_sanitized, content=self._test_body)
         for partner in self.partners[:5]:
-            self.assertSMSCanceled(partner, partner.phone_sanitized, error_code='sms_blacklist', content=self._test_body)
+            self.assertSMSCanceled(partner, partner.phone_sanitized, failure_type='sms_blacklist', content=self._test_body)
 
     def test_composer_mass_active_ids_wo_blacklist(self):
         self.env['phone.blacklist'].create([{
@@ -419,9 +419,9 @@ class TestSMSComposerMass(TestMailFullCommon):
         for partner in self.partners[8:]:
             self.assertSMSOutgoing(partner, partner.phone_sanitized, content=self._test_body)
         for partner in self.partners[5:8]:
-            self.assertSMSCanceled(partner, partner.phone_sanitized, error_code='sms_duplicate', content=self._test_body)
+            self.assertSMSCanceled(partner, partner.phone_sanitized, failure_type='sms_duplicate', content=self._test_body)
         for partner in self.partners[:5]:
-            self.assertSMSCanceled(partner, partner.phone_sanitized, error_code='sms_blacklist', content=self._test_body)
+            self.assertSMSCanceled(partner, partner.phone_sanitized, failure_type='sms_blacklist', content=self._test_body)
 
     def test_composer_mass_active_ids_w_template(self):
         with self.with_user('employee'):

addons/test_mass_mailing/tests/__init__.py

@@ -5,7 +5,7 @@ from . import test_blacklist
 from . import test_blacklist_behavior
 from . import test_blacklist_mixin
 from . import test_link_tracker
-from . import test_mailing_test
 from . import test_mailing
-from . import test_mass_mailing_server
+from . import test_mailing_server
+from . import test_mailing_test
 from . import test_performance

addons/test_mass_mailing/tests/test_blacklist_behavior.py

@@ -3,11 +3,18 @@
 
 import datetime
 
-from odoo.addons.test_mass_mailing.tests import common
-from odoo.tests.common import users
+from freezegun import freeze_time
+from unittest.mock import patch
+
 from odoo.addons.mass_mailing.models.mail_thread import BLACKLIST_MAX_BOUNCED_LIMIT
+from odoo.addons.test_mass_mailing.tests import common
+from odoo.tests import tagged
+from odoo.tests.common import users
+from odoo.tools import mute_logger
+from odoo.sql_db import Cursor
 
 
+@tagged('mail_blacklist')
 class TestAutoBlacklist(common.TestMassMailCommon):
 
     @classmethod
@@ -42,15 +49,21 @@ class TestAutoBlacklist(common.TestMassMailCommon):
         })
         self._test_mailing_bounce_w_auto_bl({'bounced_partner': bounced_partners})
 
+    @mute_logger('odoo.addons.mail.models.mail_thread')
     def _test_mailing_bounce_w_auto_bl(self, bounce_base_values):
         mailing = self.env['mailing.mailing'].browse(self.mailing_bl.ids)
         target = self.env['mailing.test.blacklist'].browse(self.target_rec.ids)
 
         # create bounced history of 4 statistics
+        traces = self.env['mailing.trace']
         for idx in range(4):
             new_mailing = mailing.copy()
-            self._create_bounce_trace(new_mailing, target, dt=datetime.datetime.now() - datetime.timedelta(weeks=idx+2))
-            self.gateway_mail_bounce(new_mailing, target, bounce_base_values)
+            new_dt = datetime.datetime.now() - datetime.timedelta(weeks=idx+2)
+            # Cursor.now() uses transaction's timestamp and not datetime lib -> freeze_time
+            # is not sufficient
+            with freeze_time(new_dt), patch.object(Cursor, 'now', lambda *args, **kwargs: new_dt):
+                traces += self._create_bounce_trace(new_mailing, target, dt=datetime.datetime.now() - datetime.timedelta(weeks=idx+2))
+                self.gateway_mail_bounce(new_mailing, target, bounce_base_values)
 
         # mass mail record: ok, not blacklisted yet
         mailing.action_put_in_queue()
@@ -78,6 +91,6 @@ class TestAutoBlacklist(common.TestMassMailCommon):
         with self.mock_mail_gateway(mail_unlink_sent=False):
             new_mailing._process_mass_mailing_queue()
         self.assertMailTraces(
-            [{'email': 'test.record.00@test.example.com', 'state': 'ignored'}],
+            [{'email': 'test.record.00@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_bl'}],
             new_mailing, target, check_mail=True
         )

addons/test_mass_mailing/tests/test_link_tracker.py

@@ -48,22 +48,23 @@ class TestLinkTracker(common.TestMassMailCommon):
         record = self.env['mailing.test.blacklist'].create({})
         code = self.link.code
         self.assertEqual(self.link.count, 1)
-        stat = self.env['mailing.trace'].create({
+        trace = self.env['mailing.trace'].create({
             'mass_mailing_id': self.mailing_bl.id,
             'model': record._name,
             'res_id': record.id,
         })
-        self.assertFalse(stat.opened)
-        self.assertFalse(stat.clicked)
+        self.assertEqual(trace.trace_status, 'outgoing')
+        self.assertFalse(trace.links_click_datetime)
 
         # click from a new IP should create a new entry and update stat when provided
         click = self.env['link.tracker.click'].sudo().add_click(
             code,
             ip='100.00.00.01',
             country_code='BEL',
-            mailing_trace_id=stat.id
+            mailing_trace_id=trace.id
         )
         self.assertEqual(self.link.count, 2)
         self.assertEqual(click.mass_mailing_id, self.mailing_bl)
-        self.assertTrue(stat.opened)
-        self.assertTrue(stat.clicked)
+        self.assertTrue(trace.trace_status, 'open')
+        self.assertTrue(trace.links_click_datetime)
+        self.assertEqual(trace.links_click_ids, click)

addons/test_mass_mailing/tests/test_mailing.py

@@ -59,11 +59,11 @@ class TestMassMailing(TestMassMailCommon):
         traces = self.env['mailing.trace'].search([('model', '=', customers._name), ('res_id', 'in', customers.ids)])
         self.assertEqual(len(traces), 3)
         customer0_trace = traces.filtered(lambda t: t.res_id == customers[0].id)
-        self.assertEqual(customer0_trace.state, 'replied')
+        self.assertEqual(customer0_trace.trace_status, 'reply')
         customer1_trace = traces.filtered(lambda t: t.res_id == customers[1].id)
-        self.assertEqual(customer1_trace.state, 'replied')
+        self.assertEqual(customer1_trace.trace_status, 'reply')
         customer2_trace = traces.filtered(lambda t: t.res_id == customers[2].id)
-        self.assertEqual(customer2_trace.state, 'sent')
+        self.assertEqual(customer2_trace.trace_status, 'sent')
 
         # check mailing statistics
         self.assertEqual(mailing.sent, 3)
@@ -245,11 +245,11 @@ class TestMassMailing(TestMassMailCommon):
             [{'email': 'test.record.00@test.example.com'},
              {'email': 'test.record.01@test.example.com'},
              {'email': 'test.record.02@test.example.com'},
-             {'email': 'test.record.03@test.example.com', 'state': 'ignored'},
-             {'email': 'test.record.04@test.example.com', 'state': 'ignored'}],
+             {'email': 'test.record.03@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_bl'},
+             {'email': 'test.record.04@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_bl'}],
             mailing, recipients, check_mail=True
         )
-        self.assertEqual(mailing.ignored, 2)
+        self.assertEqual(mailing.canceled, 2)
 
     @users('user_marketing')
     @mute_logger('odoo.addons.mail.models.mail_mail')
@@ -271,14 +271,14 @@ class TestMassMailing(TestMassMailCommon):
             mailing._process_mass_mailing_queue()
 
         self.assertMailTraces(
-            [{'email': 'test.record.00@test.example.com', 'state': 'ignored'},
-             {'email': 'test.record.01@test.example.com', 'state': 'ignored'},
+            [{'email': 'test.record.00@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_optout'},
+             {'email': 'test.record.01@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_optout'},
              {'email': 'test.record.02@test.example.com'},
              {'email': 'test.record.03@test.example.com'},
-             {'email': 'test.record.04@test.example.com', 'state': 'ignored'}],
+             {'email': 'test.record.04@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_bl'}],
             mailing, recipients, check_mail=True
         )
-        self.assertEqual(mailing.ignored, 3)
+        self.assertEqual(mailing.canceled, 3)
 
     @users('user_marketing')
     @mute_logger('odoo.addons.mail.models.mail_mail')
@@ -331,13 +331,13 @@ class TestMassMailing(TestMassMailCommon):
             mailing._process_mass_mailing_queue()
 
         self.assertMailTraces(
-            [{'email': 'test@test.example.com', 'state': 'sent'},
-             {'email': 'test@test.example.com', 'state': 'ignored'},
+            [{'email': 'test@test.example.com', 'trace_status': 'sent'},
+             {'email': 'test@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_dup'},
              {'email': 'test3@test.example.com'},
              {'email': 'test4@test.example.com'},
-             {'email': 'test5@test.example.com', 'state': 'ignored'}],
+             {'email': 'test5@test.example.com', 'trace_status': 'cancel', 'failure_type': 'mail_optout'}],
             mailing,
             mailing_contact_1 + mailing_contact_2 + mailing_contact_3 + mailing_contact_4 + mailing_contact_5,
             check_mail=True
         )
-        self.assertEqual(mailing.ignored, 2)
+        self.assertEqual(mailing.canceled, 2)

addons/test_mass_mailing/tests/test_performance.py

@@ -50,8 +50,8 @@ class TestMassMailPerformance(TestMassMailPerformanceBase):
             'mailing_domain': [('id', 'in', self.mm_recs.ids)],
         })
 
-        # runbot needs +50 compared to local
-        with self.assertQueryCount(__system__=1718, marketing=1718):
+        # runbot needs +101 compared to local (1567)
+        with self.assertQueryCount(__system__=1668, marketing=1668):
             mailing.action_send_mail()
 
         self.assertEqual(mailing.sent, 50)
@@ -90,8 +90,8 @@ class TestMassMailBlPerformance(TestMassMailPerformanceBase):
             'mailing_domain': [('id', 'in', self.mm_recs.ids)],
         })
 
-        # runbot needs +62 compared to local
-        with self.assertQueryCount(__system__=2008, marketing=2008):
+        # runbot needs +125 compared to local (1833)
+        with self.assertQueryCount(__system__=1958, marketing=1958):
             mailing.action_send_mail()
 
         self.assertEqual(mailing.sent, 50)

addons/website_sale/models/mail_compose_message.py

@@ -7,7 +7,7 @@ from odoo import api, models
 class MailComposeMessage(models.TransientModel):
     _inherit = 'mail.compose.message'
 
-    def send_mail(self, auto_commit=False):
+    def _action_send_mail(self, auto_commit=False):
         context = self._context
         # TODO TDE: clean that brole one day
         if context.get('website_sale_send_recovery_email') and self.model == 'sale.order' and context.get('active_ids'):
@@ -16,4 +16,4 @@ class MailComposeMessage(models.TransientModel):
                 ('cart_recovery_email_sent', '=', False),
                 ('is_abandoned_cart', '=', True)
             ]).write({'cart_recovery_email_sent': True})
-        return super(MailComposeMessage, self).send_mail(auto_commit=auto_commit)
+        return super(MailComposeMessage, self)._action_send_mail(auto_commit=auto_commit)

odoo/tools/mail.py

@@ -470,48 +470,10 @@ def generate_tracking_message_id(res_id):
     rndstr = ("%.15f" % rnd)[2:]
     return "<%s.%.15f-openerp-%s@%s>" % (rndstr, time.time(), res_id, socket.gethostname())
 
-def email_send(email_from, email_to, subject, body, email_cc=None, email_bcc=None, reply_to=False,
-               attachments=None, message_id=None, references=None, openobject_id=False, debug=False, subtype='plain', headers=None,
-               smtp_server=None, smtp_port=None, ssl=False, smtp_user=None, smtp_password=None, cr=None, uid=None):
-    """Low-level function for sending an email (deprecated).
-
-    :deprecate: since OpenERP 6.1, please use ir.mail_server.send_email() instead.
-    :param email_from: A string used to fill the `From` header, if falsy,
-                       config['email_from'] is used instead.  Also used for
-                       the `Reply-To` header if `reply_to` is not provided
-    :param email_to: a sequence of addresses to send the mail to.
-    """
-
-    # If not cr, get cr from current thread database
-    local_cr = None
-    if not cr:
-        db_name = getattr(threading.currentThread(), 'dbname', None)
-        if db_name:
-            local_cr = cr = odoo.registry(db_name).cursor()
-        else:
-            raise Exception("No database cursor found, please pass one explicitly")
-
-    # Send Email
-    try:
-        mail_server_pool = odoo.registry(cr.dbname)['ir.mail_server']
-        res = False
-        # Pack Message into MIME Object
-        email_msg = mail_server_pool.build_email(email_from, email_to, subject, body, email_cc, email_bcc, reply_to,
-                   attachments, message_id, references, openobject_id, subtype, headers=headers)
-
-        res = mail_server_pool.send_email(cr, uid or 1, email_msg, mail_server_id=None,
-                       smtp_server=smtp_server, smtp_port=smtp_port, smtp_user=smtp_user, smtp_password=smtp_password,
-                       smtp_encryption=('ssl' if ssl else None), smtp_debug=debug)
-    except Exception:
-        _logger.exception("tools.email_send failed to deliver email")
-        return False
-    finally:
-        if local_cr:
-            cr.close()
-    return res
-
 def email_split_tuples(text):
-    """ Return a list of (name, email) address tuples found in ``text``"""
+    """ Return a list of (name, email) address tuples found in ``text`` . Note
+    that text should be an email header or a stringified email list as it may
+    give broader results than expected on actual text. """
     if not text:
         return []
     return [(addr[0], addr[1]) for addr in getaddresses([text])