PURPOSE
Overall cleaning of subscription and exclusion management code from portal.
This code comes mainly from v12 and can now benefit from cleaning and update.
SPECIFICATIONS
In this commit we rename ``mailing.contact.subscription`` model into the
shorter ``mailing.subscription``. This is sufficient to explain the model
purpose. As we plan to add an opt-out model this also allows to have sub
models with suffixes without being too long.
We also rename ``subscription_list_ids`` field on contact model to
``subscription_ids`` as this is shorter and clearer.
Finally the ``mailing_contact_list_rel`` table name that comes from old
implementations (simple m2m table) is renamed to ``mailing_subscription``
to match the model name.
Task-2669037 (Mass Mailing: Refactor js/portal for subscription)
Part-of: odoo/odoo#86084
Only duplicate emails used to be checked when sending a mass mail.
However it is possible (e.g. using templates) to send a mass mail
to the same person containing different information.
The existing functions to allow models to specify emails
processed in the past by some other means are kept.
A new check is added in the processing that checks the full contents
of the message, subject and attachment ids.
The strings are not hashed as most situations are:
- Sending the exact same mail to everyone
-> Only need to check against one message
-> Same complexity as hashing
- Sending all different emails
-> Checking inequality of str is usually very fast
For attachments, as we cannot compare them easily.
They are ignored for the purpose of equating emails
whenever there are the same number of attachments
in the email values as there are on the composer.
This is because each email should receive its own copy
of the composer attachments. If they have a different
number of attachments, they were generated dynamically
through reports and we assume they are all different.
We can thus remove the 'document based' information
as it is implicitly infered from this check.
-------------------------
Test utils are also updated for two purposes:
1. Add optional body and attachment_name discriminents
assertMailMail assumed all emails could at least be
differentiated by subject. Our test breaks that
assumption, so we use body and attachment to find the
best-fitting email based on the data passed in.
2. Check email_formatted on recipients
When using assertMailMailWEmails, we first find the
email using the non-formatted email of the recipient.
This does not match assertSentMail which checks against
the raw email_to value, which would often be formatted.
Task-2826811
closesodoo/odoo#99541
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Before the commit, the _get_seen_list() function in the mass_mailing module was
not able to correctly identify all the duplicate email addresses in a given mass
mailing. This was because the function chose and used only one way to find an
email address for each record in the mailing list, even though there are many
ways to find an email address for a record.
For example, a crm.lead record might have an email address in its partner_id
field, but it might also have an email address in its email_normalized field.
This can vary from record to record.
To fix this issue, the _get_seen_list() function was updated to only look at the
email address to which emails have already been sent, rather than trying to
fetch it from the record itself. This ensures that all duplicate emails are
correctly identified and that no duplicate emails are sent in the mass mailing.
Task-3234378
closesodoo/odoo#135081
X-original-commit: 66f9aa25af049aa3faf0f76475a4a2b63b5d0903
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
With input 'name email@domain.com' (missing chevrons allowing to clearly spot
the email part) 'getaddresses' returns ('', 'name email@domain.com) i.e. the
whole input is considered as being the email.
To improve the heuristic we can add a fallback by recalling 'getadresses'
on the input with spaces replaced by commas when it found only an email and
no name. The new email will be split into sub pairs allowing to find the real
email and various name parts, allowing to make a new name / email pair.
Emails should not contain spaces thus this is coherent with email formation.
This fallback actually comes from a specific code done in '_parse_partner_name'
of Partner model. Supporting it directly at tools level make the behavior
coherent for all models.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@18c71edf59
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
SPECIFICATIONS
As of rfc5322 section 3.4.1 local-part is case-sensitive. However most main
providers do consider the local-part as case insensitive. With the introduction
of smtp-utf8 within odoo, this assumption is certain to fall short for
international emails. We now consider that
* if local part is ascii: normalize still 'lower' ;
* else: use as it, SMTP-UF8 is made for non-ascii local parts;
Concerning domain part of the address, as of v14 international domain (IDNA)
are handled fine. The domain is always lowercase, lowering it is fine as it
is probably an error. With the introduction of IDNA, there is an encoding
that allow non-ascii characters to be encoded to ascii ones, using 'idna.encode'.
Also remove usage of 'email_re' in mailing email check. It is too restrictive
compared to real formatting we support (or try to). Valid outgoing emails
were directly canceled, notably when containing unicode.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@3ce5fb3072
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
SPECIFICATIONS: MAIL COMPOSER IN MAILING
When using the composer with a mailing, it currently skips recipients whose
email is a multi-email due to the strict usage of 'email_normalize'.
We can improve multi-email support by effectively checking for the first
email found, using the "less strict" mode of normalize. It means more emails
are detected as valid, and therefore sent.
Due to lower support of multi-emails when sending emails, this even allows
to send multiple emails as all emails are mailed.
SPECIFICATIONS: DEFAULT RECIPIENTS
Mailings are generally done using default recipients, aka using a model method
that returns the people to mail: customers ('partner_id'), customer emails
('email_from'), specific implementation, ...
This is implementation using '_message_get_default_recipients' that returns
'partner_ids', 'email_to' and 'email_cc' that are then used in the mail
composer to generate final recipients.
In this commit we better handle the content of email fields to avoid issues
with multi-emails. For that purpose we correctly split the content of those
fields. We now have several 'email_to' for records having multi-emails instead
of a single badly-formatted 'email_to'.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@4a0d87d44f
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
SPECIFICATIONS
When having multi-emails input in an email field, 'email_normalized' field is
currently 'False', as they expect the field to contain a single email. This
has several drawbacks
* searching partners or fetching information based on emails does not work as
most tool methods use 'email_normalized' which is False (see e.g.
'_message_partner_info_from_emails', '_mail_find_partner_from_emails'
or 'find_or_create');
* blacklist is not available as it is based on 'email_normalized';
* mass_mailing wrongly considers those emails as invalid and cancel their
mail and related trace, as it tries to skip sending emails to invalid
emails;
Be more defensive and use first found email in case of multi-emails field.
Other emails are ignored. It is already an improvement that does not break
flows in stable and allow more emails to be sent.
before
-> email: '"Raoul" <raoul1@raoul.fr>, raoul2@raoul.fr'
-> email_normalized: False
after
-> email: '"Raoul" <raoul1@raoul.fr>, raoul2@raoul.fr'
-> email_normalized: raoul1@raoul.fr
A side effect is that it helps finding back some partners, as indicated in
tests where less phantom partners are created. It also helps suggested
partners / emails flow in discuss.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@90218186c5
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
SPECIFICATIONS
When building the final 'email_to' of outgoing emails using 'formataddr' we
have issues if email contains multi emails or formatted email. Main fix of
this commit is to extract emails and rebuild the 'email_to' list based on
found emails.
E.g. partner Raoul - email: "Raoul" <raoul@raoul.fr>
-> before: to: "Raoul" <"Raoul" <raoul@raoul.fr>> (double format)
-> after: to: "Raoul" <raoul@raoul.fr>
E.g. partner Raoul - email: raoul1@raoul.fr, raoul2@raoul.fr
-> before: to: "Raoul" <raoul1@raoul.fr, raoul2@raoul.fr>
single email with multiple emails, depends on server fault tolerance)
-> after: to: "Raoul" <raoul1@raoul.fr>, "Raoul" <raoul2@raoul.fr>
multi emails
Fix that computation by using all normalized emails found in 'email' fields
and rebuilding a formatted email based on name + those emails. We do not
use `email_formatted` as it is not really multi-enabled. We prefer a local
defensive approach to be as tolerant as possible with respect to user inputs.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@1c4b704149
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
RATIONALE
Add tests related to not standard usage of email field. Two main use cases
are tested here
* formatted emails: `"Full Name" <email@domain.com>` stored into the 'email'
field;
* multi emails: `email1@domain.com, email2@domain.com` stored into a single
'email' field;
Additional tests: tests with unicode / ascii / case / wrong formatting are also
added to check the support in normalize and format methods.
IMPLICATION
Email field is generally managed as "containing a valid email". This means
it is sometimes used as it in 'formataddr' as well as to perform searches or
identification checks. Example of issue: partner 'Raoul' has a formatted email
like "Raoul" <raoul@raoul.fr>. Using 'formataddr' in email_from leads to
from: "Raoul" <"Raoul" <raoul@raoul.fr>>
-> which is incorrect (but often dynamically corrected by email servers);
Email field holding multi-emails are not normalized, as current normalize
is done only if the field holds a single email. It means
* no easy finding based on 'email_normalized', e.g. various tools like
'_mail_find_partner_from_emails' or 'find_or_create' do not find partners
based on this email;
* no exclusion list management;
* issue with formatting, like
to: "Raoul" <raoul@raoul.fr,raoul.other@raoul.fr>
-> which is incorrect (but often dynamically corrected by email servers);
USAGE: OUTGOING EMAILS
Those use cases currently generate faulty outgoing emails. This is valid for
recipients ('email_cc', 'email_to') as well as author ('email_from').
For formatted emails: `email_to` is formatted again based on name and email
which leads to sending emails to `"Full Name" <"Other"<email@domain.com>>`.
Note that multi emails without formatting may work as it leads to email_to
`"Full name" <email1@domain.com,email2@domain.com>`. Some outgoing email
servers correctly send multiple emails. It depends on their fault
tolerance.
USAGE: FIND BASED ON EMAIL (NORMALIZED)
When searching for partners (e.g. using '_mail_find_partner_from_emails' or
'find_or_create') normalized version of input is used.
In case of multi emails sanitize is 'False', as normalization expects a single
email in the field. Therefore no partner is found. In processes that do a
"search or create" (e.g. using a template on a record) this leads to creating
a new partner (or several partners in case of multi emails) each time.
USAGE: OTHER FLOWS
Other flows are build on top of '_mail_find_partner_from_emails' / 'create'
of outgoing emails and are impacted by formatted email / multi email usage.
Those include notably
* mass_mailing: '_message_get_default_recipients' should be defensive to
give correct values when creating mailing emails;
* mass_mailing: faulty emails is based on normalize and multi-emails are
considered as faulty and ignored;
* after post hook: '_message_post_after_hook' tries to link messages without
author (but email_from) with newly-created partners, when partners are
created from chatter. It is therefore impacted by those corner cases;
* marketing_automation: built on top of mass_mailing and suffers from the
same issues;
USAGE: UNICODE
Unicode in emails should be supported. 'formataddr' and IrMailServer notably
received fixes to support unicode. Some check performed on email addresses
fail when unicode is involved, which leads to some emails not being sent
while they could.
SPECIFICATIONS
Add tests related to those corner cases. Also add tests for computation of
`email_formatted` field of Partner model. It currently generates wrong email
values for the same corner cases (multi emails, formatted emails).
Tests are also added for the computation of `email_normalized` field used
notably for blacklists. It is not computed currently when being in multi
email mode which prevents from any blacklist mechanism as well as make
email finding harder. `_mail_find_partner_from_emails` tool method is also
tested with multi email as it uses the same heuristic as normalized email
field.
Tests are also added for mass mailing, when having to mail documents that
have a partner with formatted emails / multi-emails, or that have an email
field with formatted emails / multi-emails.
Also restore a test removed at odoo/odoo@afcb734908 while it should have been
updated to state that email addresses containing non-ascii characters are
supported.
Add some tests for tools methods used in various email processing flows.
Unicode tests are also added.
In future commits we will try to make email usage a bit more defensive to
try to lessen issues with that kind of use cases.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@fc8442f133
Part-of: odoo/odoo#134934
Always apply the blacklist in mass_mail composition mode regardless of the
recipient model implementing mail.thread.blacklist or not.
This solves the problem of mail sent to black listed address for model not
inheriting from mail.thread.blacklist.
Technical notes:
- it has been done in mail.compose.message _get_blacklist_record_ids ignoring
the mixin mail.thread.blacklist to avoid model change in stable.
- some tests have one added query because the blacklist is now queried for each
batch mail sends even if the model of the recipient doesn't implement
mail.thread.blacklist.
Task-2834862
closesodoo/odoo#118497
X-original-commit: 263e86114c60650f421354e165364afcd4122461
Signed-off-by: Dufays Pierre-Yves (pydu) <pydu@odoo.com>
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose
=======
The bounce emails aren't stored in Odoo, which can complicate the
debugging of the email sending.
Now, we store this bounce email, and we allow the user to read it from
the interface, so he can easily find the issue when an email sending
fail.
Specifications
==============
The bounce email is stored on the mail notification for standard emails
sending, and on the mailing traces when using mass mailing.
For some email providers (e.g. Yahoo), the "Final-Recipient" header is
not present. Normally, it allows us to retrieve the original recipient
of the email which bounced and then the partner. So if this header is
not there in a bounce email, we take the first recipient of the parent
<mail.message>.
Change the way that we parse the email body, for the bounce email.
For most email providers, the first mail body is the one that contains
the error and the next one contains the parent email body. So, the
current logic might ignore this body for Outlook and Yahoo.
Task-2116296
closesodoo/odoo#105923
Related: odoo/enterprise#34051
Signed-off-by: Warnon Aurélien (awa) <awa@odoo.com>
Add possibility to add a message when adding or removing a blacklist entry
for both mail and phone_validation (used for SMS). This replaces the
``action_remove_with_reason`` method.
When updating active flag this is added as a tracking note to avoid having
several messages. Indeed message is concatenated with the tracking itself
instead of adding message for tracking + a message for the log itself. When
a new record is created, a note is logged.
Task-2710804 (Mail: Clean MailThread API)
Prepares Task-2150462 (Mass Mailing: Unsubscribe flow refactoring)
Part-of: odoo/odoo#106568
Add a model and a test allowing to test the seen list using raw SQL based
on partner_id field. Test indicates a not stored partner_id field currently
crashes beyond redemption.
Task-2852943
X-original-commit: a15bda3bed339a0ac422a3c82ec9cc295a11a785
Part-of: odoo/odoo#94532
Only the group_salesman can see the opportunities statbuttons
Fix a few not-smart mass_mailing tests:
- running the mass mailing queue processes the messages as the user
running the queue, and will try to send the demo messages, depending
on demo data (when re-running the tests) the test user may not have
the accesses required; sending just the one test email avoids that
issue
- the leads kpi assumes the user has access to leads, that ain't
necessarily the case
For the second issue, when sending statistics if the mailing has a
user `_prepare_statistics_email_values` should be called with that
user as "current" (cf `_action_send_statistics`), so we can just work
off of the current env, though it might be a good idea to eventually
check that.
closesodoo/odoo#89774
X-original-commit: 39af865800c6752b60171f16d6056ceb3c5a1636
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Jinja as a templating engine was problematic in differents respect:
- introduce external dependency to Odoo (less controll)
- add another templating mechanism in the stack
- specific feature in qweb cannot be reused
- difficulty in rendering easily editable templates
- more knowledge required with no betterment
By replacing jinja with qweb we can now build tools to edit a qweb
that will work with the previously jinja encoded document
(essentially `mail.template` records).
There is a catch however. Some email fields (eg. email_to) used jinja
syntax for rendering dynamic variables (ie. ${object.something} and
${object.something_that_should_not_be_escaped | safe}).
We still want user to use dynamic variables for some char fields (eg.
subject, from, to, ...). We made a new rendering engine called
"inline_template" that will render an expression enclosed by `{{` and
`}}`.
To be able to edit the templates from the backend interface, a
plugin to the Odoo editor has been made for seamlessly edit the
document.
This qweb plugin includes:
- make dynamic variables (eg. `<t t-out="variable"/>`) not editable
(for preventing the user to shoot himself in the foot)
- group and hide related logical branching (ie. t-if, t-elif, and t-else)
in order to see only one at once
- a floating select input to switch visibility of a particular logical
branching
Task-27033
X-original-commit: odoo/odoo@68182baff4
Part-of: odoo/odoo#77377
PURPOSE
Improve modeling and performances of mass mailing by manually updating trace
status instead of using complex computed fields and lessening fields usage.
Remove some fields and keep only relevant metrics to simplify and clean trace
model.
SPECIFICATIONS
In this commit we clean the way trace state is managed. Currently it is a
computed field based on several datetime fields. However this generates a
lot of noise in the table as well as unnecessary computation
* there are several columns (one for each state) storing datetime at
which status was reached. Generally only 2 or 3 contain relevant
information;
* state could be set in code directly to avoid a computed field based on
many triggers;
* recomputing it each time a date changes is not necessarily necessary;
* state value can always be updated manually as this is main done through
some automated server update (mailgateway, link clicks, ...);
As trace states and its triggers should not be updated manually it is better
to synchronize it in code flow. When there is an exception or update done
through sending or gateway status is updated as well accordingly. Various
datetime fields are also updated at the same time. In order to align with
notification model mail and sms trace status are updated to a classic field.
Only last status update is now kept as there is no need to store the entire
history of status change.
We keep only a datetime for relevant metrics: open, reply and click. Other
datetime bring no real value. Knowing when a trace was in error or bounced
is not necessary. Indeed exception generally indicates a server issue (at
sending), cancel indicates a data issue (at sending) and bounce depends on
customer email server.
Status update is removed as using write_date is sufficient. Once created
traces are updated only when an external event occurs (opened, replied, ...).
It allows to simplify trace model.
We also rename ignored field into canceled to match naming use through mail
and sms.
QUERY COUNTERS
This change has some positive update on query counters when sending mailings
as traces have less unnecessary status update compared to priori this change.
LINKS
Task ID-2377974
Community PR odoo/odoo#61467
Enterprise PR odoo/enterprise#14633
Upgrade PR odoo/upgrade#1907
RATIONALE
Currently there are differences between mail and sms error management
especially when sending them in batch (mass mode). Moreover cancel
(ignored) and error (exception) states meaning is not clear. Finally
some failure types management between mail and sms can be cleaned.
SPECIFICATIONS
Meaning of ignored / error we want to enforce now is
* error: there was something wrong at sending and user has an action to
perform, i.e. server failed -> check its logs;
* canceled: invalid recipients due to contact information or mailing
configuration (blacklist, opt out, void or invalid email or phone number).
In indicates issues linked to records themselves;
In this task we also add failure information granularity on mailing traces
linked to email like what is done currently on SMS. This can be related to
mailing (blacklist, optout, duplicates) or related to recipient (no recipient,
incorrectly formatted).
We also correctly distinguish optout from blacklist when sending SMS.
SPECIFIC USE CASES
* recipient without email / number: mail / sms is set as canceled, trace is
ignored;
* recipient with invalid email (no @) / number (formatting impossible):
mail / sms is set as canceled, trace is ignored;
-> we now distinguish when possible a void email from a wrong email using
a newly-added selection key (mail_email_missing);
* recipient with email / number blacklisted: mail / sms is canceled, trace
is ignored;
* recipient with email / number that optouted from mailing: mail / sms is
canceled, trace is ignored;
* recipient with email that bounces: mail is sent and will be set as bounce
when receiving bounce in gateway; trace follow same path;
* recipient with number that bounces: not supported as currently no support
of bounce through IAP;
* mail server error, IAP error: mail / sms is set as exception / error,
trace is set in exception;
This means we introduce new failure types on mailing.trace model to reflect
those failure types
* ``mail_missing``: missing email (different from wrong value);
* ``mail_bl``: blacklisted;
* ``mail_optout``: optouted;
* ``mail_dup``: duplicated email skipped during mass email send;
We introduce ``sms_optout`` on SMS and trace models as it was merged with
blacklist previously. Now both errors are distinguished.
LINKS
Task ID-2377974
Community PR odoo/odoo#61467
Enterprise PR odoo/enterprise#14633
Upgrade PR odoo/upgrade#1907
The optimization 2ccf0bd0dc does not take into account that to
"unblacklist" a user, you have the archive the mail.blacklist record so
only mail.blacklist active records need to be taken into account.
Added test without the fix fails with:
AssertionError: False is not true : MailTrace: email
test.record.02@test.example.com (recipient res.partner(), state:
sent, record: mailing.test.blacklist(166,)): found 0 records (1 expected)
opw-2536304
closesodoo/odoo#71277
X-original-commit: 965c67cb16d2a4ac10d35a01da3af311867a47e6
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
PURPOSE
Remove hardcoded list of models on which mailing is possible. Indeed it is
not modular and not really smart with enterprise code not being reachable
in community.
SPECIFICATIONS
Replace hardcoded list of models on which mailing (both mail or sms) is
possible by a computed searchable field on ``ir.model`` based on a class
attribute.
It allows to cleanly define models having mass mailing capabilities and add
this attribute in bridge modules (when existing) or directly on base model
definition to avoid bridge modules.
In this commit we introduce a basic ``_mailing_enabled`` class attribute
activating mailing on model.
Mailing models may also have a ``_mailing_get_default_domain`` method allowing
to define a custom default domain when sending a marketing mailing on records
on this class.
Mailing models can now define a ``_mailing_get_opt_out_list(_sms)`` method
allowing to define custom behavior to fetch opt-outed records. Instead of
defining a model-based behavior on Mailing itself, it now calls the model
defined one. We still have two methods, one for mailing and one for SMS
opt out computation as it relies on different underlying models and fields.
LINKS
Task ID-2431217
COM PR odoo/odoo#67322
ENT PR odoo/enterprise#16876
UPG PR odoo/upgrade#2236
Propagate reply_to radio keys (update and new) to mass mailing in order to have
a coherent naming (was thread and email). This naming is also coherent with
gateway naming (message_update and message_new).
LINKS
Task ID-2117639
COM PR odoo/odoo#40931
ENT PR odoo/enterprise#17941
UPG PR odoo/upgrade#2419
Purpose of this commit is to add tests related to answers through mail gateway
when having reply-to mode of mailing set to either updating existing threads
(aka replying on mailed records), either creating new threads (aka using the
reply-to alias configuration).
LINKS
Task ID-2117639
COM PR odoo/odoo#40931
In this commit we try to simplify tests organization by
* merging mailing_internals and mass_mailing tests, as they globally handle
mailing-related tests (blacklist, opt-out, gateway update through replies
and clicks);
* move wizard (mailing.test) tests in their own file as they target a
specific model;
* add some loggers to reduce noise;
* add some users wrappers to test using real users;
LINKS
Task ID-2117639
COM PR odoo/odoo#40931
Thibault Delavallée