Purpose
=======
The bounce emails aren't stored in Odoo, which can complicate the
debugging of the email sending.
Now, we store this bounce email, and we allow the user to read it from
the interface, so he can easily find the issue when an email sending
fail.
Specifications
==============
The bounce email is stored on the mail notification for standard emails
sending, and on the mailing traces when using mass mailing.
For some email providers (e.g. Yahoo), the "Final-Recipient" header is
not present. Normally, it allows us to retrieve the original recipient
of the email which bounced and then the partner. So if this header is
not there in a bounce email, we take the first recipient of the parent
<mail.message>.
Change the way that we parse the email body, for the bounce email.
For most email providers, the first mail body is the one that contains
the error and the next one contains the parent email body. So, the
current logic might ignore this body for Outlook and Yahoo.
Task-2116296
closesodoo/odoo#105923
Related: odoo/enterprise#34051
Signed-off-by: Warnon Aurélien (awa) <awa@odoo.com>
RATIONALE
Purpose of this commit is to cleanup main post helpers and have a more easy
and understandable way of calling them.
SUMMARY
We now have two main API methods, based on business flow: either posting
on documents, either sending a mass mailing. Indeed those two flows are
different
* post: create message, then launch notification process by taking into
account subtype, followers, ...
* mail: create mails in batch with recipients being based on template or
given partners. No notifications is involved, only maybe traces if a
mass mailing is linked
Delegate QWeb rendering to the render mixin (i.e. _render_template_qweb_view)
in order to have a single point to forge evaluation context and re-use
existing rendering code.
SPECIFICATIONS
Main API helpers are now
* ``message_post_with_source``: (batch) post on records, using an ir.ui.view
(given a record or its xml id) or a mail.template record (given a record or
its xml id). When using a template, a composer is called to post on each
record (as batch post is not yet supported). When using a view, a direct
call to message_post using the rendered bodies is done, one record at a
time.
* ``message_mail_with_source``: send a mass mailing on records, acting like
invoking the mail composer in mass mode. Same arguments are valid, either
a reference to a view, either a reference to a mail template.
Other helpers are
* ``_message_log_with_view``: (batch) log on records, using an ir.ui.view
to render the body using QWeb (no notification process);
* ``_message_log(_batch)``: (batch) log on records (no notification process);
* ``message_notify``: notify partners on records (creating notifications
specifically for some people while message itself is not displayed in
chatter);
Code migration
* ``message_post_with_template`` in "mass mode": use ``message_mail_with_source``
and set the template record as source;
* ``message_post_with_template`` in "comment" mode: use ``message_post_with_source``
and set the template record as source;
* ``message_post_with_view``: its main usage was to post on a document, in which
case it generally can be replaced by ``message_mail_with_source`` using
the view reference as source;
Task-2710804 (Mail: Clean MailThread Posting API)
Part-of: odoo/odoo#99482
RATIONALE
Purpose of this commit is to be explicit in subtype chosen when invoking the
message composer / calling message_post. As default value may not always be
clear, better be explicit in case the composer default value changes.
SPECIFICATIONS
Add explicit references to subtype when it is not obvious what will be the
final subtype, notably when using helpers (post_with_view or template which
uses the composer that is not crystal clear in its subtype management).
In this commit we also add support of XMLID-based subtype when invoking the
composer. A ``default_subtype_xmlid`` context key is transformed into a
``default_subtype_id``, to be used notably in JS where we cannot easily
use a ``ref``-like statement. Post API now also supports 'subytpe_xmlid'
argument allowing to give the xml id and ease calling the methods.
Use ``_xmlid_to_res_id`` to get directly the ID of subtypes in order to
avoid useless queries from ``ref`` that does an exists.
Also remove useless values given to post API, notably author_id that is by
default the current users' partner.
Task-2710804 (Mail: Clean MailThread Posting API)
Part-of: odoo/odoo#99482
Purpose of this commit is to use the newly introduced feature to log a message
when adding or removing an entry in block list of emails in email marketing
application.
Logging itself is improved, as it now contains
* source of the request (unsubscribe link, or manual ask through the portal
unsubscription page);
* link to the source mailing;
* link to the document that generated the click (a mailing contact, a sales
order, ...);
Task-2710804 (Mail: Clean MailThread API)
Prepares Task-2150462 (Mass Mailing: Unsubscribe flow refactoring)
Part-of: odoo/odoo#106568
PURPOSE
Improve modeling and performances of mass mailing by manually updating trace
status instead of using complex computed fields and lessening fields usage.
Remove some fields and keep only relevant metrics to simplify and clean trace
model.
SPECIFICATIONS
In this commit we clean the way trace state is managed. Currently it is a
computed field based on several datetime fields. However this generates a
lot of noise in the table as well as unnecessary computation
* there are several columns (one for each state) storing datetime at
which status was reached. Generally only 2 or 3 contain relevant
information;
* state could be set in code directly to avoid a computed field based on
many triggers;
* recomputing it each time a date changes is not necessarily necessary;
* state value can always be updated manually as this is main done through
some automated server update (mailgateway, link clicks, ...);
As trace states and its triggers should not be updated manually it is better
to synchronize it in code flow. When there is an exception or update done
through sending or gateway status is updated as well accordingly. Various
datetime fields are also updated at the same time. In order to align with
notification model mail and sms trace status are updated to a classic field.
Only last status update is now kept as there is no need to store the entire
history of status change.
We keep only a datetime for relevant metrics: open, reply and click. Other
datetime bring no real value. Knowing when a trace was in error or bounced
is not necessary. Indeed exception generally indicates a server issue (at
sending), cancel indicates a data issue (at sending) and bounce depends on
customer email server.
Status update is removed as using write_date is sufficient. Once created
traces are updated only when an external event occurs (opened, replied, ...).
It allows to simplify trace model.
We also rename ignored field into canceled to match naming use through mail
and sms.
QUERY COUNTERS
This change has some positive update on query counters when sending mailings
as traces have less unnecessary status update compared to priori this change.
LINKS
Task ID-2377974
Community PR odoo/odoo#61467
Enterprise PR odoo/enterprise#14633
Upgrade PR odoo/upgrade#1907
USE CASE
Send a mailing to some recipients. Set reply-to of this mailing to an alias
creating records on a model inheriting from utm.mixin. UTM informations on
mailing are not propagated to the newly-created records.
SPECIFICATIONS
When an user answers to a mail of a mailing with an alias creating records
on an UTM enabled model, propagate UTM from the mailing to the new record.
In order to set UTM informations during record creation, "message_new" has
been overridden in mass_mailing. That way it is automatically available on
all models inheriting from utm.mixin.
A test simulating this scenario has been added. It checks if UTMs info are
correctlwy set on the created record by comparing them to the mailing's UTM
info.
Task Id 2210334
closesodoo/odoo#48602
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Consider an email going through the mail gateway. Its headers References
and In-Reply-To are used in mail gateway to find if it is an answer to an
existing discussion thread.
Notably in mass mailing it is used to set mailing traces (called statistics
before 13.0) as opened and replied. However currently only the References
header is used.
In this commit we now use References or In-Reply-To like what is done in
mail gateway.
== 13.2 FORWARD PORT SPECIFIC ==
Code in tests has been updated to use new mockups tools and variables available
in test_mail
LINKS
Task ID 2257717
PR #51445
Forward-port-of: #51319
Forward-port-of: #51247
X-original-commit: 9921e38692e7dc4547afc34acc395da340eca410
Fix a mixup between `bounced_msg_id` that is sometimes tought as a list
of string or directly as a string.
Now `bounced_msg_id` is always a list or `False` if there is no bounce.
opw-2157793
closes#43084closesodoo/odoo#43091
X-original-commit: b0e0cee9f9623d972c4dee26d3536600dc95f276
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules: `email.(charset|header|mime|utils)`.
The new `email.message.EmailMessage` API exposes a much easier interface
to create multiparted emails [1], is capable of doing all the necessary
headers value conversion (RFCs [2045], [2047], [2049]) [2] and get/set
different flavor of payload (text/bytes) in a straightforward way [3].
All headers are structured in a way to support python native types, i.e.
the `Date` header supports `datetime.datetime` objects and automatically
performs the required formatting. The same goes for multi-valued headers
like the `To` header, one can directly set a python list of values, it
will be automatically be formatted according to the RFCs.
The dedicated encoding and decoding functions are no more needed thus
has been removed has part of the refactor. FTR, [RFC2231] is an update
of [RFC2047] and based on tests we've just conducted, both GMail and
Thunderbird now support RFC2231 encoding just fine in all headers:
attachments names, From, etc.
[1]: http://docs.python.org/3/library/email.message.html
[2]: http://docs.python.org/3/library/email.headerregistry.html
[3]: http://docs.python.org/3/library/email.contentmanager.html
[2045]: https://tools.ietf.org/html/rfc2045
[2047]: https://tools.ietf.org/html/rfc2047
[2049]: https://tools.ietf.org/html/rfc2049
[2231]: https://tools.ietf.org/html/rfc2231closesodoo/odoo#35929
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Purpose of this commit is to lessen complexity of stack chain in mail main
methods. Removing decorators is a good start for this.
LINKS
Task 1919267
PR #30597
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
The number of message bounce is incremented each time the email bounce
on a specific email address. To have a correct information, we should
reset to zero this counter if we receive a mail from this address.
Indeed, if we receive an email from an email address, this email is
active and the message bounce number (if > 0) is not relevant anymore.
Only models inheriting form blacklist are impacted by this improvement
in order to limit a bit side effect.
Also, add a check in autoblacklist rule. No need to check the stats
if message_bounce is < 5.
This commit introduces
* ``_routing_reset_bounce`` routing method managing the bounce reset;
* ``_message_reset_bounce`` model method that resets bounce counter
in blacklist enabled models;
LINKS
Related to task ID : 1893155
Linked to PR #33340
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Purpose
* move bounce information detection in message parsing. It allows to have
this information available in various steps of routing instead of having
to manually re-compute them;
* handle bounce in specific methods allowing easy override;
* improve bounce management, notably when detecting a bounce not linked
to the bounce alias configuration;
* better integration with blacklist mechanism;
Specifications
* compute bounce information in ``_message_parse_extract_bounce``.It parses
bounce information and returns a dictionary allowing to update parsed email
values;
* remove override in mass_mailign that basically does what mail already
does;
* manage bounce in ``_routing_handle_bounce``;
* when detecting a bounce, correctly call the bounce management method on
all models inheriting from blacklist;
* correctly update bounce counter;
* bounced mailing traces and automatic blacklist in mass mailing should
be done in ``_routing_handle_bounce``;
* add some tests;
LINKS
Related to task 1893155
Linked to PR #33340
PURPOSE
Mass mailing is currently a bit messy. As SMS will be added as a way to notify
people in mass through SMS let us take this opportunity to somehow clean this
application: organization, light code cleaning, model renaming.
SPECIFICATIONS
Rename mail.mail.statistics to mailing.trace and mail.statistics.report
to mail.trace.report. Rationale :
* mail.mail.statistics is linked to mail.mail model. Soon this model will
hold data related to SMS sending. It makes sense to be broader in the
naming;
* mailing.trace is more inlined with marketing.trace model that is the
marketing automation model using it in marketing automation (enterprise
application);
* mailing.trace is shorter to write;
* mail.statistics.report model should sense to be updated at the same
time;
MIGRATION
mail.mail.statistics model -> mailing.trace
mail_mail_statistics table -> mailing_trace
mail.statistics.report model -> mail.trace.report
fields updated (w column change)
* link.tracker.click: mail_stat_id -> mailing_trace_id
fields updated (no column change)
* mail.mail: statistics_ids -> mailing_trace_ids
* mail.mass_mailing: statistics_ids -> mailing_trace_ids
LINKS
Task ID 2037906
Preparing task ID 1997464 (SMS addition in mass mailing)
PR #34938
Multi is the default api for methods, it is not necessary to explicitly
decorate methods with it, adds clutter and most people use it because
they see that the rest of the code uses it.
Done with `find . -type f -name '*.py' | xargs sed -i '/@api.multi/d'`
Purpose of this commit is to clean some bits of code, notably calls to
message_post/log as well as notification methods. It will ease performance
improvement work.
Small optimization: account: read content after extension check
Parameter cleaning
* use message log with kwargs instead of args;
* remove message post after hook useless parameters;
* remove _notify_email_recipients useless message parameter;
* remove message_notify useless send_after_commit parameters;
* remove message post params matching default values;
Other improvements
* remove message_post commands support for partners and channels;
* only calls message_post with ids list for channels and partners. We
don't support mix of ids and command anymore to simplify code;
* remove support of private discussion in mail.thread adding partners
as recipients, as there is no use anymore;
Related to task 1943901
Linked to PR #32404
Purpose: make some methods from mail.thread mixin private and update their
naming if necessary. Some methods prepare data for business flows and deal
with internal information. Public methods should either use of give some
access to those methods. Computation itself should be keps internal.
In this commit we put message_receive_bounce as private. Future commit(s)
will probably rewrite part of the bounce logic.
Some addons are updated accordingly to the method change.
Related to task ID 1911679
Linked to PR #29483
Fix to commit: a1d6064dcc
Which erroneously put a mass_mailing method in mail,
as mail.mail.statistics is defined in mass_mailing.
Since it does not depend on the records, it been extracted from the
loop.
opw 1890461
closesodoo/odoo#27431
A mass mailing can be considered both as bounced and opened. This can
occur with some email providers, such as Gmail. Indeed, Gmail sends
out-of-office replies to the `Return-Path` address, which is also used
for bounced emails. Therefore, it is not possible to determine
accurately if a message sent to to the `Return-Path` is a genuine
bounced email, or an out-of-office reply.
However, if the recipient opens the email, it is possible to determine
that the bounce was counted incorrectly. In this case, we empty the
`bounced` field. Similarly, we only write on the `bounced` filed if the
`opened` field is empty. This could occur if the recipient opens the
email before the out-of-office reply is received.
As a bonus, we also make sure to avoid updating the status if no route
was found. No route means that the email was bounced, so no reason to
set it as opened.
opw-1869957
In order to avoid sending mail indefinitely to a wrong email address,
Take the mail statistics for the recipient of the last 3 month :
if more than 5 mails bounced (with interval of more than 1 week)
the email is blacklisted.
Currently when people answers an email its statistics are set to replied
updating its replied field. However if the blank gif used to track the
opening fails for whatever reason we could have mail considered as replies
but not opened.
We consider that it is not common to reply to an email people did not
open. Let us therefore set a replied email as also opened.
In case a default value is set for the field `mass_mailing_name` of the
mail composer, the automatic messages sent via message_post_with_* are
considered as a mass mailing. It may also create an AccessError if the
current user is not a mass mailing user.
In the case of the "assign user" email (for auto-subscribed users), the
url wasn't shorten, which resulted in invalid links (opw-752941) (fixed
by 3228694bdd)
Handle bounces using bounce alias directly in mailgateway. Previously bounces
were used only in mass mailing to update campaign statistics. Now the
mailgateway tries to find data from standard delivery status emails. This
data is used to update state of emails sent to customers, to display it in
the Chatter. It is also used to increment the message_bounce counter on the
bounced partner and bounced record, if any and if the field exists.
Previous more generic code that detect bounces is kept. This code is more specific
to standard delivery failure notifications by parsing its content.
Previously Return-Path were composed using bounce_alias-<mail_id>-<model>-
<thread_id> . However this may lead to return adresses not supported by
the incoming mail server. We now form the bounce alias using bounce_alis+
<mail_id>-<model>-<thread_id>. Using a + indicates that the right part of
the address is for information and used to manage the bounce. Only the
left part is used for the email routing, allowing the return email to
effectively land in the alias mailbox.
mail.mail and mail.thread have methods to decode headers. This commit
move them in mail tools. It is indeed simplier to have all email tools
in the same file. Moreover they have been renamed to avoid confusion
with standard library methods: decode and decode_header already exist.
* decode in mail_message is now decode_smtp_header
* decode_header in mail_thread is now decode_message_header
A method to data from references is also moved into tools. A regex was
already defined in tools. Now the method using it is already put in tools.
Mailgateway will now simply call this method.
- Preserved explicit 3rd-party copyright notices
- Explicit boilerplate should not be necessary - copyright law applies
automatically in all countries thanks to Berne Convention + WTO rules,
and a reference to the applicable license is clear enough.
The old-api model._all_columns contains information about model._columns and
inherited columns. This dictionary is missing new-api computed non-stored
fields, and the new field objects provide a more readable api...
This commit contains the following changes:
- adapt several methods of BaseModel to use fields instead of columns and
_all_columns
- copy all semantic-free attributes of related fields from their source
- add attribute 'group_operator' on integer and float fields
- base, base_action_rule, crm, edi, hr, mail, mass_mailing, pad,
payment_acquirer, share, website, website_crm, website_mail: simply use
_fields instead of _all_columns
- base, decimal_precision, website: adapt qweb rendering methods to use fields
instead of columns
- [FIX] bounce regex: too many emails were considered as bounce and therefore
not displayed in the chatter and lost for the communication history. The regex
was not correctly looking for the bounce alias in the email_to.
- [FIX] invite email: replying to the invitation email (invitation as new
follower) now replies to the user sending the invitation.
- [FIX] mass_mailing: added a column to store the id of the original email
in addition to the many2one column. The many2one is set to null when deleting
the original email. As the information is necessary, it is saved on another
field. The many2one is necessary for indexes purpose as the inverse of
a one2many.
std-odoo