Also impacts mass_mailing_sms, test_mail_sms,
test_mass_mailing
This PR adds support to receive sms delivery reports.
Before this PR, an SMS was considered 'sent' when successfully
handled by the third party. The user couldn't know if/when an
SMS was actually sent for delivery or delivered to the
recipient's device.
This was similar to the behavior for emails as delivery reports
are not commonly used (and not supported in Odoo).
With this work, the SMS `pending` state is introduced in mail,
mass_mailing, sms and mass_mailing_sms contexts although only fully
used in the latter two modules (+tests of course).
Because of the huge cost related to upgrading very large existing
databases, the following compromises were made:
1. An email and sms notification/trace SENT means DELIVERED.
Those that are sent but NOT DELIVERED are PENDING.
The difference between email and sms traces reinforced with this PR
is that an email sent will be counted as "sent" ~ "delivered"
unless an error is returned for emails while for SMS it can only be
reached if a delivery report is received.
2. The Link between an SMS uuid (shared with trusted parties) and
the tracking records (notifications or traces) is done via an
explicit relationship table (sms_tracker) instead of via a new field.
This however allowed to nicely concentrate the state update logic.
A `process` state is added to represent an intermediate
step in the sending process, such as held at IAP for SMS.
A few adjustments are also included to update for IAP api v3.
Also, adapts and includes new tests.
Task-2560666
Part-of: odoo/odoo#133392
PURPOSE
Globally improve usability and features given by mailing portal about exclusion
list and opt-out management.
SPECIFICATIONS
Add unsubscribe headers to email sent by mass mailings. As emails are already
parsed to change the generic 'unsubscribe_from_list' link to an email-specific
link we can add the List-Unsubscribe header at the same time. Add header for
post 'List-Unsubscribe=One-Click' to avoid one-click unsubscribe when readers
crawl email links.
Task-2150462 (Mass Mailing: Improve subscription management)
Part-of: odoo/odoo#86084
Allow partner to unfollow a document from a follow up email of that document
through an unsubscribe URL in the email even if not connected.
It works for internal user for follow up on any document and for any partner on
follow up of document tagged as authorizing being unfollowed by any partner (
slide.channel and slide.slide).
Technical note:
The unfollow block is rendered in mail_thread and updated for each recipient in
mail_mail. We don't render it in mail_mail because we don't have the language
of the message at that point.
Depending on the partner and the related document, the unfollow block is:
- either removed if the user cannot unfollow the document (for example if it
doesn't follow it)
- or updated with partner and document information + a security token
Task-3061864
Part-of: odoo/odoo#107978
Bug
===
Since 6185f14807 we check the <mail.mail>
existence before marking the <mailing.trace> as opened, but since
57ae1b9b8b61f5f4719a8a81e9d0d21fab58cfda we remove the <mail.mail>
automatically when we send them.
The result is that this endpoint always raise a 500 error.
To be: the <mail.mail> existence shouldn't be checked in this endpoint
(the token is valid for the raw integer id).
Task-3234519
closesodoo/odoo#115842
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
RATIONALE
When sending a ``MailMail`` some data preparation is done. This is done
directly in ``_send`` and in sub-methods. As a given MailMail may lead to
several emails being sent we have to go from a mail to a list of emails
to send :
* one email for ``email_to``. They all receive the same email, as those are
just a list of emails to contact and no specific post-processing is done;
* one email for each partner in ``recipient_ids``. It enables a partner-based
update of the body, for example for links or traces in mass mailing;
SPECIFICATIONS
In this commit we move code so that all preparation is done in a sub-method
``_prepare_outgoing_list``. That way it can be cleanly overridden to add or
modify values before sending the actual emails.
This commit does not change behavior and calls done to ``build_email``. This
will be updated in the next commits, notably to improve "cc" management.
Task-2710804 (Mail: Clean MailThread Posting API)
Task-3093268 (Mail: Stop spamming CCs)
Prepares Task-2684479 (Mail: Better report errors when sending emails)
Part-of: odoo/odoo#99482
PURPOSE
Overall cleaning of subscription and exclusion management code from portal.
SPECIFICATIONS
In this commit we rename methods generating tokens (for recipients and report
management). Otherwise it is hard to distinguish them as naming is not really
convenient to use
* _unsubscribe_token -> was actually generating token for a given recipient
on a given mailing;
* _get_unsubscribe_token -> was actually generating token for mailing
users to deactivate D+1 KPI reporting;
Hash for mail tracking is also moved in its own method to avoid having the
same code twice. Better delegate it to a tool sub method.
Task-2710804 (Mail: Clean MailThread API)
Task-2669037 (Mass Mailing: Subscription page code cleaning)
Prepares Task-2150462 (Mass Mailing: Unsubscribe flow refactoring)
Part-of: odoo/odoo#106568
Purpose
Fix bugs in link replacement (html and text version)
1/ The replacement of urls in text (not html) was buggy when
* a string includes multiple urls, and
* `base_url` is one of them and not the first
* Also in SMS marketing when adding the sms `id`
When it came to base_url's turn, the replace function for the content would
replace the `base_url` part of a previously shortened url instead of the
distinct `base_url` link.
2/ Ampersand character
A) The ampersand character prevented replacement of an url inside a `Markup`.
Previous tests passed as the tracker was created but the url was not replaced.
B) The ampersand character was not recognized as part of an url for simple
strings.
3/ Replacement of already short links
A faulty logic made it possible to replace "/r/" urls when no "blacklist" was
passed.
Commit
* Ensures that only the base_url link would be replaced instead in this case.
* Adds support for urls with "&"
* Does prevent shortening short urls
* Improves link conversion performance (regex + avoid duplication when there
are several occurrences of the same url)
* Adds multiple unit tests
Task-2783844
closesodoo/odoo#91164
X-original-commit: 61026806d831b49a7cf5c44ffef3d762986781ce
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
PURPOSE
Improve modeling and performances of mass mailing by manually updating trace
status instead of using complex computed fields and lessening fields usage.
Remove some fields and keep only relevant metrics to simplify and clean trace
model.
SPECIFICATIONS
In this commit we clean the way trace state is managed. Currently it is a
computed field based on several datetime fields. However this generates a
lot of noise in the table as well as unnecessary computation
* there are several columns (one for each state) storing datetime at
which status was reached. Generally only 2 or 3 contain relevant
information;
* state could be set in code directly to avoid a computed field based on
many triggers;
* recomputing it each time a date changes is not necessarily necessary;
* state value can always be updated manually as this is main done through
some automated server update (mailgateway, link clicks, ...);
As trace states and its triggers should not be updated manually it is better
to synchronize it in code flow. When there is an exception or update done
through sending or gateway status is updated as well accordingly. Various
datetime fields are also updated at the same time. In order to align with
notification model mail and sms trace status are updated to a classic field.
Only last status update is now kept as there is no need to store the entire
history of status change.
We keep only a datetime for relevant metrics: open, reply and click. Other
datetime bring no real value. Knowing when a trace was in error or bounced
is not necessary. Indeed exception generally indicates a server issue (at
sending), cancel indicates a data issue (at sending) and bounce depends on
customer email server.
Status update is removed as using write_date is sufficient. Once created
traces are updated only when an external event occurs (opened, replied, ...).
It allows to simplify trace model.
We also rename ignored field into canceled to match naming use through mail
and sms.
QUERY COUNTERS
This change has some positive update on query counters when sending mailings
as traces have less unnecessary status update compared to priori this change.
LINKS
Task ID-2377974
Community PR odoo/odoo#61467
Enterprise PR odoo/enterprise#14633
Upgrade PR odoo/upgrade#1907
PURPOSE
Add a block that allows recipients to check the email in a browser in case
their mail service provider does not render it well or to see it full screen.
SPECIFICATIONS
A thumbnail has been added in the snippet thumbnails list of the mail editor.
It allows to add an header snippet. This snippet contains a link allowing the
recipient to see the mail in its browser instead of its mail client.
Add a route to display a mailing content. This route should be recipient
dependent as mailings could contain jinja markers. Using the same access token
and check as unsubscribe route this public route allows to display a mailing
content directly rendered for a given customer.
_get_unsubscribe_url method is moved from mail_mail to mailing as it does
not really belong to the mail model. Indeed it is mailing dependent.
A view URL leading to the view in browser route is added. It is managed like
the unsubscribe url: magically avoid shortening, magically rewrite when
sending email.
Task ID-2239327
PR #49886
Purpose of this commit is to improve tracking of opened traces. Notably
a token is added to ensure we do not mess with traces and have unique
tracking URLs.
MIGRATION REMARK
Emails sent before the migration will not be marked as opened anymore after
migration. We recommend to avoid sending statistically important mass mailings
about one week before migrating database. Indeed statistics show that most of
open emails happen within the first week after being sent.
Task 2223146
closesodoo/odoo#49139
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
State field on trace model is computed based on other (datetime) fields.
It seems therefore unnecessary to force its value.
LINKS
Prepares task ID 2238597 (notification and trace models cleaning)
Task ID 1906925 (mass mailing tests cleaning)
PR #50169
PURPOSE
Move links shortening tools on mail.render.mixin to have rendering and
post procesing tools available on that mixin.
SPECIFICATIONS
Move html and text links shortening methods on mail.render.mixin class.
That way rendering and post processing is done in that mixin instead of
being specific to link.tracker model. Indeed those methods are tools and
not really business related methods.
Future commits will improve the way mail and sms composer use the render
mixin. This commit is therefore both a cleaning and preparing commit.
LINKS
Task ID 1963529
Community PR odoo/odoo#32397
PURPOSE
Move links shortening tools on mail.render.mixin to have rendering and
post procesing tools available on that mixin.
LINKS
Task ID 1963529
Community PR odoo/odoo#32397
From now on mail.mail is considered as a technical model. Indeed people should
not really manually craft mails by hand. Instead various functional flows
should either send mails, either craft mails based on some user input.
We therefore make mail restricted to admin users. Flows creating mail.mail
are updated to use sudo, and ensure it was done in a context that makes
sense to delegate this power to the user.
Task ID 1853147
PR #32243
Methods _get_unsubscribe_url and _get_tracking_url
create url with a db parameter (GET), but it isn't used
in the targeted controller. Then we removed it.
task-2087379
closesodoo/odoo#40568
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose is to prepare future cleanups and speed improvements. In this commit
we use create multi on mail.mail, allowing to batch their creation when
necessary.
Task ID 1853147
PR #39272
PURPOSE
SMS are a powerful marketing tool. For instance it is perfect to announce a
sale or to communicate a coupon code, to welcome a new customer in a fidelity
program, ...
Purpose of this task is to integrate SMS sending in batch in mass mailing. It
will use same mailing objects but sending SMS instead of emails. Some metrics
and flows will have to be slightly updated at the same time.
SPECIFICATIONS
Prepare mass mailing module to addition of mass_mailing_sms.
Mailing model: mailing type
* add a mailing_type selection field;
* mass mailing contains only 'mail';
* synchronize medium accordingly;
* update actions of mass mailing application to add a domain on mailing
type being 'mail';
Mailing contact
* remove is_email_valid field as its purpose is achieved by email_normalized
field coming from address mixin (added by blacklist management);
Mailing contact subscription
* clean a bit fields and views as this should stay a technical model;
Trace model and report: trace type
* add a trace_type selection field;
* mass mailing contains only 'mail';
Various
* clean some bits of code in views, remove old code bits;
* add anchors to ease view inheritance to be able to customize views for
SMS mailings;
* ensure all views in mass mailing filter content on mail type (mailing
and traces being type mail only);
* rename some methods to be more updated with current guidelines, notably
main action methods;
LINKS
Task 1997464
PR #34424
Original SMS addition: Task 1922163 (4287481)
PURPOSE
Mass mailing is currently a bit messy. As SMS will be added as a way to notify
people in mass through SMS let us take this opportunity to somehow clean this
application: organization, light code cleaning, model renaming.
SPECIFICATIONS
Rename mail.mass_mailing model to mailing.mailing. Rationale :
* mailing is now a prefix for mass mailing models;
* mailing.mailing is easier to read / find / understand;
Note that mail.mass_mailing.campaign is not updated as it is likely to be
removed soon and replaced by simple utm.campaign model.
MIGRATION
mail.mass_mailing model -> mailing.mailing
mail_mass_mailing table -> mailing_mailing
LINKS
Task ID 2037906
Preparing task ID 1997464 (SMS addition in mass mailing)
PR #34938
PURPOSE
Mass mailing is currently a bit messy. As SMS will be added as a way to notify
people in mass through SMS let us take this opportunity to somehow clean this
application: organization, light code cleaning, model renaming.
SPECIFICATIONS
Rename mail.mail.statistics to mailing.trace and mail.statistics.report
to mail.trace.report. Rationale :
* mail.mail.statistics is linked to mail.mail model. Soon this model will
hold data related to SMS sending. It makes sense to be broader in the
naming;
* mailing.trace is more inlined with marketing.trace model that is the
marketing automation model using it in marketing automation (enterprise
application);
* mailing.trace is shorter to write;
* mail.statistics.report model should sense to be updated at the same
time;
MIGRATION
mail.mail.statistics model -> mailing.trace
mail_mail_statistics table -> mailing_trace
mail.statistics.report model -> mail.trace.report
fields updated (w column change)
* link.tracker.click: mail_stat_id -> mailing_trace_id
fields updated (no column change)
* mail.mail: statistics_ids -> mailing_trace_ids
* mail.mass_mailing: statistics_ids -> mailing_trace_ids
LINKS
Task ID 2037906
Preparing task ID 1997464 (SMS addition in mass mailing)
PR #34938
Multi is the default api for methods, it is not necessary to explicitly
decorate methods with it, adds clutter and most people use it because
they see that the rest of the code uses it.
Done with `find . -type f -name '*.py' | xargs sed -i '/@api.multi/d'`
Before this patch, this method was altering the signature of its parent method, moving the position of `failure_type` to 1 and making the other 2 arguments kw-only.
It seems this accidentally didn't break anything because all calls happened to be done in kwarg mode. However, it's very possible that a downstream module that is not based on `mass_mailing` and makes positional calls gets broken when `mass_mailing` is installed.
The fix is to respect original method signature.
closesodoo/odoo#34648
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The default value for failure_type was the 'NONE', instead of being
simply False/NULL. This represents a significant waste of tablespace
with no added value.
Switching to true NULL value for representing the absence of error
also save some precious time during database upgrades on databases with
large numbers of emails.
While fixing the values for failure_type, also fixed a few typos and
untranslated strings.
See original feature at 133eeb1bbf
When a mail is in failure state, a red envelope appears next to the message
in a thread but it was difficult to send the mail again.
This tasks will allow users to send mail again easily, or mark notification
as cancelled if the user want to ignore this failure.
A notification will appear in sender systray while mail are in failure.
Task: #46158
PR: #24628
If the `web.base.url` contains a trailing `/`, the replacement of the
`/unsubscribe_from_list` link won't work since the string to replace
will be `my_url//unsubscribe_from_list` instead of
`my_url/unsubscribe_from_list`.
Fixes#24731
opw-1848572
When installing the website with a lang different than the one set
on the user, the button unsubscribe in the mass mailing snippets
didn't work because the unsubscribe link contains the code of the
language. The function send_get_email_dict in model mail.mail didn't
expect this behavior and so couldn't set the right unsubscribe link
in the mail.
opw:1850696
Purpose is to lessen query count by avoiding to browse and/or fetch data when
not necessary. This commit does two main things
* first one is to avoid checking for existing statistics in mass_mailing
where only checking existing of a mass_mailing on a mail to send is
sufficient. It allows to avoid fetching data from statistics table;
* second one is to rewrite part of the post_process code called after sending
emails. Using list comprehension and mailing_id existence on mail_mail
allows to save several queries on standard post flows;
Most tests benefit from this code update. On overall community runbot about
3K queries are saved with those small optimizations.
Send methods of mail.mail model are set private and simplified. Those
methods have been written when a lot of overrides were necessary to
generate notification emails. Since then the flow has changed but
methods have not been updated completely. This commit therefore simplified
a bit the call chain and parameters.
This commit is purely technical and does not hold any functional change.
Purpose is to clean code in order to prepare future changes in mail
application.
Mail thread now holds a method to replace local links by global links.
Let us therefore use it in mass mailing as it does basically the same
job. Adding base tag is not necessary anymore since there are not
local links or images anymore.
In Python 3, all of these were "consolidated" under urllib(.request,
.parse, .errors) which is inconvenient.
Since we already have hard dependencies on requests and
werkzeug(.urls, which is a backport of Python 3's unicode-aware
urllib.parse) migrate *everything* to that.
A sticking point is urllib2.URLError, those were (mostly) replaced by
the slightly more general IOError which URLError extends.
link_tracker use a regex to determine the content of
a 'href' tag, taking the 'mailto:xxxx@yyyy.zz' case
into account. mass_mailing use another regex that simply
don't.
So when sending a mass mailing containing email link,
the email will be appended by url path of tracking
url '/m/<stat_id>'.
This commit makes mass_mailing use the same regex as
link_tracker (which is in the dependencies of mass_mailing).
opw-727779
Florian Charlier