Prepares the move of ICP to mail before replacing them by dynamic alias
domains. Improve test coverage, notably for edge cases. Continue to make
tests more explicit after odoo/odoo#131492. Some tests are also merged to
lessen number of different tests when possible, notably when only a test
parameter differs (like giving an SMTP session or not).
Clean ICP and mail servers setup in test classes allowing to remove some
unnecessary extra initialization. Cleanup a mock in mail.
Task-3453347 (Mail: Move Mail ICP from Base to Mail)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130750
RATIONALE
This prepares the move of ICP to mail before replacing them by dynamic alias
domains.
Cleanup tests: try to use loops with input / expected to better understand
the various test cases, add some comments, improve logs when failing to
find the right sent email. Rename tests to have a better test structure
when reading logs.
Remove a test from odoo/odoo@3b6c20805c that adds nothing except testing
the test suite.
OTHER ADDONS
In test_mail: have a specific class for testing servers as other data is
not necessary, and it allows to have a tag for it.
In mass mailing: concatenate test about server finding, several tests can be
done in a single unit test.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-3453347 (Mail: Move Mail ICP from Base to Mail)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
closesodoo/odoo#131492
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Rename alias domain and aliases used a test data. This allows to make
them easier to read, follow, grep and understand.
Activate multi-company on alias and gateway tests, ensuring it currently
has few impact on tests.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130768
When using message_post, the body format must be explicitly specified.
If html is expected, a Markup object should be used.
If text is given, the content will be escaped.
Before this PR:
message_post was unaware if the content of a message was HTML or
text. This lead to multiple situation where the content was
incorrectly considered as HTML and led to display errors.
In
self.message_post(body="Hello %s!" % self.name)
if the name contained HTML, it would be evaluated.
In
self.message_post(body="Contact Raoul <raoul@caramail.be>")
the email would not be displayed as considered as unknown HTML and
discarded by the sanitizer
Now each call must explict the type of content.
Use the escape() helper to properly combine Markup and translations.
It would also be acceptable to use Markup() to wrap a static
translation but escape is better as one can not guarantee the content
of a translation.
closesodoo/odoo#111850
Related: odoo/documentation#3612
Related: odoo/enterprise#36728
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
'TestMailCommon' was useless and is replaced by the 'MailCommon' class defined
directly in Mail addon, easing inheritance and imports.
Task-3263512
Part-of: odoo/odoo#117606
There is only one legitimate usage of `_patch_method` (base_automation)
and none of `_revert_method`. The other uses are in tests and they are
all wrong: if the test crashes between the `_patch_method` and the
`_revert_method`, the method will not be reverted.
For the only proper usage of `_patch_method`, move the code to this
place. Correct tests using `_patch_method`/`_revert_method` by calling
the `patch` method of `BaseCase`.
Also, remove the `api.returns` from the `create` method of `BaseModel`
as it is useless and confusing. In fact, we never use it because we
have a special treatment at the RPC level for the `create` method
(see `_call_kw_model_create`).
closesodoo/odoo#110370
Signed-off-by: Raphael Collet <rco@odoo.com>
Fix usage of CCs when sending emails. Currently CCs are added to all outgoing
emails, which leads to unnecessary spam. Recipients are currently computed like
* create an outgoing email based on 'email_to';
* create one outgoing email / partner in 'recipient_ids';
* add CCs to all outgoing emails;
They are now sent only once, either with the email_to if given, either as
standalone. That way partners receive specific emails (due to tracking and
partner specific links management), email_to / email_cc are sent on their own.
Task-3093268 (Mail: Stop spamming CCs)
Prepares Task-2684479 (Mail: Better report errors when sending emails)
Part-of: odoo/odoo#99482
In this commit we add some logs when we fail to evaluate the 'headers' field
on a 'mail.mail' record, used to populate the headers of outgoing emails.
Some tests are added to ensure we do not crash if headers is malformed, and
to cover the feature itself as it was still not really done.
Task-2710804 (Mail: Clean MailThread Posting API)
Task-3093268 (Mail: Stop spamming CCs)
Prepares Task-2684479 (Mail: Better report errors when sending emails)
Part-of: odoo/odoo#99482
If source is something like
"admin@éxample.com" <admin@éxample.com>
candidates founds are
['"admin@\xc3\xa9xample.com"', 'admin@\xc3\xa9xample.com']
and the first one raises an error because of "".
Malformed addresses should be ignored.
opw-2982426
closesodoo/odoo#102353
X-original-commit: 35ad2dd630a8ed173c6a9275585eac46b9a19363
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
Purpose
=======
Historically, we use a char field on the <mail.mail> to match the field type
on the <mail.template>. But even it's useful to have a char field on the mail
template (which can contains QWeb code or Jinja previously to Odoo 15.0) it is
not that useful for the <mail.mail> to keep this char type, because the value
is already rendered. Moreover this forces to have some code convert and store
datetime values using the standard format and in UTC.
We now correctly use a datetime field, as all other fields of that kind in
Odoo.
Task-2826699 (Mail: use datetime for scheduled_date mail field instead of char)
Prepares Task-2207626 (Rating: Delay rating notification to ease feedback)
Part-of: odoo/odoo#95623
Co-authored-by: Thibault Delavallée <tde@odoo.com>
Purpose of this commit is to add tests on ``failure_reason`` usage of
notification model. Tests already exists for MailMail and other fields
but not that specific one. Followup of odoo/odoo@d895f3514e
New tests are added to check the usage of email_to and email_cc when
sending emails. Notably wrong usage of email_cc (lost formatting, copy
of all sent emails) is asserted to be fixed afterwards.
A performance test about batch sending is also added.
Prepares Task-2684479 (Mail: Better send error storage and display)
Part-of: odoo/odoo#96223
``scheduled_date`` field of ``mail.mail`` is a char field since its addition
in 2015 (see odoo/odoo@364b4ba06d). As its first usage was in combination with
mail templates, a char field was used to simplify its implementation.
However this technically allows to store whatever value in that field. Using
it in a filter with a datetime argument is quite strange. A workaround is
to try to parse as much as possible the inputs, remove timezone information,
try to localize it, and have it in regular server format to enable filtering
on it.
We consider value should be set in UTC. If we have a specific timezone set
on the input we localize it to UTC. Otherwise we consider the input was done
in UTC, as all datetime fields. It is the role of the business code generating
mail.mail to either give the timezone, either already convert into UTC.
This might solve the following bug
Step to reproduce:
activate the developer mode
go to settings - technical - emails
create a new email and set the Scheduled Send Date in the future
run the scheduled action Mail: Email Queue Manager
Current behavior:
the email is sent and the action does not consider the filter
Expected behavior:
the email is not sent and will only be sent when the scheduler detects
that scheduled_date exceeds the current time.
Task-2833300
opw-2823106
closesodoo/odoo#94937
X-original-commit: 5c113cb9d54e051132a9a50deb2fcf61f9c9dc2c
Signed-off-by: Warnon Aurélien (awa) <awa@odoo.com>
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Bug
===
When the admin open an email, he might get an error message if he can
not access one of the attachment of the email.
Now the admin is able to view / edit the attachments he has access to.
Task-2464212
Part-of: odoo/odoo#78734
PURPOSE
Add tests related to mail failure management and its effect on mail.mail
and related mail.notification records.
SPECIFICATIONS
Test wrong configurations when sending emails (email_from, email_to) and/or
use mocks to enable sending issues (SMTP issues, Memory errors, assertion
errors, ...) that may happen when sending emails.
Add unit tests for as much use cases as possible to assert current behavior
even if some tests show that it is not always accurate. Notably
* invalid / missing email failure types that are not correctly distinguished
even if it should;
* email_from invalid value goes a bit incognito;
* sometimes mail is marked as sent even if it was not sent;
Future commits will (probably) fix or improve behavior while impacting as
less as possible the global behavior of the outgoing gateway that is quite
tricky to modify.
Task-2673913 (TestMail: Cleanup and improve test coverage)
Part-of: odoo/odoo#86393
mail_* = mail, test_mail, test_mail_full, test_mass_mailing
PURPOSE
Improve tooling of mail tests. Use SMTP mockup class available in base tests
in mail tests to have a better and simplified mock tool when dealing with
mail tests. Remove old fashioned code.
SPECIFICATIONS
Include ``MockSmtplibCase`` directly into ``MockEmail`` so that we re-use
existing components for mocking outgoing emails. Also make parameters for
gateway coherent between those two classes (use same test values).
Remove 'sim_error' weird parameter of Mail ``mock_mail_gateway`` and use
standard mock 'side_effect' instead in some specific tests, notably on
SMTP.connect() mock.
Add some assert / tools method and improve existing tools, used for creating
or improving test data. This will be used in future commits when adding new
tests.
Add outgoing mail servers data in mail tests, allowing to make tests closer
to real life use cases.
Remove unnecessary calls to ``_init_mail_gateway`` as it is automatically
done in ``MailCommon.setUpClass()``.
Try to consolidate starting test data, notably with countries, phone numbers
or email in mind to be sure test mail are deterministic.
Task-2673913 (TestMail: Cleanup and improve test coverage)
Part-of: odoo/odoo#86393
When the system broadcasts an email response to document followers,
if the config parameters `mail.force.smtp.from` or
`mail.dynamic.smtp.from` are defined, it will rewrite the `From`
address to avoid spoofing the sender's domain.
**NOTE**: As of 15.0, this is based on the `from_filter` setting on the
corresponding ir.mail_server, rather than the abovementioned config
parameters, but the rest of the discussion stands.
For example, if the `mail.catchall.domain` is set to `example.com` and
an email response comes from:
"John D" <john@doe.com>
it will rewrite it to:
"John D (john@doe.com)" <notifications@example.com>
This will make sure the system never sends outgoing email for an external
domain, as it has no authority for doing so, and that could
break mail filtering/authentication rules (SPF, DMARC, etc.)
During this "encapsulation rewrite step", both the original Sender name
and their email are preserved, and put into the quoted "name" field of
the rewritten address. It seems sensible to preserve as much information
as possible about the original sender.
Unfortunately, the inclusion of the Sender email in the final name makes
it appear to some inbox providers as if the message is trying to
deceptively impersonate another person (as many phishing schemes would).
As of November 2021 GMail at least does this, and will hide the name in
the UI when it happens. It will keep only the rewritten email, which is not
very useful in the case of a notification (even though it's more
technically correct, of course).
This patch removes the original email from the rewritten notification,
keeping only the name, considering that the email is not the most
important part, and it's better to have one of the two than none.
So after the patch, the rewritten address is now:
"John D" <notifications@example.com>
When there is no name in the original address, we keep only the local
part of the email, to avoid the same display issue. The recipient will
have to identify the sender based on the context / past messages.
closesodoo/odoo#81807
X-original-commit: 3c65ec5a8191a392980ceb0a8c584767eae405f1
Signed-off-by: Olivier Dony <odo@odoo.com>
RATIONALE
Prepare code cleaning and optimization in mail, mass_mailing and SMS by
cleaning models for readability and code complexity and footprint reduction.
SPECIFICATIONS
Purpose is to prepare future changes by easing its grep. Notification is too
much heavily used through the codebase and finding it was quite hard among
other noise.
Rename ``notification`` field to ``is_notification``.
LINKS
Task ID-2377974
Community PR odoo/odoo#61467
Enterprise PR odoo/enterprise#14633
Upgrade PR odoo/upgrade#1907
PURPOSE
Be aligned with mail_mail model naming as well as convention used in SMS
application (sms_sms_id, sms_sms_int) and mass mailing application (using
mail_mail_id on mailing.trace model).
LINKS
Task ID-2377974
Community PR odoo/odoo#61467
Enterprise PR odoo/enterprise#14633
Upgrade PR odoo/upgrade#1907
Co-Authored-By: Rémy Voet <ryv@odoo.com>
Co-Authored-By: Thibault Delavallée <tde@odoo.com>
PURPOSE
=======
We want to increase the score of the emails sent by Odoo and we want to
avoid them to be marked as spam by the mail clients (gmail, outook...).
SPECIFICATIONS
===============
From filter
-----------
Add a new field on the "ir.mail_server" which is "from_filter". This
field defines the email address for which the outgoing email server can
be used.
The "from_filter" can either define an email address or a domain name.
Use the system parameter "mail.default.from" which allow us to define
a default email address which is used to encapsulate the emails
(default: notifications@<catch.all.domain>).
Mail server priorities
----------------------
When sending an email, we read the FROM header and,
- We first look for a mail server which match the entire mail FROM
in that case, we do not change the email header (not needed)
- If not found, we search a mail server which matches the domain name of
the mail from (do not need to change the headers in that case)
- If not found, find the mail server linked to the "notifications"
email (defined in the system parameter). Then change the FROM header
to the notification email, and put the old one in the name part of
this header.
E.g.
Initial mail from: "Admin" < admin@example.com >
Final mail from: "Admin (admin@odoo.com)" < notifications@odoo.com >
- If no notification email is configured or if no mail server are
found for the notification email, fallback to the old system and
spoof the FROM header. In that case we do not have the choice if we
want to send the email, he will probably be marked as spam.
Sending method priority
-----------------------
In the mail server models, we defined some priorities,
1. Forced SMTP session
2. Forced mail server
3. Try to find the best mail server (see "Mail server priorities")
4. If not found, read the odoo-bin arguments
Bounce
------
As there's no standard for bounce address, we put it in the envelope
(smtp_from). But in some case, it might be considered as spoofing. So,
we use the bounce address ONLY if the mail server is configured for the
entire domain name.
One behavior which might be broken is the following; we send an email as
"std@gmail.com" and the bounce address is on the domain "odoo.com".
Before we received the bounce notifications but we were spoofing the
local part and the domain.
Now
- if a mail server is configured for GMAIL, we do not use the bounce
address (and we might not receive the bounce notification)
- if no mail server is configured for GMAIL, but one is configured for
"odoo.com"
- the FROM header will be "notifications@odoo.com"
- the FROM envelope will be the bounce address
=> In this situation we are spoofing only the local part of the email
but it's allowed as the mail server is configured for the entire
domain name
LINKS
=====
Task-2367946
odoo/odoo#61853odoo/upgrade#1903
Starting from this commit bounce addresses set in emails will not use any
plus addressing. They will always be ``bounce_alias#@alias_domain`` and
not ``bounce_alias+<mail_id>-<mail_model>-<mail_res_id>@alias_domain``.
Reasons are
* this plus addressing adds information we do not use anymore since a long
time as bounced messages are found using their message ID and not any
information coming from this bounce address;
* this generates unnecessary noise;
* not all email providers really support plus addressing as a mean to
generate "fake" additional email addresses;
This is the followup of odoo/odoo#71244 where a solution to avoid plus
addressing was done in table. We can now safely remove this feature in
master as a cleaning step.
Task ID-2577328
closesodoo/odoo#73363
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Since odoo/odoo@f4524f03c3 plus addressing is not used anymore
for handling bounces. Indeed it relies on references / in reply to to find
original message that bounced. It is therefore not necessary to enforce the
use of plus addressing.
As some provider do not support plus addressing as a way to contact left-part
of email with sub-informations people should have a way to deactivate plus
addressing used in bounce aliases.
To preserve backwards compatibility for stable versions old behavior is
retained unless a new `mail.bounce.alias.static` ICP is set with a truthy
value.
Fix https://github.com/odoo/odoo/issues/71242 by dropping requirement of plus addressing.
@Tecnativa TT29827
Closes#71242
Task ID-2547347
PR odoo/odoo#72347
X-original-commit: df2d955bf41b01556e1b84bb1204aac045c95a63
RATIONALE
A bit of history. Link between a message and its recipients has been added
at first mail refactoring towards a Chatter / Discuss feature. It was done
in v7 at d64f3c9783 with the base addition of mail notification model (lots
of commits follow that one but that's the first one about notification).
Due to some people thinking that it was unnecessary to keep a model for
notification it has been removed in v9 at 88b8cd0587. Notification table was
renamed from mail_notification to mail_message_res_partner_needaction_rel.
It was proven to be a mistake even if those "some people" were warned and
model made its way back to Odoo in v10 at 72dfcae2a4 . Table name mail_message
_res_partner_needaction_rel was kept to ease migration and backward
compatibility.
It is now time to complete the circle and rename it to mail_notification.
SPECIFICATIONS
Rename ``mail_message_res_partner_needaction_rel`` to ``mail_notification`` .
RIP JEM.
Never forget.
LINKS
Task ID-2477444
Prepares Task ID-2377974 (trace management cleaning task)
Prepares Task ID-2070632 (channel members main task)
Prepares Task ID-2419762 (channel members followup task)
COM PR odoo/odoo#67382
UPG PR odoo/upgrade#2245
Co-Authored-By: Rémy Voet <ryv@odoo.com>
Co-Authored-By: Thibault Delavallée <tde@odoo.com>
It has been a recurrent request from customers to be able to send email
messages to email addresses containing non-ascii characters. [IDNA] is a
domain extension to allow unicode characters in domain names. [SMTPUTF8]
is a SMTP extension to allow unicode in any header.
IDNA defines the [punycode] encoding which translates unicode to an
ascii representation. This encoding MUST be used to encode domains.
SMTPUTF8 is an SMTP extension that allow utf-8 in all headers on the
envelope.
[IDNA] https://tools.ietf.org/html/rfc5890
[SMTPUTF8] https://tools.ietf.org/html/rfc6531
[punycode] https://tools.ietf.org/html/rfc3492
Task: 2116928
opw-2229906
opw-2248251
closesodoo/odoo#47709
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
This reverts commit 976e560a87.
Even if it didn't looked like a bad idea, this need some more thinking.
This new version of query count creates random failure of runbot builds.
closesodoo/odoo#43820
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
- take advantage of runbot multi-build capabilities
- get similar result when running them locally during dev
- better detect when other modules add extra queries
Query counts are split in the base value (testing with just test_mail installed)
+ the extra modules overhead.
Part of task-2178641
closes odoo/odoo#43666
Pr: #43666
Signed-off-by: Alexandre Kühn (aku) <aku@odoo.com>
From now on mail.mail is considered as a technical model. Indeed people should
not really manually craft mails by hand. Instead various functional flows
should either send mails, either craft mails based on some user input.
We therefore make mail restricted to admin users. Flows creating mail.mail
are updated to use sudo, and ensure it was done in a context that makes
sense to delegate this power to the user.
Task ID 1853147
PR #32243
PURPOSE
Currently tools and asserts for mail tests are located inside test_mail
module. It makes difficult to re-use them in apps tests or force them to
write custom quick and dirty tools and asserts.
SPECIFICATIONS
Update tests to new tools / asserts / helpers / classes defined in mail
and test_mail. Notably
* ``BaseFunctionalTest`` class is replaced by the new ``TestMailCommon``
pimped one;
* ``assertNotifications`` is replaced by ``assertSinglePostNotifications``
(shortcut for a simple message_post) or ``assertPostNotifications``
(complete asserts involving several messages and notifications);
* correctly invoke ``mock_mail_gateway`` when mocking email sending;
* replace manual check of sent emails or ``assertEmails`` by
``assertSentEmails``;
* remove custom mocks / checks and replace them by now standard mocks
and assertions (if any);
In this commit we update: test_mail_followers, test_mail_mail,
test_mail_template.
LINKS
Task ID 2068986
PR #38070
A patched method was not unpatched. Which triggered issues in a staging
master branch trying to untie a bit mail tests.
closesodoo/odoo#40099
X-original-commit: 5dbf4c56765c3661eb2dca24e90bc0b420918e9a
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
PURPOSE
In order to rewrite some tests, remove low-level tests and add coverage
first step is to reorganize a bit test_mail content. Several tests cases
are spread among several files and finding back some feature coverage
tests is not easy.
SPECIFICATIONS
Quickly clean common class and lessen data creation by default. Offer some
tool methods, notably to create portal users used in some tests but not
all, or to create mail templates.
LINKS
Task 1958697
This is related to revision
71f188082f
This is nice to have a meaningful explanation
for the user to understand why his mail is not sent,
this is better to not interrupt the mail queue when doing so.
Because of the above revision,
when the mail queue iterated on a mail failing because
of an ASCII encoding issue,
the mail queue was interrupted,
and on the next cron call it failed again and again
on the same mail, therefore leading to the mail queue
to never be processed.
Fixes#27804
Purpose of this commit is to use the newly introduced helper to create
test users in a quick way and reduce code duplication. A mail specific
helper tool function is defined based on the standard one defined in test
allowing to pass a custom context. It bypasses mailing features in order
to speedup the creation.
This commit is linked to task ID 1889703 and PR #27526.
Sending email to addresses containing unicode characters
cannot be parsed by python and result in a UnicodeEncodeError.
This PR catch that specific exception and output a meaningful
error message.
Decision has been taken to not trigger any error message
client-side has it would have needed modifications on some
sensitive low-level backend code.
To reproduce:
- Configure odoo so it uses an email server
- Send an invoice to a partner whoose email address contains
some unicode characters (exemple: æøåÇÀ@example.com)
opw-1892409
closesodoo/odoo#27804
Thibault Delavallée