PURPOSE
Allow alias domains to be multiple, notably to be used in a multi company
environment where each company has its own alias domain.
SPECIFICATIONS
Now that alias domains are used in Odoo codebase there is no usage anymore
for the old config parameters. So long and thanks for all the fish !
Task-36879 (Mail: Support Multi Domains Aliases)
Part-of: odoo/odoo#76734
PURPOSE
Allow alias domains to be multiple, notably to be used in a multi company
environment where each company has its own alias domain.
SPECIFICATIONS: MAIL.MAIL
Update MailMail to use alias domains. Notably "Return-Path" headers are
now computed based on alias domain when possible, using the recently added
fields on 'mail.message' model for that purpose. Fallback is to use current
company's bounce email when mail_mail creation is done outside of classic
mail flows or without that information.
SPECIFICATIONS: IR.MAIL.SERVER
Update IrMailServer and low-level stack to use alias domains. This has an
impact notably on default values computation for from and bounce emails
* '_get_default_bounce_address' is called when there is no 'Return-Path'
given. Most classic mail flows will set it according to current record
company / alias domain. Fallback when not set is to fallback on current
company's bounce email, computed based on its alias domain;
* '_get_default_from_address' is used in two use cases
* computing a default 'email_from' for outgoing emails when it is not set.
In most classic mail flows it is set based on current user's email. If
not set fallback on current company's notification emails is considered
as a safe bet, replacing the global configuration parameter;
* overriding the 'email_from' of emails that are considered spoofing the
mail server, allowing to wrap the sending into a 'notifications@domain'
generic sender. For those we should try to keep record's information as
it may be called in classic mail flows;
* '_get_default_from_filter' is added in base and overridden in mail to
either use 'mail.default.from_filter' ICP, or use the one defined on
the alias domain. Supporting both is still an option, as its behavior
is implemented for basic email sending, without mail being available.
Those methods are updated to try to support multi domains / multi company
setup. However as those defaults are located ar ir.mail_server level it is
not always easy to have complete environment information, hence fallbacking
on current company's parameters when no better information is provided.
A test about 'mail.default.from' is removed, as it was testing a default_from
outside of catchall domain. It is not possible anymore as default_from is now
part of domain definition. As multi domains is supported, no need to support
exotic configuration like that.
SPECIFICATIONS: FROM MAIL.MAIL TO OUTGOING EMAILS
When sending emails based on MailMail, we now prepares sending groups based
on MailServer, email_from, but also alias domain to which the mail belongs to.
Information about alias domain (e.g. notifications email based on default_from
and bounce email) is propagated to low-level email preparation methods. It
uses the context as it is the easiest way to propagate information to that
level without hacking too much models or calls.
Task-36879 (Mail: Support Multi Domains Aliases)
Part-of: odoo/odoo#76734
PURPOSE
Allow alias domains to be multiple, notably to be used in a multi company
environment where each company has its own alias domain.
SPECIFICATIONS
Make mail gateway support alias domains instead of relying on configuration
parameters. This implies the following changes
* destination alias check is now based on full email by default. Previously
only left-part of aliases were checked. Optionally an allowed list of
domains could be additionally checked. Default from now on is to check
the complete email e.g. 'sales@mydomain.com' != 'sales@mydomain.in';
* detection of direct write to catchall implies checking all domains
catchall emails;
* detection of write to bounce implies checking all domains bounce emails;
* when having to send bounce emails using the bounce alias as mailer-daemon,
find the bounce email from the relevant company;
However we have to ease transition from the old ICP-based model used since
ages to the new domain-based model. Notably a common usage of mail gateways
is to do mail forwarding e.g. forward mail from domainA to domainB without
rewriting destination. It means that e.g. sales@mail.domainA should be
considered as a valid alias equivalent to sales@mail.domainB. This was
working due to left-part only check of destination aliases. In order to
keep this setup working after migration a flag is added on aliases allowing
to keep the detection of those aliases based only on local parts.
In summary: When searching for aliases, mailgateway now either checks for
exact email, either for matching local parts when the flag is active. This
is not the default behavior, as we want a stricter comparison of emails by
default but it will be the default behavior at **migration time**.
The 'mail.catchall.domain.allowed' configuration parameter is kept. It is
used only for left-part check aliases, allowing to limit the scope of the
match.
Task-36879 (Mail: Support Multi Domains Aliases)
Part-of: odoo/odoo#76734
RATIONALE
As multi-company tolerant alias domains will soon replace the usage of
configuration parameters, having them in base then replaced by more advanced
models in mail would be complicated to handle and not useful. Move those
ICP to 'mail' so that all mail configuration is done in that module.
SPECIFICATIONS
Move config parameter used for alias domains configuration in 'mail' module.
Base should be as simple as possible and let mail deal with mail server
complexity.
Move 'mail.{bounce/catchall}.alias' used with 'mail.alias.domain' to make
bounce and catchall emails. Move 'mail.default.from' as it will be integrated
into alias domains in some form.
Note that 'mail.default.from_filter' stays as an ICP in base as it is a
more global default parameter. It is used as default value in 'connect' when
no mail_server is used and no from_filter can be retrieved.
Some tests in 'base' are either fixed, either moved directly into 'mail'.
We now differentiate base behavior (without ICP) from configurable behavior
(with ICP in mail).
Task-3453347 (Mail: Move Mail ICP from Base to Mail)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130750
From filter could be ill-defined, like ' ' or ','. This commit just make
some code more defensive against those values.
Task-3453347 (Mail: Move Mail ICP from Base to Mail)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130750
Split '_get_test_email_addresses' into two methods allowing to generate the
'from' and 'to' when testing SMTP connection. As 'email_to' is always the
same better have a small method for it. Moreover it eases overrides if
some code wants to tune the from / to by overriding only the necessary one.
Task-3453347 (Mail: Move Mail ICP from Base to Mail)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130750
Prepares the move of ICP to mail before replacing them by dynamic alias
domains. Improve test coverage, notably for edge cases. Continue to make
tests more explicit after odoo/odoo#131492. Some tests are also merged to
lessen number of different tests when possible, notably when only a test
parameter differs (like giving an SMTP session or not).
Clean ICP and mail servers setup in test classes allowing to remove some
unnecessary extra initialization. Cleanup a mock in mail.
Task-3453347 (Mail: Move Mail ICP from Base to Mail)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130750
RATIONALE
This prepares the move of ICP to mail before replacing them by dynamic alias
domains.
Cleanup tests: try to use loops with input / expected to better understand
the various test cases, add some comments, improve logs when failing to
find the right sent email. Rename tests to have a better test structure
when reading logs.
Remove a test from odoo/odoo@3b6c20805c that adds nothing except testing
the test suite.
OTHER ADDONS
In test_mail: have a specific class for testing servers as other data is
not necessary, and it allows to have a tag for it.
In mass mailing: concatenate test about server finding, several tests can be
done in a single unit test.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-3453347 (Mail: Move Mail ICP from Base to Mail)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
closesodoo/odoo#131492
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Have all ir.mail.server tests as well as their configuration-related tests
moved into 'test_ir_mail_server' file. That way all tests are contained in the
same file, easing their update.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130768
Rename alias domain and aliases used a test data. This allows to make
them easier to read, follow, grep and understand.
Activate multi-company on alias and gateway tests, ensuring it currently
has few impact on tests.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130768
Purpose
=======
Some SMTP configuration can be used for many domains names. E.G.,
our own SMTP configuration can be used for both "mail.odoo.com" and
"odoo.com". So we want to be able to set a list of allowed domains
name, separated by a comma, instead of a single domain.
Task-3332488
closesodoo/odoo#122234
Signed-off-by: Warnon Aurélien (awa) <awa@odoo.com>
Bug
===
When the from_filter of an outgoing mail server contains a domain
(e.g. company.com), and when the system parameter "mail.default.from"
is a full email address, with a different domain name
(e.g. notification@example.com) we concatenate both value which produce
an invalid email (e.g. notification@example.com@company.com).
Instead, we always give the priority to the from_filter, and fallback
on "noreply" for the local part when needed.
Task-3230917
closesodoo/odoo#115890
X-original-commit: bf02481de9932de2e0246ac36fcf59f37a89b511
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Since python/cpython#18544, unittest mock is not able to properly find
the "odoo.tools.config".
When trying to patch the `options` dictionnary, it leads to
`AttributeError: module 'odoo.tools.config' has no attribute 'options'`
In order to have the tests
working in python <= 3.11, we have to import the config module and patch
the options in place.
Part-of: odoo/odoo#112450
Purpose
=======
On our saas, when users configure a mail server, they can't use anymore
the "odoo.com" configuration (which is stored in the odoo-bin argument).
To allow them to continue using the SMTP CLI configuration, we add a
new "smtp_authentication". When this authentication method is chosen,
all the "connection" fields of the mail server are ignored, and the
connection information are taken from the odoo-bin arguments. So they
can choose the from filter of this SMTP configuration, the priority...
Task-3061882
closesodoo/odoo#106297
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
During tests, when matching ir.mail_server has `smtp_encryption` set to
`starttls`, connect() will crash with an AttributeError as follows:
2022-09-22 07:04:27,882 20445 ERROR tests-150-starttls odoo.addons.base.tests.test_ir_mail_server: ERROR: TestIrMailServer.test_mail_server_send_email
Traceback (most recent call last):
File "/home/odoo/src/odoo/15.0/odoo/tools/misc.py", line 804, in deco
return func(*args, **kwargs)
File "/home/odoo/src/odoo/15.0/odoo/addons/base/tests/test_ir_mail_server.py", line 261, in test_mail_server_send_email
IrMailServer.send_email(message, mail_server_id=self.server_domain.id)
File "/home/odoo/src/odoo/15.0/odoo/addons/base/models/ir_mail_server.py", line 575, in send_email
smtp = self.connect(
File "/usr/lib/python3.8/unittest/mock.py", line 1081, in __call__
return self._mock_call(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 1085, in _mock_call
return self._execute_mock_call(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 1146, in _execute_mock_call
result = effect(*args, **kwargs)
File "/home/odoo/src/odoo/15.0/odoo/addons/base/models/ir_mail_server.py", line 313, in connect
connection.starttls(context=ssl_context)
AttributeError: 'TestingSMTPSession' object has no attribute 'starttls'
This commit adds a dummy `starttls()` method to `TestingSMTPSession` to
gracefully handle such case.
closesodoo/odoo#101962
X-original-commit: ec156901dd4db2fc7a2827b1eb78515f6259699e
Signed-off-by: Julien Castiaux <juc@odoo.com>
Signed-off-by: Xavier Alt (xal) <xal@odoo.com>
mail_* = mail, test_mail, test_mail_full, test_mass_mailing
PURPOSE
Improve tooling of mail tests. Use SMTP mockup class available in base tests
in mail tests to have a better and simplified mock tool when dealing with
mail tests. Remove old fashioned code.
SPECIFICATIONS
Include ``MockSmtplibCase`` directly into ``MockEmail`` so that we re-use
existing components for mocking outgoing emails. Also make parameters for
gateway coherent between those two classes (use same test values).
Remove 'sim_error' weird parameter of Mail ``mock_mail_gateway`` and use
standard mock 'side_effect' instead in some specific tests, notably on
SMTP.connect() mock.
Add some assert / tools method and improve existing tools, used for creating
or improving test data. This will be used in future commits when adding new
tests.
Add outgoing mail servers data in mail tests, allowing to make tests closer
to real life use cases.
Remove unnecessary calls to ``_init_mail_gateway`` as it is automatically
done in ``MailCommon.setUpClass()``.
Try to consolidate starting test data, notably with countries, phone numbers
or email in mind to be sure test mail are deterministic.
Task-2673913 (TestMail: Cleanup and improve test coverage)
Part-of: odoo/odoo#86393
Purpose
=======
Allow a full email address in the system parameter "mail.default.from".
This is useful if we want to send the emails with a domain different
from the domain used to receive them. E.G. a user on our SaaS can
configure an outgoing mail server for Outlook, use his email address as
the default one (so all emails will be encapsulated into his email
address) but keep our default configuration to receive the emails
(so the catchall domain still remains "mycompany.odoo.com").
Task-2738816
closesodoo/odoo#83325
X-original-commit: 21f685d1f58690e1c9c1c6198461652b108e1e18
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
We have our own html2plaintext, already used in lot of use cases instead of
just a few for the html2txt library.
Notably for emails: most emails going through Odoo stack use our simple
html2plaintext to format the body alternative. When no body alternative
is given to ``build_email`` an alternative is built using the library to
remove. Using our own parser allows to have the same results compared to
using ``MailMail.send()``. Difference lies in spaces and new lines as well
as markdown. Our html2plaintext is a bit simple and does not try to generate
Markdown but generates a simple plaintext version.
This also helps solving some issues with depending on that library.
Task-2702034
closesodoo/odoo#82486
X-original-commit: b3b9627b655cd7cb928925affed6cc8d92661e8d
Related: odoo/enterprise#23364
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose of this commit is to add some tests about body alternative done
in build_email when none is given. It uses a library we are about to
remove and testing it is therefore necessary.
Task-2702034
X-original-commit: 92b102c87bbdb9073f794a414a41460e4146acd2
Part-of: odoo/odoo#82486
When the system broadcasts an email response to document followers,
if the config parameters `mail.force.smtp.from` or
`mail.dynamic.smtp.from` are defined, it will rewrite the `From`
address to avoid spoofing the sender's domain.
**NOTE**: As of 15.0, this is based on the `from_filter` setting on the
corresponding ir.mail_server, rather than the abovementioned config
parameters, but the rest of the discussion stands.
For example, if the `mail.catchall.domain` is set to `example.com` and
an email response comes from:
"John D" <john@doe.com>
it will rewrite it to:
"John D (john@doe.com)" <notifications@example.com>
This will make sure the system never sends outgoing email for an external
domain, as it has no authority for doing so, and that could
break mail filtering/authentication rules (SPF, DMARC, etc.)
During this "encapsulation rewrite step", both the original Sender name
and their email are preserved, and put into the quoted "name" field of
the rewritten address. It seems sensible to preserve as much information
as possible about the original sender.
Unfortunately, the inclusion of the Sender email in the final name makes
it appear to some inbox providers as if the message is trying to
deceptively impersonate another person (as many phishing schemes would).
As of November 2021 GMail at least does this, and will hide the name in
the UI when it happens. It will keep only the rewritten email, which is not
very useful in the case of a notification (even though it's more
technically correct, of course).
This patch removes the original email from the rewritten notification,
keeping only the name, considering that the email is not the most
important part, and it's better to have one of the two than none.
So after the patch, the rewritten address is now:
"John D" <notifications@example.com>
When there is no name in the original address, we keep only the local
part of the email, to avoid the same display issue. The recipient will
have to identify the sender based on the context / past messages.
closesodoo/odoo#81807
X-original-commit: 3c65ec5a8191a392980ceb0a8c584767eae405f1
Signed-off-by: Olivier Dony <odo@odoo.com>
Purpose
=======
When there are multiple databases on a single server, it is necessary
to be able to set the "from_filter" for the implicit SMTP server
on a per-database basis, to allow 'opt-in' to the automatic wrapping
of email notifications.
When set to e.g. `notifications@example.com`, and a default SMTP
server is defined in the server-wide config or CLI, outgoing emails
will be automatically rewritten to come from this email, unless the
From/Return-Path domains match the `mail.catchall.domain` parameter.
Task-2710632
closesodoo/odoo#81277
X-original-commit: 8b468e1f7a3dfeb2bcfa83bff55c4b5adf6b2212
Signed-off-by: Olivier Dony <odo@odoo.com>
PURPOSE
=======
We want to increase the score of the emails sent by Odoo and we want to
avoid them to be marked as spam by the mail clients (gmail, outook...).
SPECIFICATIONS
===============
From filter
-----------
Add a new field on the "ir.mail_server" which is "from_filter". This
field defines the email address for which the outgoing email server can
be used.
The "from_filter" can either define an email address or a domain name.
Use the system parameter "mail.default.from" which allow us to define
a default email address which is used to encapsulate the emails
(default: notifications@<catch.all.domain>).
Mail server priorities
----------------------
When sending an email, we read the FROM header and,
- We first look for a mail server which match the entire mail FROM
in that case, we do not change the email header (not needed)
- If not found, we search a mail server which matches the domain name of
the mail from (do not need to change the headers in that case)
- If not found, find the mail server linked to the "notifications"
email (defined in the system parameter). Then change the FROM header
to the notification email, and put the old one in the name part of
this header.
E.g.
Initial mail from: "Admin" < admin@example.com >
Final mail from: "Admin (admin@odoo.com)" < notifications@odoo.com >
- If no notification email is configured or if no mail server are
found for the notification email, fallback to the old system and
spoof the FROM header. In that case we do not have the choice if we
want to send the email, he will probably be marked as spam.
Sending method priority
-----------------------
In the mail server models, we defined some priorities,
1. Forced SMTP session
2. Forced mail server
3. Try to find the best mail server (see "Mail server priorities")
4. If not found, read the odoo-bin arguments
Bounce
------
As there's no standard for bounce address, we put it in the envelope
(smtp_from). But in some case, it might be considered as spoofing. So,
we use the bounce address ONLY if the mail server is configured for the
entire domain name.
One behavior which might be broken is the following; we send an email as
"std@gmail.com" and the bounce address is on the domain "odoo.com".
Before we received the bounce notifications but we were spoofing the
local part and the domain.
Now
- if a mail server is configured for GMAIL, we do not use the bounce
address (and we might not receive the bounce notification)
- if no mail server is configured for GMAIL, but one is configured for
"odoo.com"
- the FROM header will be "notifications@odoo.com"
- the FROM envelope will be the bounce address
=> In this situation we are spoofing only the local part of the email
but it's allowed as the mail server is configured for the entire
domain name
LINKS
=====
Task-2367946
odoo/odoo#61853odoo/upgrade#1903
Thibault Delavallée