WHY:
Some mail servers may provide `Message-Id` header in a section typed
`text/rfc822-headers` instead of a `message/rfc822`
According to https://tools.ietf.org/html/rfc6522#section-4 that section's body
actually contains the headers from the bounced email.
STEPS: a way to reproduce it should be using postfix v2.5+ with setting
bounce_size_limit=1. See
https://github.com/odoo/odoo/pull/42340#issuecomment-617605521
BEFORE: `bounced_message_id` is empty and thus the little email envelope icon
doesn't turn red, nor do the email resend features trigger in the UI. You could
be sending an invoice and never knowing it was bounced.
AFTER: bounces from mail servers that return the `text/rfc822-headers` part will
be handled properly.
---
@Tecnativa TT21170
opw-2162067
opw-2344252
closes#42340closes#62551closesodoo/odoo#62732
X-original-commit: 375ba5b37053a599922dd61c9e787553d6171291
Signed-off-by: Ivan Yelizariev // IEL <yelizariev@users.noreply.github.com>
Purpose of this commit is simply to add tests about alias defaults.
Code protects it against being badly formatted. However having
tests ensuring it is always a good idea to avoid regression.
Task ID-2361179
closesodoo/odoo#60026
X-original-commit: 0075cc02f9a11dcddaf9ca7d70cc1adf7596f6d3
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Document creation or update is still done without auto subscribe. Indeed user
running mailgateway or owning alias is not necessarily linked to the email
author. That way we avoid auto subscription of irrelevant people.
Posting message based on incoming email is now allowing auto subscription if
there is an author found during email parsing. We also ensure this author is
not root, to be sure he is not added in followers of documents.
LINKS
Task ID-2326281
PR odoo/odoo#56560Closesodoo/odoo#38383
X-original-commit: fe27f9fe0cf9581c15a2a9e45125abe82dd303ad
PURPOSE
Have a cleaner test_mail addons
SPECIFICATIONS
Keep only mail-related tests, move odoobot in test mail full, send "update
notification" tests in mail (specific to mail). Merge some test files to
lessen number of files, perform light file renaming.
Split test mail models file to prepare some cleaning in those models and tests.
LINKS
Prepares Task ID 2238597 (clean notification models)
Prepares Task ID 2083854 (improve mass mailing technical flows)
PR #49891
Purpose of this commit is to allow multi-create in alias mixin by correctly
creating / updating aliases in batch.
Task ID 1919277
Community PR odoo/odoo#41160
Enterprise PR odoo/enterprise#6983
Upgrade PR odoo/upgrade#872
Co-Authored-By: Rémy Voet <ryv@odoo.com>
Co-Authored-By: Thibault Delavallée <tde@odoo.com>
Clean the usage of mail aliases and more specifically its associated mixin
`mail.alias.mixin` .
Stop using context for model of aliases, and correctly give model_id and
parent_model_id to the call chain through a cleaned code easier to override.
We also merge methods get_alias_model_name and get_alias_values in single one
called before record creation (alias first values) and right after (to have
values depending on actual record).
LINKS
Task ID 1919277
Community PR odoo/odoo#41160
Enterprise PR odoo/enterprise#6983
Upgrade PR odoo/upgrade#872
Co-Authored-By: Rémy Voet <ryv@odoo.com>
Co-Authored-By: Thibault Delavallée <tde@odoo.com>
PURPOSE
Make aliases usage more unique, prevent re-using bounce or catchall aliases
and remove auto-uniqueness of aliases.
SPECIFICATIONS
Before this commit, user could create an email alias having same name as
catchall/bounce email alias, which should not happen. Also, while creating
or duplicating alias, if the same name was already available, a new unique
name was generated by adding a sequence number to existing name.
This behavior is not considered as a good one as it magically creates aliases
different from what user expects. User could even not own the newly-created
alias, leading to a broken mail gateway.
In this commit we improve that behavior by ensuring that no duplicate alias
name should be entered while creation / updation for both catcall/bounce and
mail alias. Also, while duplicating an alias, name will now be blank by default
to force user to enter the name. Finally when creating an alias an error is
raised if the name is already taken.
Task ID 2160070
PURPOSE
When incoming emails bounce due to alias security bounce email is quite
generic. Purpose of this task is to ease its customization and update.
SPECIFICATIONS
In order to improve the flexibility of alias, add a customizable html field
on the alias model. This html content will be send as bounce email core content
in case of bounced/unauthorized mail received for this alias,
Obviously it has no effect on 'everyone' security setting as no email will
bounce due to that issue.
If it is not set a default generic mail will be send depending on security
setting. It allows to keep void html fields when no specific bounce content
is required
In HR, an old template allowing some light customization for employee based
security option is removed as it is completely replaced by the new feature.
Also add references message-id of the mail received to the answer so that
threads are correctly set.
LINKS
Task ID 2126509
When mailgateway receives an incoming email being a reply to a thread but
containing a recipient being an alias linked to another model, it is
considered as a forward to that new alias. It therefore skips the reply
step in routing and applies rules related to new thread, aka checking
all recipients.
Consider this use case: receives a email on a task from "project@domain"
project being the alias of the project which creates new tasks when not
routing replies. Reply / forward it to "project@domain, sales@domain" in
order to transfer it to the sales team (sales alias creates new leads).
As this is a forward, both aliases are evaluated, leading to a new task and a
lead which is not what we expect.
We solve this issue by removing alias linked to the ignored reply model
when considering recipients.
PR #46764
Task ID 2121551
closesodoo/odoo#46784
X-original-commit: 7cff7787cb42cc9d3d0cb4c5af41a9f09d59d1ff
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
/!\ This commit is a manual forward-port of two PR targetting originally
11.0: #40442 and its fix / partial revert #41042. /!\
Purpose of this commit is to check all recipients of incoming emails when
trying to determine the route to apply. Notably
* an incoming email should be considered as a write to catchall only if all
recipients are catchall. Catchall + a valid alias should take the alias
into account;
* an incoming email sent to the bounce alias should be considered as a bounce
even if another valid alias is in recipients and whatever the order. Indeed
it indicates an issue and bounce is considered as more important;
* forward to an alias linked to another model should check all recipients
and not only the first one. Otherwise reply_alias, forward_alias is
considered as a reply whereas it should be considered as a forward when
considering all recipients;
Tests are added according to those specifications.
Please see original PRs for more details about the content, the comments
done on it and the various discussions.
PR #46764
Task ID 2121551
X-original-commit: df00bd2d3bddf9a2b56e8554380669025e146d21
This reverts commit 976e560a87.
Even if it didn't looked like a bad idea, this need some more thinking.
This new version of query count creates random failure of runbot builds.
closesodoo/odoo#43820
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
- take advantage of runbot multi-build capabilities
- get similar result when running them locally during dev
- better detect when other modules add extra queries
Query counts are split in the base value (testing with just test_mail installed)
+ the extra modules overhead.
Part of task-2178641
closes odoo/odoo#43666
Pr: #43666
Signed-off-by: Alexandre Kühn (aku) <aku@odoo.com>
PURPOSE
Clean posting process and improve mail.message definition and comprehension.
SPECIFICATIONS
In order to be more explicit subtype parameter is renamed to subtype_xmlid.
It therefore clearly indicates it should be a valid subtype Xml ID. Support
of ill formatted Xml IDs is removed because there is no reason to try to
add some random prefix. Give something that exists or go to hell, punk !
LINKS
Task ID 2071556
PR #38692
PURPOSE
Currently tools and asserts for mail tests are located inside test_mail
module. It makes difficult to re-use them in apps tests or force them to
write custom quick and dirty tools and asserts.
SPECIFICATIONS
Update tests to new tools / asserts / helpers / classes defined in mail
and test_mail. Notably
* ``BaseFunctionalTest`` class is replaced by the new ``TestMailCommon``
pimped one;
* ``assertNotifications`` is replaced by ``assertSinglePostNotifications``
(shortcut for a simple message_post) or ``assertPostNotifications``
(complete asserts involving several messages and notifications);
* correctly invoke ``mock_mail_gateway`` when mocking email sending;
* replace manual check of sent emails or ``assertEmails`` by
``assertSentEmails``;
* remove custom mocks / checks and replace them by now standard mocks
and assertions (if any);
In this commit we update: test mail gateway
LINKS
Task ID 2068986
PR #38070
Somehow followup of dbb6cfdbf8 . Purpose is
to use formataddr from tools instead of manually handcrafting the full
email address format. Indeed the tools one correctly encapsulates and
format name and email.
Task ID 2068986
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules: `email.(charset|header|mime|utils)`.
The new `email.message.EmailMessage` API exposes a much easier interface
to create multiparted emails [1], is capable of doing all the necessary
headers value conversion (RFCs [2045], [2047], [2049]) [2] and get/set
different flavor of payload (text/bytes) in a straightforward way [3].
All headers are structured in a way to support python native types, i.e.
the `Date` header supports `datetime.datetime` objects and automatically
performs the required formatting. The same goes for multi-valued headers
like the `To` header, one can directly set a python list of values, it
will be automatically be formatted according to the RFCs.
The dedicated encoding and decoding functions are no more needed thus
has been removed has part of the refactor. FTR, [RFC2231] is an update
of [RFC2047] and based on tests we've just conducted, both GMail and
Thunderbird now support RFC2231 encoding just fine in all headers:
attachments names, From, etc.
[1]: http://docs.python.org/3/library/email.message.html
[2]: http://docs.python.org/3/library/email.headerregistry.html
[3]: http://docs.python.org/3/library/email.contentmanager.html
[2045]: https://tools.ietf.org/html/rfc2045
[2047]: https://tools.ietf.org/html/rfc2047
[2049]: https://tools.ietf.org/html/rfc2049
[2231]: https://tools.ietf.org/html/rfc2231closesodoo/odoo#35929
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules. Among them the `email.utils.formataddr`
function, this function takes a `(name, email)` pair and returns a
string value suitable for From, To and Cc headers.
The stdlib function is capable of handling several character encoding
and two binary-to-ascii encoding. Odoo only uses `utf-8` which is
compatible with the base64 b2a encoding. The re-implementation of that
function has been simplified to only support base64-ed utf-8 and ascii.
[PEP 594]: https://www.python.org/dev/peps/pep-0594/\#email-lagacy-api
[PEP 594][1] is going to depreciate legacy `email.message.Message` API in
PY3.8 in favor `email.message.EmailMessage`.
All `email.message_from_x` constructor functions take an optional
`policy` argument. The policy can either be `policy.compat32` to
generate an `email.message.Message` instance, either be `policy.SMTP`
to generate an `email.message.EmailMessage` instance.
The [current][2] default policy is `policy.compat32` but will change in
a future version of python. It is recommended to always specify the
policy.
[1]: https://www.python.org/dev/peps/pep-0594/#email-legacy-api
[2] https://docs.python.org/3.7/library/email.parser.html#email.parser.BytesParser
PURPOSE
In order to rewrite some tests, remove low-level tests and add coverage
first step is to reorganize a bit test_mail content. Several tests cases
are spread among several files and finding back some feature coverage
tests is not easy.
SPECIFICATIONS
Split cc tests to either discuss or mail gateway tests. It allows to remove an
unnecessary file.
LINKS
Task 1958697
PURPOSE
In order to rewrite some tests, remove low-level tests and add coverage
first step is to reorganize a bit test_mail content. Several tests cases
are spread among several files and finding back some feature coverage
tests is not easy.
SPECIFICATIONS
Reorganize tests, notably
* put message_post related test in test_message_post;
* have only composer related test in test_message_compose(r);
* put alias tests in test_mail_gateway;
* move some discuss tests to post tests as they are linked to notification
details, not really Discuss features.
LINKS
Task 1958697
In resend wizard tests
In some cases tuple comparison of states seems to fail. Indeed order seems
to change in some cases depending on installation order. With this commit
we correctly check partner / notification status one by one, avoiding
to rely on order / set length that is not guaranteed.
In mail gateway tests
Add a specific order on name to be clearer about what we mean when fetching
users as a fallback, until better heuristic. Using name is sufficient as
we have no need to rely on complex partner display_name.
Task 2060220 (testing SMS integration)
PURPOSE
Currently mailgateway runner user is used to create most documents through
incoming emails. Purpose of this commit is to try to link document creation
and update to users, using the email to make the matching.
SPECIFICATIONS
New sudo implementation allow to bypass access rights while keeping effective
user as creator. Use this mechanism to create documents still using the
mailgateway runner user while linking creator to matching user.
Heuristic to use to match email_from and users
* if route is linked to an alias: check its owner record and check in its
followers;
* find users having this emails;
* if route is linked to an alias: use alias_user_id field;
* fallback on mailgateway user;
Other side effect is that creator and author of incoming email linked to the
new record will now better match the document creator when linked to an
existing user.
LINKS
Task 1919267
PR #30597
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Purpose
* move bounce information detection in message parsing. It allows to have
this information available in various steps of routing instead of having
to manually re-compute them;
* handle bounce in specific methods allowing easy override;
* improve bounce management, notably when detecting a bounce not linked
to the bounce alias configuration;
* better integration with blacklist mechanism;
Specifications
* compute bounce information in ``_message_parse_extract_bounce``.It parses
bounce information and returns a dictionary allowing to update parsed email
values;
* remove override in mass_mailign that basically does what mail already
does;
* manage bounce in ``_routing_handle_bounce``;
* when detecting a bounce, correctly call the bounce management method on
all models inheriting from blacklist;
* correctly update bounce counter;
* bounced mailing traces and automatic blacklist in mass mailing should
be done in ``_routing_handle_bounce``;
* add some tests;
LINKS
Related to task 1893155
Linked to PR #33340
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Parsing email values: make message_parse effectively prepare all required
values for later processing
* update message_parse to add some more data directly in parsed dictionary
and avoid having to parse value again in later processing and computation.
* notably recipients computation is now done directly in parsing. It
includes email check and reconstruction using tools methods. That way
routing just has to handle those values and not compute part of them;
* from and cc are correctly managed with parsed and cleaned (sanitized)
versions in dictionary;
* remove all code decoding message header and replace them by fetching
the parsed value instead;
* filter parsed values before calling message_post to ensure right values
are given to it instead of having to manage them in message_post. Indeed
caller has to give the right input;
* make parsing sub-methods (extract payload, post process payload) have
a dictionary as input and output to always manage a dictionary of values
through the parsing methods and update their namespace;
Routing: remove deprecated code parts
* remove use of email_references regex. Indeed emails are routed using their
messageId and not the regex anymore since several versions [1];
* clean some unnecessary variables;
* simplify variable type input, like message_parse accepts only valid
email.message instances and to avoid several decode / encode. Tests are
updated accordingly;
Update docstrings of those methods.
LINKS
Related to task 1893155
Linked to PR #33340
[1] See df037e2959 and 0028c42136
Message post should always be called on a record (ensure_one). That way we
ensure posting a message is always done in a record's context with right
values computed (reply_to, followers, ...)
Message_notify can be called on record or on mail_thread and must have
partner_ids. It is based on the recently modified user_notification mechanism
and allow to notify a partner on a record or just to push him a message
(aka, not linked to a record).
Small performance improvement
* browse recipients instead of search in _notify_email_recipients;
* todo in future optimizations: mayybe be improve by searching on ids
and is_blacklist immediately;
Related to task 1943901
Linked to PR #32404
Purpose: make some methods from mail.thread mixin private and update their
naming if necessary. Some methods prepare data for business flows and deal
with internal information. Public methods should either use of give some
access to those methods. Computation itself should be keps internal.
In this commit we put message_partner_info_from_emails as private. As it is
used in discuss JS we define a controller that controls access to this method.
It explicitly checks for related document access rule and rights before
calling the private method.
Some addons are updated accordingly to the method change.
Related to task ID 1911679
Linked to PR #29483
This commit add some tests related to the mail gateway: more bounce management
tests and some additional thread formation tests. Some test asserts about
bounce / blacklist management are commented as they are not completely working
currently. This will be improved in master soon.
Some cleaning in also done in all mail gateway tests. Notably some call to
tool methods are cleaned / simplified, duplicate tests are removed. Some
low-level checks are removed.
A new test model is added for mail gateway: mail.test.gateway. It is a
chatter model with blacklist enabled on it. It allows to tests the
various blacklist-related overrides and features as well as all basic
mail gateway features.
Sub-part of task 1853147 (pre-cleaning before implementing mail gateway
improvements)
Linked to PR #32974
We don't want to notify inactive followers,
especially in the case of oddobot that was considered
as a inactive partner in _notify_compute_recipients.
The computed recipient data for odoobot are
(pid:2 active:False pshare:True notif:None)
It was considered as a partner and notified by email.
This fix simply removes inactive partner from partner to notify.
closesodoo/odoo#31734
Signed-off-by: Toufik Benjaa (tbe) <tbe@odoo.com>
Purpose of this commit is to use the newly introduced helper to create
test users in a quick way and reduce code duplication. A mail specific
helper tool function is defined based on the standard one defined in test
allowing to pass a custom context. It bypasses mailing features in order
to speedup the creation.
This commit is linked to task ID 1889703 and PR #27526.
message_post and various mail.thread related code still support a thread_model
context key allowing to redirect message post on another model. Purpose of
this commit is to get rid of that context key and support model / res_id
fields coming from either the record on which we run, either parameters of
message_post. Indeed those parameter match the one used as fields on mail
message. It makes sense to support them for customization if necessary
instead of using a context key.
This context key exists since a lot of time. It is difficult to find the
source as history is quite complex when the mail was refactored in 2012.
See notably 6aa4f80b2d and 2c4ed841b9 for example.
Jairo Llopis