Some external tools send email as pure html (no multipart) and when
parsing such email we ends up having the raw HTML as body (text)
This commit ensure we correctly parse and sanitize the body as HTML
for such emails.
closesodoo/odoo#132127
Task-id: 3451889
X-original-commit: a4763d214bfc7b09510226e552f749f9cd95870e
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Signed-off-by: Xavier Alt (xal) <xal@odoo.com>
Rename alias domain and aliases used a test data. This allows to make
them easier to read, follow, grep and understand.
Activate multi-company on alias and gateway tests, ensuring it currently
has few impact on tests.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130768
When parsing an email containing an xml attachment, the `email` python
module will decode the base64 attachment using the charset or ascii if
the charset is missing.
In some cases, the payload is in UTF-8 but the charset is omitted. This
results in replacement characters for the non ASCII characters.
The solution is to force the charset to UTF-8, since it is a superset of
ASCII that should not be a problem.
NB1: Omitting the charset for text/xml is not recommended. See the RFC
(section 6.4): https://www.ietf.org/rfc/rfc2376.txt
opw-3144519
closesodoo/odoo#126474
X-original-commit: e3a5b46c7b4f5030e165a6d943ec2275aea5d583
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
Signed-off-by: Julien Van Roy (juvr) <juvr@odoo.com>
When receiving an email on a mailbox with an alias that triggers the
creation of invoices, 4 bugs could occur.
1. If the xml received contains replacement characters (U+FFFD �), and
the charset of the part of the email is "US-ASCII" the encoding of the
string will fail, preventing the rest of the flow to be completed. Be
more resilient, encode the string and ignores these characters if this
case occurs.
NB: sometimes, the charset is omitted for a Content-type: text/xml. This
is valid but not recommended (see:
https://www.ietf.org/rfc/rfc2376.txt). In this case, the default used is
"US-ASCII". This means that any non-ascii char will be lost (they are
replaced by the replacement character: �, see:
https://github.com/python/cpython/blob/3.10/Lib/email/contentmanager.py#L67)
when decoding the attachment.
2. When the xml attachment is created in Odoo, the mimetype is
'text/plain' (rather than 'application/xml'). Thus, the
`_decode_attachment` needs to be more flexible when guessing the type of
the attachment (to know which function to use to read the content of the
attachment and create the invoice).
3. When creating an invoice from an email with an xml attachment, the
xml is attached as the `message_main_attachment_id`. It's only later on
that the content of the xml is read and we possibly find the PDF in
base64 inside. When creating the PDF attachment, it was not set as the
`message_main_attachment_id`, so the PDF was not rendered on the right
part of the invoice form view. Add a clause to replace the
`message_main_attachment_id` in such a case.
4. When the xml attachment represents a credit note, the move_type of
the invoice created by the email alias needs to be changed. Indeed, the
invoice is created before decoding the attachment, so we can only change
the `move_type` later.
opw-3144519
opw-3149649
closesodoo/odoo#121076
X-original-commit: 1e193a92b9c84e75b958985f8067873b90f686e0
Signed-off-by: Laurent Smet <las@odoo.com>
Signed-off-by: Julien Van Roy <juvr@odoo.com>
Steps to reproduce:
- Configure incoming mail server and set it to create X record
on incoming mails (X can be any model with a chatter)
- Create a CSV file and set the encoding to UTF-16
- Send the CSV file through Gmail to the Odoo instance
- Go to model X and open the created record
- In the chatter, click/download the CSV file
- Open the downloaded file with Geany (or any file editor that can
show the file encoding)
Issue:
The file encoding is not the same as the original file (utf-8 instead
of utf-16).
Working with Outlook.
Cause:
The difference between Outlook and Gmail is that Gmail provides the
charset of the file.
The content of the mail is retrieved using `email` python lib.
The lib will try to retrieve the charset of the file and fallback
on `ASCII` if not available, then return the decode content.
```python
def get_text_content(msg, errors='replace'):
content = msg.get_payload(decode=True)
charset = msg.get_param('charset', 'ASCII')
return content.decode(charset, errors=errors)
```
Example:
content = b'd\x00a\x00,\x00,\x00,\......'
Outlook:
charset = 'ASCII'
return => 'd\x00a\x00,\x00,\x00...'
Gmail:
charset = 'UTF-16LE'
return => 'da,,,,,\n,,,,,\....'
In the post process of the attachment, the content is encoded in
'utf-8' (to then encoded in base64) before creating the attachment
record.
Content encoded to 'utf-8':
Outlook: b'd\x00a\x00,\x00,\x00...'
Gmail: b'da,,,,,\n,,,,,\n....'
Therefore, when writing the file on the disk, the encoding is based
on the binary content.
Solution:
When parsing the mail, add the encoding charset to the `info` variable.
Then, when creating the attachment, use the charset in `info` (or
fallback on 'utf'8' if no charset set) to encode the content.
opw-3089009
closesodoo/odoo#118694
X-original-commit: 2d1b13e68d8bb204ecfe12a9d3db519c4264592c
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Signed-off-by: Nasreddin Boulif (bon) <bon@odoo.com>
Based on RFC3462, a Content-Type text/rfc822-headers
exists and provide a mechanism to label and return
only the RFC 822 headers of a failed message (bounce)
These are only the headers and not the full message.
The Content-Type-Encoding should be either 7-bit(US-ASCII)
or Quoted-Printable (QP) as in the section 2 of the RFC.
Spawn the error:
After getting reported by a customer, I had to reproduce
by spamming wrong outlook addresses and add logging
in a sh database on the message_process of mail_thread.py
and logged the `message` variable.
After few retry, I got one of the part that was defined
as followed:
Content-Description: Undelivered Message Headers
Content-Type: text/rfc822-headers
Content-Transfer-Encoding: quoted-printable
The `get_payload()`function used was only assuming
that there is a full email on that part and that
it could only be encoded as an email, which was not
the case in this situation (quoted-printable:
76 characters per line, character `=` used
as the end of line character).
opw-3064589
task-3131561
closesodoo/odoo#110901
X-original-commit: f86f4b671696178f8fa42b81d8753d2591578431
Signed-off-by: Julien Castiaux <juc@odoo.com>
Co-authored-by: Baptiste Vergote <bve@odoo.com>
Some mail clients set a wrong content-type for PDF attachments, they
set `binary/octet-stream` which doesn't exist[^1]. Odoo would crash
with a traceback in case it tried to extract the content of such
malformed attachment because Python doesn't know how to decode such
content-types.
Looking at the actual content of the emails we received, we would had
expected the content-type to be `application/octet-stream` instead. It
is fair to assume that the mail client software is buggy or poorly
configured and that it assumes that `application/octet-stream` and
`binary/octet-stream` represent the same thing.
Following Postel's Law[^2], it is better to still handle those malformed
emails, assuming the content-type is `application/octet-stream`.
opw-3030113
opw-2716507
[^1]: https://www.rfc-editor.org/rfc/rfc2046#section-3
[^2]: https://en.wikipedia.org/wiki/Robustness_principleclosesodoo/odoo#107317
X-original-commit: e6169325b968711a0a28444818b34aa1932a6bae
Signed-off-by: Julien Castiaux <juc@odoo.com>
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Co-authored-by: Julien Castiaux <juc@odoo.com>
Steps to reproduce:
1.) Create a custom email domain and incoming email server on a database, set the
Actions to Perform on Incoming Mails to Create a new record: Helpdesk Ticket.
2.) Set an email alias for a Helpdesk team, set the assignment method to
balanced/random, assign some users to the team.
3.) Set another email address to forward emails to the alias for the Helpdesk team.
4.) Emails received directly by the email alias will create tickets and assign
properly, emails that are forwarded to the email alias will fall back on assignment
defaults.
Explanation:
When we get the "Delivered-To" field for the message dictionnary we use
decode_message_header and the message.get_all() function, this function
returns a list with two addresses but it is transformed back into a string
in decode_message_header with a space as separator. This create an issue
when we use email_split_and_format on this string as it uses
email.utils.getaddresses that expects a list of headers field or a text
where addresses are separated with a comma instead of a string with the
header fields separated by " ". Because of that getaddresses fails to get
the right addresses and the recipients field of the message dictionnary is
missing the right address. Hence when we check if the alias is in this
values it does not find it and use the default fall back.
Solution:
To solve the issue we set the separator as a comma in decode_message_header.
opw-2917543
closesodoo/odoo#98761
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose
=======
In this commit, we will fix the loops that occur when you send an email
to an alias (to create a ticket e.g.). In that case Odoo can reply
"Your ticket has been created" and then the auto-replier of the user
can reply to this email and the loop occurs.
Specifications
==============
To solve this issues, we add 2 system parameters
- <mail.gateway.loop.minutes>, 120 minutes by default
- <mail.gateway.loop.threshold>, 20 by default
When an email is sent to an alias, we look on the last records created
<mail.gateway.loop.minutes> minutes ago. If we overcome the limit
<mail.gateway.loop.threshold> the email is ignored.
Alias creation detection
========================
To detect the number of records created by a specific email address we
use the `_primary_email` attribute set on the model.
Task-2294034
closesodoo/odoo#78597
Related: odoo/enterprise#21772
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose of this commit is to highlight an issue that may happens easily with
`crm` that is made generic here within `test_mail`.
`crm` alters the context when creating a new record adding in this case
`default_type` to it][1]. The returned record contains that altered context.
his results in other records created from it trying to assign that same default
value for `type`. This is a very common name for fields, and happens to exist
in `ir.attachment` too.
If you create an alias for incoming leads in your DB with default values
`{"type": "lead"}` (something very common) and then an email comes to that
alias that contains an inlined base64 image, the attachment creation process
would simply fail.
Obtained error is ``ValueError: Wrong value for ir.attachment.type: 'lead'`` .
[1]: https://github.com/odoo/odoo/blob/272602193f5647f7f2270ed6ec68777625a139dd/addons/crm/models/crm_lead.py#L310-L311
X-original-commit: 99434b2e8528c10fcc9cb6860765e0ddcaa364c8
Part-of: odoo/odoo#77005
Co-authored-by: Thibault Delavallee <tde@odoo.com>
When body does not contain any tag or content parsing currently fails
with an ``lxml.etree.ParserError``. To avoid that we can improve condition
about void body: stripping void characters allows to avoid that traceback.
Task-2641572
PR odoo#76159
Closes odoo#75625
X-original-commit: 708fe3e74991c144a05c6bdcafa1931672e36ce9
Part-of: odoo/odoo#77005
Co-authored-by: Thibault Delavallee <tde@odoo.com>
Some emails are wrongly formatted mainly due to old servers. If Final-Recipient
header is void or wrongly encoded it currently crashes. This fix ensure there
is no crash, even if bounce detection could be incomplete.
Task-2641572
PR odoo/odoo#76159Closesodoo/odoo#75618
X-original-commit: f437967a1fa4fca56c88e1cf79586805101ab712
Part-of: odoo/odoo#77005
Co-authored-by: Thibault Delavallee <tde@odoo.com>
WHY:
Some mail servers may provide `Message-Id` header in a section typed
`text/rfc822-headers` instead of a `message/rfc822`
According to https://tools.ietf.org/html/rfc6522#section-4 that section's body
actually contains the headers from the bounced email.
STEPS: a way to reproduce it should be using postfix v2.5+ with setting
bounce_size_limit=1. See
https://github.com/odoo/odoo/pull/42340#issuecomment-617605521
BEFORE: `bounced_message_id` is empty and thus the little email envelope icon
doesn't turn red, nor do the email resend features trigger in the UI. You could
be sending an invoice and never knowing it was bounced.
AFTER: bounces from mail servers that return the `text/rfc822-headers` part will
be handled properly.
---
@Tecnativa TT21170
opw-2162067
opw-2344252
closes#42340closes#62551closesodoo/odoo#62732
X-original-commit: 375ba5b37053a599922dd61c9e787553d6171291
Signed-off-by: Ivan Yelizariev // IEL <yelizariev@users.noreply.github.com>
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules: `email.(charset|header|mime|utils)`.
The new `email.message.EmailMessage` API exposes a much easier interface
to create multiparted emails [1], is capable of doing all the necessary
headers value conversion (RFCs [2045], [2047], [2049]) [2] and get/set
different flavor of payload (text/bytes) in a straightforward way [3].
All headers are structured in a way to support python native types, i.e.
the `Date` header supports `datetime.datetime` objects and automatically
performs the required formatting. The same goes for multi-valued headers
like the `To` header, one can directly set a python list of values, it
will be automatically be formatted according to the RFCs.
The dedicated encoding and decoding functions are no more needed thus
has been removed has part of the refactor. FTR, [RFC2231] is an update
of [RFC2047] and based on tests we've just conducted, both GMail and
Thunderbird now support RFC2231 encoding just fine in all headers:
attachments names, From, etc.
[1]: http://docs.python.org/3/library/email.message.html
[2]: http://docs.python.org/3/library/email.headerregistry.html
[3]: http://docs.python.org/3/library/email.contentmanager.html
[2045]: https://tools.ietf.org/html/rfc2045
[2047]: https://tools.ietf.org/html/rfc2047
[2049]: https://tools.ietf.org/html/rfc2049
[2231]: https://tools.ietf.org/html/rfc2231closesodoo/odoo#35929
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Purpose
* move bounce information detection in message parsing. It allows to have
this information available in various steps of routing instead of having
to manually re-compute them;
* handle bounce in specific methods allowing easy override;
* improve bounce management, notably when detecting a bounce not linked
to the bounce alias configuration;
* better integration with blacklist mechanism;
Specifications
* compute bounce information in ``_message_parse_extract_bounce``.It parses
bounce information and returns a dictionary allowing to update parsed email
values;
* remove override in mass_mailign that basically does what mail already
does;
* manage bounce in ``_routing_handle_bounce``;
* when detecting a bounce, correctly call the bounce management method on
all models inheriting from blacklist;
* correctly update bounce counter;
* bounced mailing traces and automatic blacklist in mass mailing should
be done in ``_routing_handle_bounce``;
* add some tests;
LINKS
Related to task 1893155
Linked to PR #33340
This commit add some tests related to the mail gateway: more bounce management
tests and some additional thread formation tests. Some test asserts about
bounce / blacklist management are commented as they are not completely working
currently. This will be improved in master soon.
Some cleaning in also done in all mail gateway tests. Notably some call to
tool methods are cleaned / simplified, duplicate tests are removed. Some
low-level checks are removed.
A new test model is added for mail gateway: mail.test.gateway. It is a
chatter model with blacklist enabled on it. It allows to tests the
various blacklist-related overrides and features as well as all basic
mail gateway features.
Sub-part of task 1853147 (pre-cleaning before implementing mail gateway
improvements)
Linked to PR #32974
Xavier ALT