Steps to reproduce:
- Configure incoming mail server and set it to create X record
on incoming mails (X can be any model with a chatter)
- Create a CSV file and set the encoding to UTF-16
- Send the CSV file through Gmail to the Odoo instance
- Go to model X and open the created record
- In the chatter, click/download the CSV file
- Open the downloaded file with Geany (or any file editor that can
show the file encoding)
Issue:
The file encoding is not the same as the original file (utf-8 instead
of utf-16).
Working with Outlook.
Cause:
The difference between Outlook and Gmail is that Gmail provides the
charset of the file.
The content of the mail is retrieved using `email` python lib.
The lib will try to retrieve the charset of the file and fallback
on `ASCII` if not available, then return the decode content.
```python
def get_text_content(msg, errors='replace'):
content = msg.get_payload(decode=True)
charset = msg.get_param('charset', 'ASCII')
return content.decode(charset, errors=errors)
```
Example:
content = b'd\x00a\x00,\x00,\x00,\......'
Outlook:
charset = 'ASCII'
return => 'd\x00a\x00,\x00,\x00...'
Gmail:
charset = 'UTF-16LE'
return => 'da,,,,,\n,,,,,\....'
In the post process of the attachment, the content is encoded in
'utf-8' (to then encoded in base64) before creating the attachment
record.
Content encoded to 'utf-8':
Outlook: b'd\x00a\x00,\x00,\x00...'
Gmail: b'da,,,,,\n,,,,,\n....'
Therefore, when writing the file on the disk, the encoding is based
on the binary content.
Solution:
When parsing the mail, add the encoding charset to the `info` variable.
Then, when creating the attachment, use the charset in `info` (or
fallback on 'utf'8' if no charset set) to encode the content.
opw-3089009
closesodoo/odoo#118694
X-original-commit: 2d1b13e68d8bb204ecfe12a9d3db519c4264592c
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Signed-off-by: Nasreddin Boulif (bon) <bon@odoo.com>
Based on RFC3462, a Content-Type text/rfc822-headers
exists and provide a mechanism to label and return
only the RFC 822 headers of a failed message (bounce)
These are only the headers and not the full message.
The Content-Type-Encoding should be either 7-bit(US-ASCII)
or Quoted-Printable (QP) as in the section 2 of the RFC.
Spawn the error:
After getting reported by a customer, I had to reproduce
by spamming wrong outlook addresses and add logging
in a sh database on the message_process of mail_thread.py
and logged the `message` variable.
After few retry, I got one of the part that was defined
as followed:
Content-Description: Undelivered Message Headers
Content-Type: text/rfc822-headers
Content-Transfer-Encoding: quoted-printable
The `get_payload()`function used was only assuming
that there is a full email on that part and that
it could only be encoded as an email, which was not
the case in this situation (quoted-printable:
76 characters per line, character `=` used
as the end of line character).
opw-3064589
task-3131561
closesodoo/odoo#110901
X-original-commit: f86f4b671696178f8fa42b81d8753d2591578431
Signed-off-by: Julien Castiaux <juc@odoo.com>
Co-authored-by: Baptiste Vergote <bve@odoo.com>
Some mail clients set a wrong content-type for PDF attachments, they
set `binary/octet-stream` which doesn't exist[^1]. Odoo would crash
with a traceback in case it tried to extract the content of such
malformed attachment because Python doesn't know how to decode such
content-types.
Looking at the actual content of the emails we received, we would had
expected the content-type to be `application/octet-stream` instead. It
is fair to assume that the mail client software is buggy or poorly
configured and that it assumes that `application/octet-stream` and
`binary/octet-stream` represent the same thing.
Following Postel's Law[^2], it is better to still handle those malformed
emails, assuming the content-type is `application/octet-stream`.
opw-3030113
opw-2716507
[^1]: https://www.rfc-editor.org/rfc/rfc2046#section-3
[^2]: https://en.wikipedia.org/wiki/Robustness_principleclosesodoo/odoo#107317
X-original-commit: e6169325b968711a0a28444818b34aa1932a6bae
Signed-off-by: Julien Castiaux <juc@odoo.com>
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Co-authored-by: Julien Castiaux <juc@odoo.com>
Steps to reproduce:
1.) Create a custom email domain and incoming email server on a database, set the
Actions to Perform on Incoming Mails to Create a new record: Helpdesk Ticket.
2.) Set an email alias for a Helpdesk team, set the assignment method to
balanced/random, assign some users to the team.
3.) Set another email address to forward emails to the alias for the Helpdesk team.
4.) Emails received directly by the email alias will create tickets and assign
properly, emails that are forwarded to the email alias will fall back on assignment
defaults.
Explanation:
When we get the "Delivered-To" field for the message dictionnary we use
decode_message_header and the message.get_all() function, this function
returns a list with two addresses but it is transformed back into a string
in decode_message_header with a space as separator. This create an issue
when we use email_split_and_format on this string as it uses
email.utils.getaddresses that expects a list of headers field or a text
where addresses are separated with a comma instead of a string with the
header fields separated by " ". Because of that getaddresses fails to get
the right addresses and the recipients field of the message dictionnary is
missing the right address. Hence when we check if the alias is in this
values it does not find it and use the default fall back.
Solution:
To solve the issue we set the separator as a comma in decode_message_header.
opw-2917543
closesodoo/odoo#98761
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose
=======
In this commit, we will fix the loops that occur when you send an email
to an alias (to create a ticket e.g.). In that case Odoo can reply
"Your ticket has been created" and then the auto-replier of the user
can reply to this email and the loop occurs.
Specifications
==============
To solve this issues, we add 2 system parameters
- <mail.gateway.loop.minutes>, 120 minutes by default
- <mail.gateway.loop.threshold>, 20 by default
When an email is sent to an alias, we look on the last records created
<mail.gateway.loop.minutes> minutes ago. If we overcome the limit
<mail.gateway.loop.threshold> the email is ignored.
Alias creation detection
========================
To detect the number of records created by a specific email address we
use the `_primary_email` attribute set on the model.
Task-2294034
closesodoo/odoo#78597
Related: odoo/enterprise#21772
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose of this commit is to highlight an issue that may happens easily with
`crm` that is made generic here within `test_mail`.
`crm` alters the context when creating a new record adding in this case
`default_type` to it][1]. The returned record contains that altered context.
his results in other records created from it trying to assign that same default
value for `type`. This is a very common name for fields, and happens to exist
in `ir.attachment` too.
If you create an alias for incoming leads in your DB with default values
`{"type": "lead"}` (something very common) and then an email comes to that
alias that contains an inlined base64 image, the attachment creation process
would simply fail.
Obtained error is ``ValueError: Wrong value for ir.attachment.type: 'lead'`` .
[1]: https://github.com/odoo/odoo/blob/272602193f5647f7f2270ed6ec68777625a139dd/addons/crm/models/crm_lead.py#L310-L311
X-original-commit: 99434b2e8528c10fcc9cb6860765e0ddcaa364c8
Part-of: odoo/odoo#77005
Co-authored-by: Thibault Delavallee <tde@odoo.com>
When body does not contain any tag or content parsing currently fails
with an ``lxml.etree.ParserError``. To avoid that we can improve condition
about void body: stripping void characters allows to avoid that traceback.
Task-2641572
PR odoo#76159
Closes odoo#75625
X-original-commit: 708fe3e74991c144a05c6bdcafa1931672e36ce9
Part-of: odoo/odoo#77005
Co-authored-by: Thibault Delavallee <tde@odoo.com>
Some emails are wrongly formatted mainly due to old servers. If Final-Recipient
header is void or wrongly encoded it currently crashes. This fix ensure there
is no crash, even if bounce detection could be incomplete.
Task-2641572
PR odoo/odoo#76159Closesodoo/odoo#75618
X-original-commit: f437967a1fa4fca56c88e1cf79586805101ab712
Part-of: odoo/odoo#77005
Co-authored-by: Thibault Delavallee <tde@odoo.com>
WHY:
Some mail servers may provide `Message-Id` header in a section typed
`text/rfc822-headers` instead of a `message/rfc822`
According to https://tools.ietf.org/html/rfc6522#section-4 that section's body
actually contains the headers from the bounced email.
STEPS: a way to reproduce it should be using postfix v2.5+ with setting
bounce_size_limit=1. See
https://github.com/odoo/odoo/pull/42340#issuecomment-617605521
BEFORE: `bounced_message_id` is empty and thus the little email envelope icon
doesn't turn red, nor do the email resend features trigger in the UI. You could
be sending an invoice and never knowing it was bounced.
AFTER: bounces from mail servers that return the `text/rfc822-headers` part will
be handled properly.
---
@Tecnativa TT21170
opw-2162067
opw-2344252
closes#42340closes#62551closesodoo/odoo#62732
X-original-commit: 375ba5b37053a599922dd61c9e787553d6171291
Signed-off-by: Ivan Yelizariev // IEL <yelizariev@users.noreply.github.com>
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules: `email.(charset|header|mime|utils)`.
The new `email.message.EmailMessage` API exposes a much easier interface
to create multiparted emails [1], is capable of doing all the necessary
headers value conversion (RFCs [2045], [2047], [2049]) [2] and get/set
different flavor of payload (text/bytes) in a straightforward way [3].
All headers are structured in a way to support python native types, i.e.
the `Date` header supports `datetime.datetime` objects and automatically
performs the required formatting. The same goes for multi-valued headers
like the `To` header, one can directly set a python list of values, it
will be automatically be formatted according to the RFCs.
The dedicated encoding and decoding functions are no more needed thus
has been removed has part of the refactor. FTR, [RFC2231] is an update
of [RFC2047] and based on tests we've just conducted, both GMail and
Thunderbird now support RFC2231 encoding just fine in all headers:
attachments names, From, etc.
[1]: http://docs.python.org/3/library/email.message.html
[2]: http://docs.python.org/3/library/email.headerregistry.html
[3]: http://docs.python.org/3/library/email.contentmanager.html
[2045]: https://tools.ietf.org/html/rfc2045
[2047]: https://tools.ietf.org/html/rfc2047
[2049]: https://tools.ietf.org/html/rfc2049
[2231]: https://tools.ietf.org/html/rfc2231closesodoo/odoo#35929
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Purpose
* move bounce information detection in message parsing. It allows to have
this information available in various steps of routing instead of having
to manually re-compute them;
* handle bounce in specific methods allowing easy override;
* improve bounce management, notably when detecting a bounce not linked
to the bounce alias configuration;
* better integration with blacklist mechanism;
Specifications
* compute bounce information in ``_message_parse_extract_bounce``.It parses
bounce information and returns a dictionary allowing to update parsed email
values;
* remove override in mass_mailign that basically does what mail already
does;
* manage bounce in ``_routing_handle_bounce``;
* when detecting a bounce, correctly call the bounce management method on
all models inheriting from blacklist;
* correctly update bounce counter;
* bounced mailing traces and automatic blacklist in mass mailing should
be done in ``_routing_handle_bounce``;
* add some tests;
LINKS
Related to task 1893155
Linked to PR #33340
This commit add some tests related to the mail gateway: more bounce management
tests and some additional thread formation tests. Some test asserts about
bounce / blacklist management are commented as they are not completely working
currently. This will be improved in master soon.
Some cleaning in also done in all mail gateway tests. Notably some call to
tool methods are cleaned / simplified, duplicate tests are removed. Some
low-level checks are removed.
A new test model is added for mail gateway: mail.test.gateway. It is a
chatter model with blacklist enabled on it. It allows to tests the
various blacklist-related overrides and features as well as all basic
mail gateway features.
Sub-part of task 1853147 (pre-cleaning before implementing mail gateway
improvements)
Linked to PR #32974
Nasreddin Boulif (bon)