With input 'name email@domain.com' (missing chevrons allowing to clearly spot
the email part) 'getaddresses' returns ('', 'name email@domain.com) i.e. the
whole input is considered as being the email.
To improve the heuristic we can add a fallback by recalling 'getadresses'
on the input with spaces replaced by commas when it found only an email and
no name. The new email will be split into sub pairs allowing to find the real
email and various name parts, allowing to make a new name / email pair.
Emails should not contain spaces thus this is coherent with email formation.
This fallback actually comes from a specific code done in '_parse_partner_name'
of Partner model. Supporting it directly at tools level make the behavior
coherent for all models.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@18c71edf59
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
SPECIFICATIONS
As of rfc5322 section 3.4.1 local-part is case-sensitive. However most main
providers do consider the local-part as case insensitive. With the introduction
of smtp-utf8 within odoo, this assumption is certain to fall short for
international emails. We now consider that
* if local part is ascii: normalize still 'lower' ;
* else: use as it, SMTP-UF8 is made for non-ascii local parts;
Concerning domain part of the address, as of v14 international domain (IDNA)
are handled fine. The domain is always lowercase, lowering it is fine as it
is probably an error. With the introduction of IDNA, there is an encoding
that allow non-ascii characters to be encoded to ascii ones, using 'idna.encode'.
Also remove usage of 'email_re' in mailing email check. It is too restrictive
compared to real formatting we support (or try to). Valid outgoing emails
were directly canceled, notably when containing unicode.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@3ce5fb3072
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
SPECIFICATIONS
When having multi-emails input in an email field, 'email_normalized' field is
currently 'False', as they expect the field to contain a single email. This
has several drawbacks
* searching partners or fetching information based on emails does not work as
most tool methods use 'email_normalized' which is False (see e.g.
'_message_partner_info_from_emails', '_mail_find_partner_from_emails'
or 'find_or_create');
* blacklist is not available as it is based on 'email_normalized';
* mass_mailing wrongly considers those emails as invalid and cancel their
mail and related trace, as it tries to skip sending emails to invalid
emails;
Be more defensive and use first found email in case of multi-emails field.
Other emails are ignored. It is already an improvement that does not break
flows in stable and allow more emails to be sent.
before
-> email: '"Raoul" <raoul1@raoul.fr>, raoul2@raoul.fr'
-> email_normalized: False
after
-> email: '"Raoul" <raoul1@raoul.fr>, raoul2@raoul.fr'
-> email_normalized: raoul1@raoul.fr
A side effect is that it helps finding back some partners, as indicated in
tests where less phantom partners are created. It also helps suggested
partners / emails flow in discuss.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@90218186c5
Part-of: odoo/odoo#134934
Move tests currently in mail about helpers defined in 'tools/mail.py' directly
into base. No need to test it in mail, there is no specific override or
behavior change compared to base.
Regroup partner related tests in base into 'test_res_partner'. This breaks
part of test history but it helps knowing which features are tested.
Task-2612945 (Mail: Defensive email formatting)
Part-of: odoo/odoo#134934
PURPOSE
Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.
RATIONALE
Add tests related to not standard usage of email field. Two main use cases
are tested here
* formatted emails: `"Full Name" <email@domain.com>` stored into the 'email'
field;
* multi emails: `email1@domain.com, email2@domain.com` stored into a single
'email' field;
Additional tests: tests with unicode / ascii / case / wrong formatting are also
added to check the support in normalize and format methods.
IMPLICATION
Email field is generally managed as "containing a valid email". This means
it is sometimes used as it in 'formataddr' as well as to perform searches or
identification checks. Example of issue: partner 'Raoul' has a formatted email
like "Raoul" <raoul@raoul.fr>. Using 'formataddr' in email_from leads to
from: "Raoul" <"Raoul" <raoul@raoul.fr>>
-> which is incorrect (but often dynamically corrected by email servers);
Email field holding multi-emails are not normalized, as current normalize
is done only if the field holds a single email. It means
* no easy finding based on 'email_normalized', e.g. various tools like
'_mail_find_partner_from_emails' or 'find_or_create' do not find partners
based on this email;
* no exclusion list management;
* issue with formatting, like
to: "Raoul" <raoul@raoul.fr,raoul.other@raoul.fr>
-> which is incorrect (but often dynamically corrected by email servers);
USAGE: OUTGOING EMAILS
Those use cases currently generate faulty outgoing emails. This is valid for
recipients ('email_cc', 'email_to') as well as author ('email_from').
For formatted emails: `email_to` is formatted again based on name and email
which leads to sending emails to `"Full Name" <"Other"<email@domain.com>>`.
Note that multi emails without formatting may work as it leads to email_to
`"Full name" <email1@domain.com,email2@domain.com>`. Some outgoing email
servers correctly send multiple emails. It depends on their fault
tolerance.
USAGE: FIND BASED ON EMAIL (NORMALIZED)
When searching for partners (e.g. using '_mail_find_partner_from_emails' or
'find_or_create') normalized version of input is used.
In case of multi emails sanitize is 'False', as normalization expects a single
email in the field. Therefore no partner is found. In processes that do a
"search or create" (e.g. using a template on a record) this leads to creating
a new partner (or several partners in case of multi emails) each time.
USAGE: OTHER FLOWS
Other flows are build on top of '_mail_find_partner_from_emails' / 'create'
of outgoing emails and are impacted by formatted email / multi email usage.
Those include notably
* mass_mailing: '_message_get_default_recipients' should be defensive to
give correct values when creating mailing emails;
* mass_mailing: faulty emails is based on normalize and multi-emails are
considered as faulty and ignored;
* after post hook: '_message_post_after_hook' tries to link messages without
author (but email_from) with newly-created partners, when partners are
created from chatter. It is therefore impacted by those corner cases;
* marketing_automation: built on top of mass_mailing and suffers from the
same issues;
USAGE: UNICODE
Unicode in emails should be supported. 'formataddr' and IrMailServer notably
received fixes to support unicode. Some check performed on email addresses
fail when unicode is involved, which leads to some emails not being sent
while they could.
SPECIFICATIONS
Add tests related to those corner cases. Also add tests for computation of
`email_formatted` field of Partner model. It currently generates wrong email
values for the same corner cases (multi emails, formatted emails).
Tests are also added for the computation of `email_normalized` field used
notably for blacklists. It is not computed currently when being in multi
email mode which prevents from any blacklist mechanism as well as make
email finding harder. `_mail_find_partner_from_emails` tool method is also
tested with multi email as it uses the same heuristic as normalized email
field.
Tests are also added for mass mailing, when having to mail documents that
have a partner with formatted emails / multi-emails, or that have an email
field with formatted emails / multi-emails.
Also restore a test removed at odoo/odoo@afcb734908 while it should have been
updated to state that email addresses containing non-ascii characters are
supported.
Add some tests for tools methods used in various email processing flows.
Unicode tests are also added.
In future commits we will try to make email usage a bit more defensive to
try to lessen issues with that kind of use cases.
Task-2612945 (Mail: Defensive email formatting)
X-original-commit: odoo/odoo@fc8442f133
Part-of: odoo/odoo#134934
Have all ir.mail.server tests as well as their configuration-related tests
moved into 'test_ir_mail_server' file. That way all tests are contained in the
same file, easing their update.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130768
Rename alias domain and aliases used a test data. This allows to make
them easier to read, follow, grep and understand.
Activate multi-company on alias and gateway tests, ensuring it currently
has few impact on tests.
Task-3453577 (TestMail: Update Alias/Gateway tests for MC)
Prepares Task-36879 (Mail: Support MultiCompany Aliases)
Part-of: odoo/odoo#130768
This commit fixes the malformed comment that would sometimes comment out
the rest of the html resulting in an improper display.
this is due to the new html5 notation --!> not behing understood by
our parser.
this commit replaces any --!> into -->.
this commit also remove <!--> or <!--->
opw-2812488
closesodoo/odoo#126148
X-original-commit: 4fed0c36fedf271acc3520d8219763964753dabe
Signed-off-by: Vranckx Florian (flvr) <flvr@odoo.com>
Since python/cpython#18544, unittest mock is not able to properly find
the "odoo.tools.config".
When trying to patch the `options` dictionnary, it leads to
`AttributeError: module 'odoo.tools.config' has no attribute 'options'`
In order to have the tests
working in python <= 3.11, we have to import the config module and patch
the options in place.
Part-of: odoo/odoo#112450
The fix#14569 was not complete. This fix solve the issue explained in
that PR.
The lxml cleaner transform any utf8 character `U+00A0` to a string
` `. This cause a comparison in the editor to be erroneous
inside `HtmlFieldWysiwygAdapterComponent.updateWidget` when comparing
a value coming from the server with the value coming from the editor.
task-3138358
closesodoo/odoo#110264
X-original-commit: 05e0f65c4120ec7e1420dacab8dc678b8a30f70a
Signed-off-by: David Monjoie (dmo) <dmo@odoo.com>
Purpose of this commit is to avoid html entities in logged message by correctly
managing enclosures. For that purpose a new tool 'nl2br_enclose' is added that
eases Markup management on top of 'nl2br' simple tool.
Task-2710804 (Mail: Clean MailThread Posting API)
Part-of: odoo/odoo#99482
In website(_sale) messages are created from website forms. However those
are technical models, you should always use the MailThread API notably to
ensure values coherency. In our case using message_log seems to be what
original committers wanted to do (even creating a message as a comment
which has no effect as the notification process is not called that way).
Task-2710804 (Mail: Clean MailThread Posting API)
Part-of: odoo/odoo#99482
Send an rich HTML email from Odoo, it lands in spam whereas it would
land in inbox in 13.0.
The problem is due to a missing "MIME-Version: 1.0" header on the email.
This header is correctly set on both the html and text alternatives of
the messages but it should be set on the enveloppe too.
The problem is present in the newer EmailMessage mail API of python that
is used since 14.0. Using the newer API, it doesn't set the header on
the enveloppe itself.
This reverts commit 8663f1e20727c315924bb20fc1de1accf3014c61.
opw-3098621
closesodoo/odoo#109212
X-original-commit: c480d2bf1de7b70892130e74bb7a8d4003a8a783
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
If source is something like
"admin@éxample.com" <admin@éxample.com>
candidates founds are
['"admin@\xc3\xa9xample.com"', 'admin@\xc3\xa9xample.com']
and the first one raises an error because of "".
Malformed addresses should be ignored.
opw-2982426
closesodoo/odoo#102353
X-original-commit: 35ad2dd630a8ed173c6a9275585eac46b9a19363
Signed-off-by: Raphael Collet <rco@odoo.com>
Signed-off-by: Julien Castiaux <juc@odoo.com>
Prepare an informative and nice-looking preview from the main content of the
email body, avoiding buttons/images alt etc.
Links, images, tables are removed.
Whitespace is added after the preview to avoid including the full message in
the preview (with markup).
Tags and attributes are also added to increase compliance with HTML5 standard,
including accessibility.
One additional SQL request is required to fetch the preview sub-template.
Task-2413355
closesodoo/odoo#86266
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Jinja as a templating engine was problematic in differents respect:
- introduce external dependency to Odoo (less controll)
- add another templating mechanism in the stack
- specific feature in qweb cannot be reused
- difficulty in rendering easily editable templates
- more knowledge required with no betterment
By replacing jinja with qweb we can now build tools to edit a qweb
that will work with the previously jinja encoded document
(essentially `mail.template` records).
There is a catch however. Some email fields (eg. email_to) used jinja
syntax for rendering dynamic variables (ie. ${object.something} and
${object.something_that_should_not_be_escaped | safe}).
We still want user to use dynamic variables for some char fields (eg.
subject, from, to, ...). We made a new rendering engine called
"inline_template" that will render an expression enclosed by `{{` and
`}}`.
To be able to edit the templates from the backend interface, a
plugin to the Odoo editor has been made for seamlessly edit the
document.
This qweb plugin includes:
- make dynamic variables (eg. `<t t-out="variable"/>`) not editable
(for preventing the user to shoot himself in the foot)
- group and hide related logical branching (ie. t-if, t-elif, and t-else)
in order to see only one at once
- a floating select input to switch visibility of a particular logical
branching
Task-27033
X-original-commit: odoo/odoo@68182baff4
Part-of: odoo/odoo#77377
Purpose
=======
This commit improves the UI of the mail group module (backend and portal
view), fix some bugs and clean the code.
Bug 1
=====
If a user click 2 times on a confirmation link, an error is raised.
Bug 2
=====
A bug about the website snippets has been fixed;
- Add the snippet to a group in which you are member
- Change the group of the snippet to a group in which you are not member
- The snippet still show "Unsubscribe"
Bug 3
=====
In the portal view, when messages are filtered by month, messages which
are created in the last day of the month appear also in the next month.
Specification
=============
UI
--
Show the Allow / Ban buttons whatever is the state of the message, so
the user can ban / whitelist a message even if there's no pending email.
When we receive an email, remove the "Mailing List" footer. This was a
big issue because when some users reply many times, the footer was
duplicated on each answer.
Improve the form view of the mail group:
- On non-moderated group, show all messages instead of only pending
message
- Show the guidelines stuff on non-moderated groups
Improve the portal view
- show the attachments under every message (not only on opened message)
- improve the template to confirm the subscription of the user
- improve the portal view for mobile device
- do not show rejected messages, even for admin
- do not show the button "X replies" when the mode is not thread
Technical
---------
Split the route "/groups/subscription" into "/group/subscribe" and
"/group/unsubscribe"
Correctly quote the parent email for the emails sent by web version of
Outlook (with the HTML id "divRplyFwdMsg").
ACLs
----
Now only administrators or responsible / moderators can write / unlink
the group.
Fix an ACL issue on the group member where the responsible of a
non-moderated group could not access the members of the group.
Members with same email
-----------------------
Improve the behavior when multiple members have the same email address
(this situation is possible because there's no unique constraint on the
email field of the <res.partner> model.) We now return the most
appropriate member for the given email address and partner.
When unsubscribing by clicking on the confirmation link that we received
by email, remove all members with the same email address.
Links
=====
Task-2599676
See odoo/odoo/pull/73640
closesodoo/odoo#73640
Related: odoo/upgrade#2774
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
PURPOSE
=======
We want to increase the score of the emails sent by Odoo and we want to
avoid them to be marked as spam by the mail clients (gmail, outook...).
SPECIFICATIONS
===============
From filter
-----------
Add a new field on the "ir.mail_server" which is "from_filter". This
field defines the email address for which the outgoing email server can
be used.
The "from_filter" can either define an email address or a domain name.
Use the system parameter "mail.default.from" which allow us to define
a default email address which is used to encapsulate the emails
(default: notifications@<catch.all.domain>).
Mail server priorities
----------------------
When sending an email, we read the FROM header and,
- We first look for a mail server which match the entire mail FROM
in that case, we do not change the email header (not needed)
- If not found, we search a mail server which matches the domain name of
the mail from (do not need to change the headers in that case)
- If not found, find the mail server linked to the "notifications"
email (defined in the system parameter). Then change the FROM header
to the notification email, and put the old one in the name part of
this header.
E.g.
Initial mail from: "Admin" < admin@example.com >
Final mail from: "Admin (admin@odoo.com)" < notifications@odoo.com >
- If no notification email is configured or if no mail server are
found for the notification email, fallback to the old system and
spoof the FROM header. In that case we do not have the choice if we
want to send the email, he will probably be marked as spam.
Sending method priority
-----------------------
In the mail server models, we defined some priorities,
1. Forced SMTP session
2. Forced mail server
3. Try to find the best mail server (see "Mail server priorities")
4. If not found, read the odoo-bin arguments
Bounce
------
As there's no standard for bounce address, we put it in the envelope
(smtp_from). But in some case, it might be considered as spoofing. So,
we use the bounce address ONLY if the mail server is configured for the
entire domain name.
One behavior which might be broken is the following; we send an email as
"std@gmail.com" and the bounce address is on the domain "odoo.com".
Before we received the bounce notifications but we were spoofing the
local part and the domain.
Now
- if a mail server is configured for GMAIL, we do not use the bounce
address (and we might not receive the bounce notification)
- if no mail server is configured for GMAIL, but one is configured for
"odoo.com"
- the FROM header will be "notifications@odoo.com"
- the FROM envelope will be the bounce address
=> In this situation we are spoofing only the local part of the email
but it's allowed as the mail server is configured for the entire
domain name
LINKS
=====
Task-2367946
odoo/odoo#61853odoo/upgrade#1903
Purpose
=======
Revert the commit a757cab857
because the rewriting of the FROM of the emails sent will
be done in a smarter way in master.
Task-2367946
odoo/odoo#61853odoo/upgrade#1903
It seems quite easy to have a void content in the currently new editor
that actually holds a lot of undesired information, notably a font
tag. This is annoying when having behavior based on an html field
being empty or not. In this commit we therefore improve definition
of an 'empty' html field.
Task ID-2532529
PR odoo/odoo#71793
Before this commit there were few cases in which
is_html_empty method was not working as our expectation.
In cases such as "<p class=""><br/></p>", "<p id=""><br/></p>"
and many other cases.
In this commit we improves regex of is_html_empty method.
Mow from this commit all kind of attributes will be consider
in this method.
In this commit we also have added is_html_empty in mail template, portal
values and in report values also for rendering templates.
Task id: 2499504
X-original-commit: fd0a05f2955b9f7e9ae7233afebfd6240c9244dd
Make the email body take the entire space to avoid having some wasted space,
this will also make the user have more focus when designing an email.
Revamp the settings notebook page in order to give more clarity to the user,
and move some fields from the main form have been moved to this section to
have more space in the bottom for the email body.
In the mailing form, when the mailing is sent or is being sent, set fields
which are no longer useful for the user to change to readonly mode.
Add a wizard that enables the user to schedule a mailing, the schedule field is
still kept in the form for the user to be able to change the date when the
mailing is in the queue (if they want to send it sooner).
Display an action helper-style content when the email is empty, because,
currently, the user is left with a big white screen when the email has no
content which is not desirable.
Update the html_empty function to take into account style attributes to better
match the editor's void content.
Hide A/B testing fields from SMS mailing form view as these are not supported
for SMS marketing.
Task-2469409
closesodoo/odoo#68882
Ent-pr: https://github.com/odoo/odoo/pull/68882
Related: odoo/enterprise#18391
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose
=======
We want to be able to force the FROM headers when we sent email in
SMTP. So we can avoid the emails to be considered as spam.
Specifications
==============
This is done with 2 system parameters.
If the system parameter `mail.force.smtp.from` is set we encapsulate all
outgoing email from with the given value.
If the previous system parameter is not set and if both
`mail.dynamic.smtp.from` and `mail.catchall.domain` are set, we
encapsulate the FROM only if the domain of the email is not the same as
the domain of the catchall parameter.
Otherwise we do not encapsulate the email (same behavior as before this
commit).
Task 2367946
See odoo/odoo/pull/61853
closesodoo/odoo#70980
X-original-commit: 08a561505b92d23c4c4ec4094b0cee209ece8753
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Python 3 before 3.8 has a bug that causes the email.policy classes to
incorrectly fold and RFC2047-encode "identification fields" in email
messages. This mainly applies to Message-Id, References, and In-Reply-To
fields.
We are impacted by this bug since odoo/odoo#35929 where we switched to
using the "modern" email.message API.
RFC2047 section 5 clearly states that those headers/fields are not to be
encoded, and that would violate RFC5322.
Further, such a folded Message-Id is considered non-RFC-conformant by
popular MTAs (GMail, Outlook), which will then generate *another*
Message-Id field, causing the original threading information to be lost.
Replies to such a modified message will reference the new, unknown
Message-Id, and won't be attached to the original thread.
The solution we adopt here is to monkey-patch the SMTP policies to
special-case those identification fields and deactivate the automatic
folding, until the bug is properly and fully fixed in the standard lib.
Some considerations taken into account for this patch:
- `email.policy.SMTP` is being monkey-patched globally to make sure we
fix all possible places where Messages are being encoded/folded
- the fix is **not** made version-specific, considering that even in Python
3.8 the official bugfix only applies to Message-Id, but still fails to
protect other identification fields, like *References* and
*In-Reply-To*. The author specifically noted that shortcoming [2].
The fix wouldn't break anything on Python 3.8 anyway.
- the `noFoldPolicy` trick for preventing folding is done with no max
line length at all. RFC5322, section 2.1.1 states [3] that the maximum
length is 998 due to legacy implementations, but there is no provision
to wrap identification fields that are longer than that. Wrapping at
998 chars would corrupt the header anyway. We'll just count on the
fact that we don't usually need 1k+ chars in those headers.
The invalid folding/encoding in action on Python 3.6 (in Python 3.8 only
the second header gets folded):
```py
>>> msg = email.message.EmailMessage(policy=email.policy.SMTP)
>>> msg['Message-Id'] = '<929227342217024.1596730490.324691772460938-example-30661-some.reference@test-123.example.com>'
>>> msg['In-Reply-To'] = '<92922734221723.1596730568.324691772460444-another-30661-parent.reference@test-123.example.com>'
>>> print(msg.as_string())
Message-Id: =?utf-8?q?=3C929227342217024=2E1596730490=2E324691772460938-exam?=
=?utf-8?q?ple-30661-some=2Ereference=40test-123=2Eexample=2Ecom=3E?=
In-Reply-To: =?utf-8?q?=3C92922734221723=2E1596730568=2E324691772460444-anot?=
=?utf-8?q?her-30661-parent=2Ereference=40test-123=2Eexample=2Ecom=3E?=
```
and the expected result after the fix:
```py
>>> msg = email.message.EmailMessage(policy=email.policy.SMTP)
>>> msg['Message-Id'] = '<929227342217024.1596730490.324691772460938-example-30661-some.reference@test-123.example.com>'
>>> msg['In-Reply-To'] = '<92922734221723.1596730568.324691772460444-another-30661-parent.reference@test-123.example.com>'
>>> print(msg.as_string())
Message-Id: <929227342217024.1596730490.324691772460938-example-30661-some.reference@test-123.example.com>
In-Reply-To: <92922734221723.1596730568.324691772460444-another-30661-parent.reference@test-123.example.com>
```
[1] bpo-35805: https://bugs.python.org/issue35805
[2] https://github.com/python/cpython/pull/13397#issuecomment-493618544
[3] https://tools.ietf.org/html/rfc5322#section-2.1.1closesodoo/odoo#55656
X-original-commit: 02b78770147e2eda65a76d29c6f2fc3278581b22
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
PURPOSE
When displaying an email in a list the mail client (gmail, outlook...) computes
a preview based on the content. This preview is generally not well computed as
it contains a lot of garbage and tags while it should contain only relevant
text.
SPECIFICATIONS
To build this preview, all mail clients read the content of the email.
The only way to be able to customize the preview is to add an invisible
HTML element at the beginning of the email with the wanted preview text.
We add at the end of the preview `‌` (zero-width non-joiner) to
fill the end of the preview in order to not have the beginning of the
mail at the end of the preview. It doesn't work with simple space as
the mail clients trim each HTML element content.
Task ID-2172125
PR #49886
Since a recent commit[1], we have a utility method in tools named
'is_html_empty' that checks whether the given html content is
void(containing only formatting tags) or not. However, the re from
this method does not consider the case of self closing tags, fox ex
`<br/>`. In such cases, the method returns Falsy value even if the
content is void.
This commit fixes the issue by considering self-closing void tags
in the regular expression.
commit[1] - https://github.com/odoo/odoo/commit/974f512f5f5d3b9f80a8c3fcde290e4f55cf1230
Task - 2267689
closesodoo/odoo#53167
X-original-commit: 0fd6c9388dc959a9b6b7828850683175edb05b9d
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
It has been a recurrent request from customers to be able to send email
messages to email addresses containing non-ascii characters. [IDNA] is a
domain extension to allow unicode characters in domain names. [SMTPUTF8]
is a SMTP extension to allow unicode in any header.
IDNA defines the [punycode] encoding which translates unicode to an
ascii representation. This encoding MUST be used to encode domains.
SMTPUTF8 is an SMTP extension that allow utf-8 in all headers on the
envelope.
[IDNA] https://tools.ietf.org/html/rfc5890
[SMTPUTF8] https://tools.ietf.org/html/rfc6531
[punycode] https://tools.ietf.org/html/rfc3492
Task: 2116928
opw-2229906
opw-2248251
closesodoo/odoo#47709
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
A plaintext email is displayed in a `<pre/>` tag to conserve spacing.
But since there is no escaping, if in this text there was XML tags or
HTML entities, they would appear as HTML in browser which is not wanted.
Do note that this was not a security issue since the content will still
be subjected to the checks and foundling of HTML emails.
Without the change, the added test would fail because character &,<,>
were not escaped.
opw-2242323
closes#50003closesodoo/odoo#50123
X-original-commit: 932532b5b59e0b71c8e16dadfb2ff36c38764208
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
In odoo empty html is not always completely empty. This tool method allow
to consider html holding void tags as empty html. This happens notably when
using the editor that may store <p><br><p> even if displayed content is void.
Task ID 2126509
PR #41827
Problem description:
Using the chatter, send an email to someone having non-ascii characters
in their name: the body of the received email looks like it contains
a mix of the original body and headers.
Python encodes the name into either base64 or quoted-printable but
leaves some redundant carriage returns at the end of the header. Those
redundant carriage returns are not RFC conformant but are interpreted
as newlines by some email clients, thus are read like a headers/body
separator and the next headers are considered part of the body,
corrupting the email structure.
This problem is internal to Python and has been fixed in 3.8, cfr
bpo-34424 [1], and later backported to 3.7.4. As we also support 3.6
and earlier 3.7 and it is not possible to easily monkey-patch the
function, we fixed it by squashing duplicate carriage returns.
(There is no case where a series of bare CRs can occur in a normal
RFC5322 email message)
Task: 2003936
[1]: https://bugs.python.org/issue34424
Python 2 `email.tools.formataddr` doesn't encode not RFC-2822 compliant
realname to base64 or quoted-printable. This is the wanted behavior to
output formatted email addresses on screen.
Python 3 `email.tools.formataddr` does encode the realname to be
RFC-2822 compliant. This is the wanted behavior when connecting to a
smtp server to send emails.
That method has been deprecated by pep-594 as the new python email API
is capable of automatically formatting email address in a RFC compliant
way. As the method was still in use in a lot of modules as the
preferred way to format email addresses, a refined P3-like
implementation as been included in the tool suite.
This commit changes the default behavior so it mimics P2 implementation
with an easy way to use the P3 behavior.
Task: 2003936
It is possible to get to a situation where Odoo would try to send an email without a `From:` header address.
In such case, you're unlucky if you don't have access to the underlying deployment, or if you use multiple databases in a single Odoo instance and each of them uses a different mail configuration.
To make this configuration easier to use and cover those use cases, here I add support for a new ICP: `mail.default.from`. It will be used when present, so it shouldn't affect existing deployments. When present, it will allow a admin to configure the default sending address just with Odoo itself.
closesodoo/odoo#38874
X-original-commit: 5010ce630a7f0e01d45d99c9083318b50f9f624a
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Add the support to be able to specify exlicitely module, class and method name in test_tags
Before this commit, there is no distinction between module tag and test tag. Meaning
that --test-tag module_name will replace the default +standard tag and execute all test
of the module.
This commit tries to keep the initial behaviour, while addind new features, like explicit
modules tag with /module_name, as well as class (:class) and method (.method)
Some usage examples:
--test_tags /module will execute all standard test of module
--test_tags :class will execute all standard test with class name 'class'
--test_tags .method will execute all standard test with method name 'method'
--test_tags external/module will execute all external test of module
--test_tags */module will execute all tests of module,
--test_tags */module,-standard will execute all non standard tests of module,
--test_tags -/website:TestUiTranslate.test_admin_tour_rte_translator will disable only rte translator test
closesodoo/odoo#34756
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Related to #30326
Before #30326, when calling html_sanitize, some content may be escaped
when it shouldn't. A simple example was images with cid links containing @.
(a first fix for another use case was made in 8aff53733b)
We suspect that this escaping (added in 71a92f46e4)
was made to avoid loosing email with format '<email@domaine>' in html_sanitize
to escape them if html_sanitize is called on plain text.
Anyway, only html should be passed to html_sanitize, and therefore email of format
<email@domain> should already be escaped.
PR #30326 fixes the unwanted behaviour in 12.0 and this commit removes
this logic in master.
closesodoo/odoo#30815
The content id of an attachement can contains a @, therefore
image contained in emails where sometimes escaped by html_sanitize
as if they were emails adresses.
Task: 1915251
closesodoo/odoo#30326
The transcoding of a mail (transforming CSS into inline styles on the
element nodes of the mail) used the form:
property:value;
So when a mail is saved, it would be eg. with `style="property:value;"`
and when it is being edited "property:value" is removed if available via
CSS stylesheet given the structure.
But the sanitize_style (introduced in bfe7aafa7) adds an espace before
the value (which is more pretty but was probably not choosen in
transcoding to decrease size of mail):
property: value;
so the inline style would not be removed when editing possibly causing
conflict with the editor.
With this commit, the sanitize_style doesn't add a space. It could have
been changed in transcoding but the code is already rather slow and we
don't want to use a regex replace or two string replace instead of the
current one string replace.
opw-1841107
Purpose: When running tests, all the tests for the installed/updated
files are done. This commit adds a 'tagged' decorator that can be used to
tag tests. Combined with a new 'test-tags' CLI option, it adds the ability
to filter which tests are executed. For example, @tagged('slow') will
add a tag 'slow' to the test. The CLI option 'test-tags="slow"' will
only run tests tagged 'slow'.
One can use prefixes to select cases with tags.
'+' or no prefix means that the tests tagged with this tag are selected
for execution. '-' prefix will exclude the tests tagged with this tag.
Exclusion takes precedence over inclusion.
Also, by default, all Odoo tests cases are tagged 'standard' and with
the technical name of the module.
This means that when selecting tests with the 'test-tags'
parameter, if '-standard' is not specified, all tests tags are
going to be executed.
When tagging tests, one can remove such automatic tag by prefixing the
tag name with '-'. E.g. @tagged('-standard') will remove the standard
tag from the test.
Another example, if one wants to test the 'sale' module alone,
even without adding any 'tagged' decorator thos tests can be selected
like that: --test-tags="sale"
Tests are selected or deselected using a TagsSelector. When instanciated,
a string is passed with comma separated tests selectors like
'+slow,-standard'. When the 'check' method is called with a test as argument,
it returns True or False if the test has to be executed or not.
No formal decision there, and the stdlib API seems like a PITA as it's
based around native strings, not bytes and not text.
* try to use cross-model API instead of random encode/decode
* unicode all the mail_template things
* formataddr *requires* a text tuple now
In P2, formataddr((False, False)) "worked" (generated "False
<False>"), in P3 it will fail and blow up.
* attachments are quite definitely bytes
* remove try_coerce_ascii because it's garbage
cgi.escape is unsafe (quote=False by default) and deprecated in Python
3. We already have an openerp.tools.misc.html_escape version which
forwards to the (modern and safe) werkzeug.utils.escape, just use that
everywhere, and convert extant uses of werkzeug.utils.escape to
utils.html_escape as well so that we do the same thing everywhere.
Thibault Delavallée