Purpose
=======
Currently a custom override of Cleaner.allow_element()
exists in tools to allow object tags for SVG images.
This was due to the first prototype of website builder
and its first implementation of image and svg management.
This is not necessary anymore as using the tag is sufficient.
We can safely clean code in the cleaner, allowing to speedup
its performances. As it is used in most html fields and email
parsing each unnecessary code removed is time gainged.
Task 2215228
closesodoo/odoo#51294
X-original-commit: 4a0b09fd3b0d3b63f2df00f002e4d6825cac4eef
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
It has been a recurrent request from customers to be able to send email
messages to email addresses containing non-ascii characters. [IDNA] is a
domain extension to allow unicode characters in domain names. [SMTPUTF8]
is a SMTP extension to allow unicode in any header.
IDNA defines the [punycode] encoding which translates unicode to an
ascii representation. This encoding MUST be used to encode domains.
SMTPUTF8 is an SMTP extension that allow utf-8 in all headers on the
envelope.
[IDNA] https://tools.ietf.org/html/rfc5890
[SMTPUTF8] https://tools.ietf.org/html/rfc6531
[punycode] https://tools.ietf.org/html/rfc3492
Task: 2116928
opw-2229906
opw-2248251
closesodoo/odoo#47709
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
A plaintext email is displayed in a `<pre/>` tag to conserve spacing.
But since there is no escaping, if in this text there was XML tags or
HTML entities, they would appear as HTML in browser which is not wanted.
Do note that this was not a security issue since the content will still
be subjected to the checks and foundling of HTML emails.
Without the change, the added test would fail because character &,<,>
were not escaped.
opw-2242323
closes#50003closesodoo/odoo#50123
X-original-commit: 932532b5b59e0b71c8e16dadfb2ff36c38764208
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
* = event, website_event_track, website_sale, website_hr_recruitment,
website_livechat, website_sale, website_slides
The option to sanitize or not the forms was not available, this will
allow better flexibility on whether forms should be sanitized or not on
an HTML field.
Also we use this new param to allow forms to be added on some already
existing html fields where forms where sanitized out.
task-2209554
closesodoo/odoo#47318
Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
PURPOSE
Move links shortening tools on mail.render.mixin to have rendering and
post procesing tools available on that mixin.
SPECIFICATIONS
Move html and text links shortening methods on mail.render.mixin class.
That way rendering and post processing is done in that mixin instead of
being specific to link.tracker model. Indeed those methods are tools and
not really business related methods.
Future commits will improve the way mail and sms composer use the render
mixin. This commit is therefore both a cleaning and preparing commit.
LINKS
Task ID 1963529
Community PR odoo/odoo#32397
PURPOSE
When incoming emails bounce due to alias security bounce email is quite
generic. Purpose of this task is to ease its customization and update.
SPECIFICATIONS
In order to improve the flexibility of alias, add a customizable html field
on the alias model. This html content will be send as bounce email core content
in case of bounced/unauthorized mail received for this alias,
Obviously it has no effect on 'everyone' security setting as no email will
bounce due to that issue.
If it is not set a default generic mail will be send depending on security
setting. It allows to keep void html fields when no specific bounce content
is required
In HR, an old template allowing some light customization for employee based
security option is removed as it is completely replaced by the new feature.
Also add references message-id of the mail received to the answer so that
threads are correctly set.
LINKS
Task ID 2126509
In odoo empty html is not always completely empty. This tool method allow
to consider html holding void tags as empty html. This happens notably when
using the editor that may store <p><br><p> even if displayed content is void.
Task ID 2126509
PR #41827
When we send a SMS in the contact form view, exact body is sent by SMS and
displayed in chatter. If some manual HTML is added chatter will display it
as HTML while sms receive HTML tags.
We want that the SMS content in the chatter is the same as the SMS sent and
that HTML tags are removed to avoid being interpreted.
To achieve that goal we call html2plaintext in ``_message_sms`` and in
``prepare_log_body_value`` that are two entry points to send SMS.
We also update ``html2plaintext`` to strip result in order to avoid having
unnecessary spaces left.
Task ID 2126123
PR #40441closesodoo/odoo#46362
X-original-commit: 4b7b14a5ea5e6ab521838c01813fa478154682fd
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose of this commit is to better support custom styling used notably
for mass mailing and outgoing emails. Indeed styling is whitelisted when
storing html. Finding it in html requires some parsing. This commit improves
the regex used to find it when semicolons are involved. It allows more use
cases to be correctly supported.
Task ID #2125856
PR #40433closesodoo/odoo#41969
X-original-commit: 5235a49a5cd0f79f5590c96afa26dce25465fe06
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Python 2 `email.tools.formataddr` doesn't encode not RFC-2822 compliant
realname to base64 or quoted-printable. This is the wanted behavior to
output formatted email addresses on screen.
Python 3 `email.tools.formataddr` does encode the realname to be
RFC-2822 compliant. This is the wanted behavior when connecting to a
smtp server to send emails.
That method has been deprecated by pep-594 as the new python email API
is capable of automatically formatting email address in a RFC compliant
way. As the method was still in use in a lot of modules as the
preferred way to format email addresses, a refined P3-like
implementation as been included in the tool suite.
This commit changes the default behavior so it mimics P2 implementation
with an easy way to use the P3 behavior.
Task: 2003936
It seems the emails > < are missing.
By the way one manual formataddr is replaced by an email_formatted field.
Result is the same but let us use fields doing it for us.
These functions convert plain text links into clickable hyperlinks.
These open in new tabs but did not have the "noreferrer noopener" rel
attributes, which made them susceptible to reverse tabnabbing.
A lot of attack vectors were available to unregistered, uninvited
anonymous users and presented a significant phishing threat (such as
posting links in the instant-chat, through a mail-alias, in a forum post
or in a twitter post) and leading the operator to believe he had been
disconnected from odoo in the original tab, prompting them to enter
their credentials.
while these three places will add the noreferrer and noopener attributes
on the anchor tags generated by them, there are still many places that
create hyperlinks without the use of these functions, although most of
them are static links, they still represent a transitive security
vulnerability to the linked sites.
There are also a few modules and widgets that roll out their own links
or open new tabs unsafely using window.open(), these will need to be
patched separately.
closesodoo/odoo#37591
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Since it appears that email validation feature from Flanker v0.9.0
is completelly broken (the lib explodes when calling
address.validate_address('some@address.xxx')), Flanker is then removed
and email validation will be done only using the email_normalize method.
Task ID: 2044539
PR #37413
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules: `email.(charset|header|mime|utils)`.
The new `email.message.EmailMessage` API exposes a much easier interface
to create multiparted emails [1], is capable of doing all the necessary
headers value conversion (RFCs [2045], [2047], [2049]) [2] and get/set
different flavor of payload (text/bytes) in a straightforward way [3].
All headers are structured in a way to support python native types, i.e.
the `Date` header supports `datetime.datetime` objects and automatically
performs the required formatting. The same goes for multi-valued headers
like the `To` header, one can directly set a python list of values, it
will be automatically be formatted according to the RFCs.
The dedicated encoding and decoding functions are no more needed thus
has been removed has part of the refactor. FTR, [RFC2231] is an update
of [RFC2047] and based on tests we've just conducted, both GMail and
Thunderbird now support RFC2231 encoding just fine in all headers:
attachments names, From, etc.
[1]: http://docs.python.org/3/library/email.message.html
[2]: http://docs.python.org/3/library/email.headerregistry.html
[3]: http://docs.python.org/3/library/email.contentmanager.html
[2045]: https://tools.ietf.org/html/rfc2045
[2047]: https://tools.ietf.org/html/rfc2047
[2049]: https://tools.ietf.org/html/rfc2049
[2231]: https://tools.ietf.org/html/rfc2231closesodoo/odoo#35929
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules. Among them the `email.utils.formataddr`
function, this function takes a `(name, email)` pair and returns a
string value suitable for From, To and Cc headers.
The stdlib function is capable of handling several character encoding
and two binary-to-ascii encoding. Odoo only uses `utf-8` which is
compatible with the base64 b2a encoding. The re-implementation of that
function has been simplified to only support base64-ed utf-8 and ascii.
[PEP 594]: https://www.python.org/dev/peps/pep-0594/\#email-lagacy-api
A sanitize fields by default remove all attributes outside of a
whitelist (if sanitize_attributes on field is not set to false).
This prevent to use parallax widget in Html sanitized field since
attribute "data-scroll-background-ratio" define parallax effect speed
but was not in whitelist.
note: also keep data-interval for gallery widget
opw-2053983
closes#35961
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Parsing email values: make message_parse effectively prepare all required
values for later processing
* update message_parse to add some more data directly in parsed dictionary
and avoid having to parse value again in later processing and computation.
* notably recipients computation is now done directly in parsing. It
includes email check and reconstruction using tools methods. That way
routing just has to handle those values and not compute part of them;
* from and cc are correctly managed with parsed and cleaned (sanitized)
versions in dictionary;
* remove all code decoding message header and replace them by fetching
the parsed value instead;
* filter parsed values before calling message_post to ensure right values
are given to it instead of having to manage them in message_post. Indeed
caller has to give the right input;
* make parsing sub-methods (extract payload, post process payload) have
a dictionary as input and output to always manage a dictionary of values
through the parsing methods and update their namespace;
Routing: remove deprecated code parts
* remove use of email_references regex. Indeed emails are routed using their
messageId and not the regex anymore since several versions [1];
* clean some unnecessary variables;
* simplify variable type input, like message_parse accepts only valid
email.message instances and to avoid several decode / encode. Tests are
updated accordingly;
Update docstrings of those methods.
LINKS
Related to task 1893155
Linked to PR #33340
[1] See df037e2959 and 0028c42136
As Flanker can be used to validate an email address and was originally
used in website_crm_score module, the Flanker usage is now moved into tools.
This commit prepares the next Predictive Lead Scoring (aka PLS) ones
as PLS will use the email state as computation variable,
it will need this flanker validation to compute first the email state.
We use also the already implemented normalized email as a double validation.
Flanker is not in the required dependencies of Odoo :
So if flanker is installed, it will check if email is valid
(checking DNS and other standard rules).
If Flanker is not installed, email_validate will only normalize the email.
Anyway, the email must at least be normalizable.
Task ID : 1925439
PR #33589
In mail custom layout template display 'best regards' and signature in email
only when user exists and is not odoobot.
Border-bottom style is also now white-listed in sanitizer in order to ease
customization of emails.
Related to task ID 1873634
Linked to PR #28781
*: crm, project, maintenance, hr_recruitment
When a record is created from an email, cc can be lost. This commit
proposes a new mixin to keep cc on the record and allows to create
partner for each of them when sending a message from the chatter.
The mixin is added on the mail recors that can be created from mail.alias:
document
helpdesk.ticket
mrp.eco
quality.alert
hr.applicant
crm.lead
project.task
mail.channel
maintenance.equipment
Task: 1925001
Related to #30326
Before #30326, when calling html_sanitize, some content may be escaped
when it shouldn't. A simple example was images with cid links containing @.
(a first fix for another use case was made in 8aff53733b)
We suspect that this escaping (added in 71a92f46e4)
was made to avoid loosing email with format '<email@domaine>' in html_sanitize
to escape them if html_sanitize is called on plain text.
Anyway, only html should be passed to html_sanitize, and therefore email of format
<email@domain> should already be escaped.
PR #30326 fixes the unwanted behaviour in 12.0 and this commit removes
this logic in master.
closesodoo/odoo#30815
The content id of an attachement can contains a @, therefore
image contained in emails where sometimes escaped by html_sanitize
as if they were emails adresses.
Task: 1915251
closesodoo/odoo#30326
Odoo no longer supports python 2, thus some of these helpers can and
have been replaced by python 3 built-ins, therefore there is no need for
them to stay defined.
The removed helpers are:
* izip, imap and ifilter
* unichr, text_type
* implements_to_string, implements_iterator
* string_types, integer_types
* to_native
The python 2 shims have also been removed, and only the python 3 helpers
have been kept, because they can still be usable (i.e. accepting
both bytes and str for functions that can only accept one of the two)
[REM] pyjsparser: remove PY3 shims
They're no longer necessary as Odoo doesn't officially support python 2
anymore.
closesodoo/odoo#28519
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
Because we converted buttons into tables so inline buttons was
breaking into multiple lines so we need to add display inline and vertical align
properties in tables.
We allowed display property from mail sensitizer whitelist.
Because in this commit https://github.com/odoo/odoo/commit/cf15030ed09f7479f5b2701ba101a5bed71f7409
we use display property for align items
Some of the templates Odoo proposed do not display well on all email readers
as too much styling attributes are sanitized. This commit adds some styling
in the white list to allow having stylized templates while still having a
coherent white list.
This commit is related to task 51122 (and PR #24052).
The transcoding of a mail (transforming CSS into inline styles on the
element nodes of the mail) used the form:
property:value;
So when a mail is saved, it would be eg. with `style="property:value;"`
and when it is being edited "property:value" is removed if available via
CSS stylesheet given the structure.
But the sanitize_style (introduced in bfe7aafa7) adds an espace before
the value (which is more pretty but was probably not choosen in
transcoding to decrease size of mail):
property: value;
so the inline style would not be removed when editing possibly causing
conflict with the editor.
With this commit, the sanitize_style doesn't add a space. It could have
been changed in transcoding but the code is already rather slow and we
don't want to use a regex replace or two string replace instead of the
current one string replace.
opw-1841107
Stéphane Debauche