A sanitize fields by default remove all attributes outside of a
whitelist (if sanitize_attributes on field is not set to false).
This prevent to use parallax widget in Html sanitized field since
attribute "data-scroll-background-ratio" define parallax effect speed
but was not in whitelist.
note: also keep data-interval for gallery widget
opw-2053983
closes#35961
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Parsing email values: make message_parse effectively prepare all required
values for later processing
* update message_parse to add some more data directly in parsed dictionary
and avoid having to parse value again in later processing and computation.
* notably recipients computation is now done directly in parsing. It
includes email check and reconstruction using tools methods. That way
routing just has to handle those values and not compute part of them;
* from and cc are correctly managed with parsed and cleaned (sanitized)
versions in dictionary;
* remove all code decoding message header and replace them by fetching
the parsed value instead;
* filter parsed values before calling message_post to ensure right values
are given to it instead of having to manage them in message_post. Indeed
caller has to give the right input;
* make parsing sub-methods (extract payload, post process payload) have
a dictionary as input and output to always manage a dictionary of values
through the parsing methods and update their namespace;
Routing: remove deprecated code parts
* remove use of email_references regex. Indeed emails are routed using their
messageId and not the regex anymore since several versions [1];
* clean some unnecessary variables;
* simplify variable type input, like message_parse accepts only valid
email.message instances and to avoid several decode / encode. Tests are
updated accordingly;
Update docstrings of those methods.
LINKS
Related to task 1893155
Linked to PR #33340
[1] See df037e2959 and 0028c42136
As Flanker can be used to validate an email address and was originally
used in website_crm_score module, the Flanker usage is now moved into tools.
This commit prepares the next Predictive Lead Scoring (aka PLS) ones
as PLS will use the email state as computation variable,
it will need this flanker validation to compute first the email state.
We use also the already implemented normalized email as a double validation.
Flanker is not in the required dependencies of Odoo :
So if flanker is installed, it will check if email is valid
(checking DNS and other standard rules).
If Flanker is not installed, email_validate will only normalize the email.
Anyway, the email must at least be normalizable.
Task ID : 1925439
PR #33589
In mail custom layout template display 'best regards' and signature in email
only when user exists and is not odoobot.
Border-bottom style is also now white-listed in sanitizer in order to ease
customization of emails.
Related to task ID 1873634
Linked to PR #28781
*: crm, project, maintenance, hr_recruitment
When a record is created from an email, cc can be lost. This commit
proposes a new mixin to keep cc on the record and allows to create
partner for each of them when sending a message from the chatter.
The mixin is added on the mail recors that can be created from mail.alias:
document
helpdesk.ticket
mrp.eco
quality.alert
hr.applicant
crm.lead
project.task
mail.channel
maintenance.equipment
Task: 1925001
Related to #30326
Before #30326, when calling html_sanitize, some content may be escaped
when it shouldn't. A simple example was images with cid links containing @.
(a first fix for another use case was made in 8aff53733b)
We suspect that this escaping (added in 71a92f46e4)
was made to avoid loosing email with format '<email@domaine>' in html_sanitize
to escape them if html_sanitize is called on plain text.
Anyway, only html should be passed to html_sanitize, and therefore email of format
<email@domain> should already be escaped.
PR #30326 fixes the unwanted behaviour in 12.0 and this commit removes
this logic in master.
closesodoo/odoo#30815
The content id of an attachement can contains a @, therefore
image contained in emails where sometimes escaped by html_sanitize
as if they were emails adresses.
Task: 1915251
closesodoo/odoo#30326
Odoo no longer supports python 2, thus some of these helpers can and
have been replaced by python 3 built-ins, therefore there is no need for
them to stay defined.
The removed helpers are:
* izip, imap and ifilter
* unichr, text_type
* implements_to_string, implements_iterator
* string_types, integer_types
* to_native
The python 2 shims have also been removed, and only the python 3 helpers
have been kept, because they can still be usable (i.e. accepting
both bytes and str for functions that can only accept one of the two)
[REM] pyjsparser: remove PY3 shims
They're no longer necessary as Odoo doesn't officially support python 2
anymore.
closesodoo/odoo#28519
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
Because we converted buttons into tables so inline buttons was
breaking into multiple lines so we need to add display inline and vertical align
properties in tables.
We allowed display property from mail sensitizer whitelist.
Because in this commit https://github.com/odoo/odoo/commit/cf15030ed09f7479f5b2701ba101a5bed71f7409
we use display property for align items
Some of the templates Odoo proposed do not display well on all email readers
as too much styling attributes are sanitized. This commit adds some styling
in the white list to allow having stylized templates while still having a
coherent white list.
This commit is related to task 51122 (and PR #24052).
The transcoding of a mail (transforming CSS into inline styles on the
element nodes of the mail) used the form:
property:value;
So when a mail is saved, it would be eg. with `style="property:value;"`
and when it is being edited "property:value" is removed if available via
CSS stylesheet given the structure.
But the sanitize_style (introduced in bfe7aafa7) adds an espace before
the value (which is more pretty but was probably not choosen in
transcoding to decrease size of mail):
property: value;
so the inline style would not be removed when editing possibly causing
conflict with the editor.
With this commit, the sanitize_style doesn't add a space. It could have
been changed in transcoding but the code is already rather slow and we
don't want to use a regex replace or two string replace instead of the
current one string replace.
opw-1841107
Currently crm and project parse incoming email body following an old
syntax to find some "set field - value" lines in the email content. It is
used to update the record linked to the discussion directly from the email.
However this feature is undocumented and not really supported anymore.
Moreover we don't think being able to update records from emails is a
reliable way of working. Moreover those update is limited to
* revenue and probably for leads; this should totally be managed from
the pipeline view by lead / opportunity responsible;
* priority update coming from a "priority" entry in the incoming email;
this code seems strange as priority is probably not standard in email
headers;
* planned hours for project; this should be managed by project user
and managers through dedicated views in Odoo;
Let us remove that old feature and let people use desktop or mobile
interface to update their records.
Before this commit, if comments were found inside the body of a mail
template sent through mass_mailing, the regex in append_content_to_html
that converts possible uppercase tags to lowercase tags would replace
comments by broken html elements, this would break the in-browser parser
and thus would show partial content or no content at all, depending on
the positioning of the comments.
i.e. from <!-- This is a comment --> to <this -->
The regex in question is r'(</?)\W*(\w+)([ >])' which basically matches
opening and closing tags (this was a bad idea from the beginning btw),
the \W* will match any non-alphanumeric chars (so !-- in the
case of comments) and this would be completely discarded since it's not
a match group and when replacing we only use match groups.
This is fixed by simply stripping the \W* since it's discarded anyway,
an alternative fix would be to put it in a match group and add it when
replacing as well, but I don't know if we purposefully discard it or
not.
Neither fix is the good one, the good fix is to replace all the re.subs
by lxml which is much easier and more concise.
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
No formal decision there, and the stdlib API seems like a PITA as it's
based around native strings, not bytes and not text.
* try to use cross-model API instead of random encode/decode
* unicode all the mail_template things
* formataddr *requires* a text tuple now
In P2, formataddr((False, False)) "worked" (generated "False
<False>"), in P3 it will fail and blow up.
* attachments are quite definitely bytes
* remove try_coerce_ascii because it's garbage
* Have the cleaned up style attributes be dumped in the same order
they were originally so that we can easily validate against text
content.
* While lxml's attributes are stored in an ordereddict, namespaces are
not and thus when comparing text serialisation of namespaced
documents the ordering of namespaces can differ between runs.
Fix by comparing the LXML trees directly.
``import odoo.addons.foo as bar`` doesn't seem to properly trigger the
import hook in Python 3 (it blows up on decimal_precision and
base). Thus convert *all* examples of that pattern to the more
sensible ``from odoo.addons import foo as bar``.
cgi.escape is unsafe (quote=False by default) and deprecated in Python
3. We already have an openerp.tools.misc.html_escape version which
forwards to the (modern and safe) werkzeug.utils.escape, just use that
everywhere, and convert extant uses of werkzeug.utils.escape to
utils.html_escape as well so that we do the same thing everywhere.
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
For the purpose of fighting spam, many abuse/spam-fighting
services provide redacted message IDs. Because we use a UNIX
timestamp as the prefix for all message IDs, a redacted
message ID of the form 1234XXXXX@domain.com is generally
useless to find a `flagged` message's origin, because it
matches most of the messages sent that day.
Narrowing it down with the help of the approximate time
of delivery does not help much, as there will still be
dozens/hundreds of possible matches on a busy server.
By swapping around the random token and the timestamp in
the message ID, we should improve the uniqueness and
"greppability" of our message IDs, once redacted.
lxml.html.clean module we are using to sanitize html takes either a
tree/element or string, and return us an element of the same type.
We are passing it an unicode string, and it gives us back an unicode
string and if it is possible convert entities to utf-8.
This lead to an issue in html inline fields containing an ` `
entity, the entity was converted by the sanitizer to its unicode
character (U+00A0).
Then, when the field value is loaded to be displayed, the internal
value contains U+00A0 characters, but when it is inserted in the page
the browser converts them to ` `. Hence in this instance, the
internal value is never equal to the displayed value and a write is
always triggered even without editing.
This commit simple modify the sanitizer so value saved in database
contain ` `.
closes#14569
opw-689274
Christophe Simonis