For large lists of blocked emails, the time taken to load the records in
cache becomes prohibitive. For instance on a sample blocklist of 600k
entries, the `search([])` to load them could take 80-90s to run!
And this toll is taken every time the mail composer processes an email
batch in mass-mailing mode. Technically, most of the time is spent
iterating on the list of ids many time, in order to prefetch fields
into cache by determining what's missing.
Loading the contents of the list in raw SQL takes a fraction of that
time (400ms vs 90s).
Subsequently using a set for lookups in that blocklist is also much
faster (average time complexity O(1) vs O(n)).
Example: looking up an item in a 600k-entry blocklist is easily 5
orders of magnitude faster!
```
In [1]: blocklist = set(x[0] for x in self._cr.fetchall())
In [2]: len(blocklist)
Out[2]: 634610
In [3]: self._cr.execute("SELECT email from mail_blacklist")
In [4]: blocklist = set(x[0] for x in self._cr.fetchall())
In [5]: %timeit "hello@hello.com" in blocklist
The slowest run took 35.29 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 31.6 ns per loop
In [6]: self._cr.execute("SELECT email from mail_blacklist")
In [7]: blocklist = [x[0] for x in self._cr.fetchall()]
In [8]: %timeit "hello@hello.com" in blocklist
100 loops, best of 3: 2.24 ms per loop
```
closesodoo/odoo#57129
X-original-commit: 3b9a85754f30f0bef508d0b576ff679c87ead5da
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
PURPOSE
Since v13 some commits added code in mail thread file. This file is already
quite long and keeping it organized allow to understand its content.
SPECIFICATIONS
Some methods defined on mail.thread may actually be called on models
not inheriting from mail.thread . Instead of having model methods on mail
thread receiving a records parameter it seems better to have those available
directly on BaseModel.
Move static_message_track on BaseModel in models.py. This method is now called
_mail_track, to be coherent with othe rmail-related naming. It is used in
accounting to track values in line model that does not inherit from mail.
thread. Update accounting accordingly (followup of d862965).
Move _message_get_default_recipients_on_records in models.py. Renaming it
_message_get_default_recipients() allow to be compatible with current behavior
and current override available in some addons (like CRM, event, ...).
Move _notify_get_reply_to_on_records in models.py. Renaming it
_notify_get_reply_to() allows to be shorter and coherent.
Move _alias_check_contact_on_record in models.py. Renaming it
_alias_check_contact_() allows to be shorter and coherent. Its override in
hr is also moved on BaseModel.
LINKS
Task ID-2327096 (code cleaning)
PR #56631
X-original-commit: f5df1ed912455e5ed52a65df3149f30a9d424de0
The ir.autovacuum model purpose is to run several garbage collecting
operations like removing files from the filestore when no attachment
references them anymore.
The precedent strategy to register new garbage collection tasks was to
override the `power_on` method and to imperatively execute a vacuum
cleaning method on a given model. All calls were executed in a single
SQL transaction without any error handling, meaning a single fail during
any call resulted in a complete failure of the entire vacuum cleaning
chain.
We introduce a new `@autovacuum` api decorator, its purpose it to
register garbage collecting methods that will be safely executed in
their own transaction by the vacuum cleaner. In order to ensure this
new strategy is used, we deprecate `power_on` extensions.
By the way, garbage-collecting methods can be quite heavy and we don't
want users to directly call them. We now ensure they are private.
closesodoo/odoo#47842
Task: 2154079
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
Co-authored-by: Olivier Dony <odo@odoo.com>
We also have to update code calling directly the rendering itself. Indeed
some code bits does some rendering directly on jinja-enabled input and not
through templates. Those calls have to be updated accordingly.
LINKS
Task ID 1963529
Community PR odoo/odoo#32397
Currently the message for help tooltip of auto_delete field is not sufficiently
clear for user to understand.
In this commit, improving message for help tooltip by adding some extra details
in it so user can easily understand about it for models mail.mail, mail.template
and wizard mail_compose_message.
Task-2055702
Closes https://github.com/odoo/odoo/pull/45063closesodoo/odoo#45063
Signed-off-by: Alexandre Kühn (aku) <aku@odoo.com>
RATIONALE
Mail template model holds a field telling odoo mail engine to automatically
add the current user's signature to the body. Its use depends on the use
case
* using the template in the composer on a single record: it is displayed
in the rendered template in the composer, meaning people could change it.
This behavior is interesting as it allows to see the email content;
* using the template in the composer in mass mail mode: it is not displayed
as only the raw jinja is displayed. It is therefore not obvious that it
will be appended to the body of the mail. People could add it manually and
have 2 signatures as a result;
A mechanism automatically adding a signature to sent emails when posting a
message is already implemented and is based on template existence. If a
template has been used when posting, no signature is added in sent emails.
Otherwise it is automatically added. This behavior should not change.
Behavior will therefore be
* use a template -> specify signature usage in it manually through jinja;
* do not use a template -> signature added in sent emails;
SPECIFICATIONS
Remove user_signature.
Update template body accordingly. In customer oriented templates that are using
it and do not already contain it, manually add a call to user.signature within
the jinja code. When set to False, just remove its declaration.
Quickly clean some signature integration.
LINKS
Task ID 2089252
Community PR odoo/odoo#39482
Enterprise PR odoo/enterprise#6459
Upgrade PR odoo/upgrate#761
Related: odoo/enterprise#6459
Related: odoo/upgrade#761
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Co-authored-by: Michaël Mattiello <mcm@odoo.com>
Co-authored-by: Thibault Delavallée <tde@odoo.com>
When simply need to parse a domain, it is easier
Add .strip() on ir.ui.view as ast.literal_eval produces an syntax
error if the node starts with spaces (as done in the xpath of
hr_attendance.view_employee_form_inherit_hr_attendance)
closesodoo/odoo#43831
Related: odoo/enterprise#7894
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
From now on mail.mail is considered as a technical model. Indeed people should
not really manually craft mails by hand. Instead various functional flows
should either send mails, either craft mails based on some user input.
We therefore make mail restricted to admin users. Flows creating mail.mail
are updated to use sudo, and ensure it was done in a context that makes
sense to delegate this power to the user.
Task ID 1853147
PR #32243
Purpose of this commit is to correctly compute author_id and email_from
in mail_message and mail_mail as they depends from each other. Moreover it
is a good idea in various flows to specify email and author when giving
creation values to avoid default computation that is not always guaranteed to
be accurate notably when involving super user.
Mail message creation could lead to desynchronized values between author
and email_from. This is improved with this commit by correctly inheriting
from default_get and computing both of them at the same time instead of having
two default values. Indeed they depend on each other.
Same thing is done for mail composer. Mail Thread offers a tool method to
find email_from / author_id based on having one of those values or current
user and it is called whenever necessary.
Some calls to mail template send_mail are also cleaned.
Task ID 1853147
PR #32243
Purpose of this commit is to remove code coming from old features,
implementations and that was not completely cleaned.
Including
* remove support of old "reply" composition mode which is simply a comment
with a parent message, and is deprecated;
* remove support of old "mail.compose.message.mode" context key that was
used in Odoo 6.1;
* remove support of message_id default value in context instead of
default_parent_id;
* remove old redirection of user / partner post. This code come from
old age when people were able to post on their "profile" by posting
on their user;
* remove unnecessary method for selection field;
* remove unused imports to clean a bit code;
We also make some optional parameters in rendering methods mandatory to
ease code understanding and future modifications.
Task ID 1853147
PR #39272
if partner_ids and template_id are set, a traceback happens.
Indeed, some mail fields are not taken from the res_ids when partner_ids
is set.
When trying to update the values with the one from the template, the
existing values are popped, which raised an error as those keys weren't
set.
closesodoo/odoo#38434
X-original-commit: c8ad614718674b55e9af5edd062ded958136e7ac
Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
[PEP 594] is going to depreciate the legacy `email.message.Message` API
and its related modules. Among them the `email.utils.formataddr`
function, this function takes a `(name, email)` pair and returns a
string value suitable for From, To and Cc headers.
The stdlib function is capable of handling several character encoding
and two binary-to-ascii encoding. Odoo only uses `utf-8` which is
compatible with the base64 b2a encoding. The re-implementation of that
function has been simplified to only support base64-ed utf-8 and ascii.
[PEP 594]: https://www.python.org/dev/peps/pep-0594/\#email-lagacy-api
This branch is the combination of several optimizations in the ORM:
* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;
* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;
* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);
* make method `modified` take advantage of inverse fields to inverse
dependencies;
* filter records by evaluating a domain on records in Python;
* a computed field with `readonly=False` behaves like a normal field
with an onchange method;
* computed fields are computed in superuser mode by default.
Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.
closesodoo/odoo#35659
Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
Since #32404, the fallback case where we have no recipients and no parent_id
on message notify has a typo on mail_message. A user will receive a traceback
when using compose message wizard on a record not inheriting from mail_thread if
he forgot to add recipients.
If the typo should be fixed, notifying the user that no recipients were found and thus,
message_notify won't do anything could be interresting. We can raise a UserError
to give a more explicit failure reason.
We would like to avoid to raise this error on a low level like in message_notify
to avoid to break mail_gateway mechanism, but raising it from mail.compose.message
send_mail shouldn't have any side effect.
We can simply check the output of message_notify to know if something was done
to avoid to duplicate logic used to define recipients.
task-2046180
closesodoo/odoo#35378
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
PURPOSE
Add some improvements in mail gateway: remove private discussion, improve
bounce management, allow resetting bounce counters, improve automatic set or
reset of blacklists and ease mass mailing inheritance.
SPECIFICATIONS
Private discussion support in mail gateway means having incoming messages
not linked to a document. It means having a void model and res_id on related
messages.
Old private conversations in Odoo were implemented using messages not linked
to any document [1]. It was possible to send an email to some people through
the UX and they were able to answer it. Messages were not attached to any
document and model. There were some limitations to this feature, notably it was
limited to replies (no beginning of private discussion through mail gateway).
One way of doing it was through the use of res.users aliases [2]. There was
also a vaguely twitter-lite use of chatter on HR employee profiles linked
to users aliases, then removed with an explicit support of private
discussions [3].
It is not supported anymore since discussion between users is now done using
chat. Aliases on users have since been removed [4]. Contacting partners or
customers can be done on business documents or on channels. Simple email
discussion is not supported in Odoo as it is not the purpose of mailgateway.
This commit removes code related to private discussion in mail gateway and
some specific support in message_post method.
LINKS
Related to task 1893155
Linked to PR #33340
[1] History is a bit messy, see notably fd90140d7b and commits around
[2] User alias addition 052f2ace57
[3] End of employee twitter-like and posting on users = private discussion 89896f32bb
[4] User alias removal 029d1baf35
This attribute is misleading as it is insufficient to correctly upgrade
the database. It only renames the column in the database, but other
operations are needed, like updating the corresponding `ir.model.fields`
record (and its xmlid). The default values and the translations are also
lost during the upgrade.
Moreover, this feature was misused. It was:
- left on fields during multiple versions.
- used on reports (SQL views). This would be ok if the feature was
complete, but, as is, it was useless.
- kept unchanged after a second renaming of the field (which can happen
versions later the first rename).
- used, even when the meaning of the field changed. i.e. the field
`archived` has been renamed to the classic `active`, but the value
in the database should be switched.
Create an invoice for a customer, send him an email from the chatter, in
the mail template set a specific "Reply to" and send the email. The
"Reply-To" template mail header is ignore and the one generated by
`mail_thread._notify_get_reply_to` is used.
See also 1c57d5e and 7b0806b
opw-2041640
closesodoo/odoo#35127
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
This reverts commit 7273719b88.
Reply to header of templates should be used only for mass mailing purpose.
It is a known limitation and should not be changed in table without more
in-depth testing.
closesodoo/odoo#35354
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
This commit removes the field `datas_fname` from `ir.attachment` as
it was unnecessary and most of the time the duplicate of `name` or
`url`.
Task #1909865closesodoo/odoo#32976
Signed-off-by: Martin Geubelle (mge) <mge@openerp.com>
Purpose of this commit is to improve performance of post and notify through
various code improvements.
Containing
* making of _notify_customize_recipients a mail.thread method;
* _message_auto_subscribe_followers optimized;
* renaming and optimisation of _notify_partners;
* message_post_attachements optization;
* remove unused mail_post_autofollow_partner_ids context key;
* minimize message create queries;
* only check record related access right for thread messages not in pending
moderation;
* get attachements from values instead of db;
* storing lang in ctx from the beginning;
* don't track on write on attachments;
Related to task 1943901
Linked to PR #32404
Purpose is to make field use more obvious by correctly stating in the
name it should contain an xml id to a qweb layout used for email
notifications. This renaming is propagated through various calls and
addons.
Related to task 1943901
Linked to PR #32404
Purpose of this commit is to clean notification process: calls, methods
API, method name, variable propagation.
Contains notably
* simplify API of methods used to group recipients when sending notification
emails;
* improve and rename methods used in email notification process;
* move some methods on model itself as non mail thread records could be
mass-mailed and _notify_email_headers could be called on other records;
Related to task 1943901
Linked to PR #32404
Message post should always be called on a record (ensure_one). That way we
ensure posting a message is always done in a record's context with right
values computed (reply_to, followers, ...)
Message_notify can be called on record or on mail_thread and must have
partner_ids. It is based on the recently modified user_notification mechanism
and allow to notify a partner on a record or just to push him a message
(aka, not linked to a record).
Small performance improvement
* browse recipients instead of search in _notify_email_recipients;
* todo in future optimizations: mayybe be improve by searching on ids
and is_blacklist immediately;
Related to task 1943901
Linked to PR #32404
Purpose of this commit is to clean some bits of code, notably calls to
message_post/log as well as notification methods. It will ease performance
improvement work.
Small optimization: account: read content after extension check
Parameter cleaning
* use message log with kwargs instead of args;
* remove message post after hook useless parameters;
* remove _notify_email_recipients useless message parameter;
* remove message_notify useless send_after_commit parameters;
* remove message post params matching default values;
Other improvements
* remove message_post commands support for partners and channels;
* only calls message_post with ids list for channels and partners. We
don't support mix of ids and command anymore to simplify code;
* remove support of private discussion in mail.thread adding partners
as recipients, as there is no use anymore;
Related to task 1943901
Linked to PR #32404
The stable behaviour of the mail composer is the following:
- if a template adds new attachments, the composer only has this list
- if a template doesn't add attachments, the list is unchanged
- if no template is set, the list is cleared up
In the case of templates, we also check that both static and dynamic attachments
are added to the list.
We clean up after c6d718f211 and subsequently 516f22c398 which failed to take
into account the fact that the attachement list could be a blend of commands
and ids.
Transforming that is taken care of by _convert_to_write, which semantics was
changed by c6d718f211.
To keep the second behaviour, we thus need to add a (5,) command.
opw 2003197
closesodoo/odoo#33707
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
Multi is the default api for methods, it is not necessary to explicitly
decorate methods with it, adds clutter and most people use it because
they see that the rest of the code uses it.
Done with `find . -type f -name '*.py' | xargs sed -i '/@api.multi/d'`
Some functional flows use sudo() to create records as current user is the
public user. This user has no email. Since 1e6c3bec2c using sudo does
not take superuser identity completely as it only bypasses access rights.
It means some messages are created with public user as author, meaning no
email_from.
In this commit we fix some functional flows by setting a default email_from
in the message composer only if a real email is found. Otherwise it will have
to be set either by the template, either later in the process when creating
messages or emails.
closesodoo/odoo#34742
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
This reverts commit 6230f72d2b.
Crashing when user have no valid email is the expected behavior. Flows sending
emails must ensure either current user has valid login / email, either it
uses an user OR author_id / email_from that has a meaning in the functional
flow. Notably when public user is involved. Hiding functional issues is not
what we want.
The old tree views don't really exist anymore, this odd pseudo-flag to
dispatch between "list" and "tree" tree views has no reason to remain.
Task 1937686
closesodoo/odoo#31243
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Fine-tuning of 516f22c398.
This commit missed attachments that are linked to a mail.template,
which should be added to every mail sent using the template.
To make sure that we both fix the duplicates and the generation issue,
we add a test.
opw 2000515
opw 1998782
closesodoo/odoo#33653
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
Open a mail.compose.message wizard.
Select a template that generates an attachment.
Reselect it. A second attachment has been generated.
Can be repeated ad infinitum.
At template onchange, the template is rendered to overwrite the composers values
This includes the attachments.
Let L1 be the list of attachments generated by the template.
When _convert_to_write(values) is called with {'attachment_ids': L1},
it gets self.attachments_ids = L2, and generates a [(6, 0, L1 + L2)] command.
As a result attachments are only added,
the previous ones need to be removed manually.
Since c6d718f, the semantics of _convert_to_write changed.
Before, it always returned an empty list for new records (without an id).
In other words, wa always had L2 == [].
As a result we need to adapt the wizard to keep its old behaviour.
To solve it we bypass the _convert_to_write for 'attachment_ids' and directly
generate the 6 command with the list of ids L1 generated by the template.
We only do this for templates that add attachments to keep the templates
composable (a template only overwrites the values it defines).
Unfortunately, we don't have any way to know which attachments have been added
manually, so we can't preserve these (which was the old behaviour anyway).
opw 1997430
opw 1987169
opw 1998115
opw 1998782
closesodoo/odoo#33387
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
* mail.blacklist.mixin now clearly make inheritance on mail.thread. Indeed
we consider models using the blacklist mechanism as being used in mailing
features, meaning they will anyway inherit form mail.thread. In standard
Odoo it is the case for the 3 models using it (Lead/Opportunity, Contact
and Mailing Contact);
* define message_bounce directly in mail.blacklist.mixin. Bounce counter
is indeed linked to the blacklist and mailing mechanism;
* rename mail.blacklist.mixin to mail.thread.blacklist to ensure coherency
with mail.thread and mail.thread.cc (another mixin build on mailL.thread);
Inherit declarations in various addons are updated accordingly.
Related to task ID 1911679
Linked to PR #29483
Purpose: make some methods from mail.thread mixin private and update their
naming if necessary. Some methods prepare data for business flows and deal
with internal information. Public methods should either use of give some
access to those methods. Computation itself should be keps internal.
In this commit we put message_get_default_recipients as private. Its API
is improved to clearly separate model-inherited method (overriden in some
addons) and general tool method given by the mixin. Finally all sudo hidden
in the computation are removed. Caller now give a sudo-ed record set in order
to be more explicit. Low level method just computes the result and does not
care about sudo-ing or ACLs.
Note a model approach for this method is necessary as mass mailing using
mail.template records does not ensure records to check inherit from mail
thread.
Some addons are updated accordingly to the method change.
Related to task ID 1911679
Linked to PR #29483
Allow to specify the `model_description` when sending an email to take
into account for cases where the same model is used for different
business concepts.
opw-1915230
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
Martin Trigaux