TLDR:
* invoices are implemented using computed methods instead of onchange
* the synchronization only happens when switching tabs in the Form view
to improve perfs.
_______________________________________________________________________
The whole engine of the synchronization of Invoices to the Journal
Entries has been refactored
* by using computed fields instead of onchange functions
* by synchronizing only from invoice to journal entry in `create` and
`write`
* by saving when switching tabs on the Invoice form, to synchronize
before showing the values
This comes with numerous advantages:
* no need to call the onchange methods manually
* no need to use the Form emulator to build invoices (i.e. EDI, OCR,
intercompany, ...)
* the performance for invoices with many lines improves drastically, going
from 2 minutes to 4 seconds to create an invoice with 500 lines
* the model is more declarative, we can now see how the values are computed
instead of having the values being copied from various places.
* remove the hack in `onchange` that disabled the recursivity of it,
which was unexpected and needed to be managed manually in all the
onchange methods
This means that:
* Some fields need to be exclusively computed on journal entries values
or invoice values, more specifically the Tax Summary widget.
It is now
- computed from entry lines, when opening the view
- computed from invoice lines when changing those, because the tax lines
will need to be recomputed anyways, erasing previously set values
- set with an inverse function when saving; after the sync has been done
* Some possible operations previously possible have been dropped.
(i.e. look at the removed test test_in_invoice_line_onchange_accounting_fields_1)
This is because such a behavior was undefined (how is changing the balance going
to affect the unit price? How is the amount currency going to affect it?)
_______________________________________________________________________
Implementation Details
----------------------
The "dynamic lines", meaning the payment terms and the tax lines are now
only created in the `create` and `write` functions.
In order to reduce code duplication, it has been implemented using
context managers used in both `account.move` and `account.move.line`
These context managers help comparing the values before/after, acting
like a local `onchange`, but getting benefit from the dirty flags from
the `compute` dependences.
This is relying on computed fields on the move (`needed_terms`) and on
the lines (`compute_all_tax`) which contain the values needed for the
related move.
Depending on the needed values and the existing values (`term_key` and
`tax_key`, respectively) the context manager will determine what needs
to be created/updated/deleted.
Some related changes are to produce a `dict` instead of a `str` for the
`tax_totals` (previously `tax_totals_json`) fields, by simplicity to
reduce the complexity of IO, and simplicity of debugging, because the
logic of the field needed to change (cannot be computed at the same time
anymore since it needed the lines to be synced)
By simplicity, and also because it makes more sense, some boolean fields
have been merged into `display_type`:
* `is_rounding_line`
* `exclude_from_invoice_tab`
* `is_anglo_saxon_line`
The `price_unit`, `quantity` and other "invoice fields" are now not set
anymore on lines that are not product lines since it didn't make any
sense to have it.
Performances
------------
You have to keep in mind that a simple `create` didn't compute a lot of
fields, for instance not taxes were set, no payment terms,...
Now it does.
```python
import random
from timeit import timeit
from odoo import Command
domain = [('company_id', 'in', (False, self.env.company.id))]
products = self.env['product.product'].search(domain).ids
partners = self.env['res.partner'].search(domain).ids
taxes = self.env['account.tax'].search(domain).ids
def create(nmove, nline):
self.env['account.move'].create([
{
'move_type': 'out_invoice',
'partner_id': random.choice(partners),
'invoice_line_ids': [
Command.create({
'name': f'line{i}',
'product_id': random.choice(products),
'tax_ids': [Command.set([random.choice(taxes)])],
})
for i in range(nline)
]
}
for j in range(nmove)
])
# After | Before
print(timeit("create(1, 1)", globals=globals(), number=1)) # 0.11 | 0.09
print(timeit("create(100, 1)", globals=globals(), number=1)) # 2.76 | 2.50
print(timeit("create(500, 1)", globals=globals(), number=1)) # 14.56 | 12.34
print(timeit("create(1, 100)", globals=globals(), number=1)) # 1.03 | 5.52
print(timeit("create(1, 500)", globals=globals(), number=1)) # 3.99 | 125.02
print(timeit("create(50, 50)", globals=globals(), number=1)) # 19.44 | 79.55
```
Another metric that can be used is running the test suite with
`--test-tags=/account` (only `account` installed)
* before: 404s, 267127 queries (366 tests)
* after: 318s, 232125 queries (362 tests)
Why this commit title?
----------------------
Someone told me that this was the perfect way of naming your commits.
c04065abd8
task-2711317
closesodoo/odoo#96134
Related: odoo/upgrade#3715
Related: odoo/enterprise#29758
Signed-off-by: Laurent Smet <las@odoo.com>
Update to latest runbot state, at least for tests known to be
deterministic.
Notably mail tests are lower than before, probably due to odoo/odoo#73271closesodoo/odoo#93470
X-original-commit: 6118ecba8d807ac5ebc69ae4877e1f202b0daeb1
Related: odoo/enterprise#28308
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
It looks like in enterprise, the value of has_group is already in cache
This will lead to an additional query when checking mail.message
creation access rights in community.
This commit warmup the user _has_group ormcache to avoid an
inconsistency between community and enterprise
Stack of the additional query:
```
sql('SELECT 1 FROM res_groups_users_rel WHERE uid=%s AND gid IN\n ... (SELECT res_id FROM ir_model_data WHERE module=%s AND name=%s)') (SELECT 1 FROM res_groups_users_rel WHERE uid=29 AND gid IN (SELECT res_id FROM ir_model_data WHERE module='base' AND name='group_user'))
execute (odoo/addons/base/models/res_users.py (self._cr.execute("""SELECT 1 FROM res_groups_users_rel WHERE uid=%s AND gid IN):847)
_has_group (odoo/tools/cache.py (value = d[key] = self.method(*args, **kwargs)):90)
lookup (called at <decorator-gen-111> ():2)
_has_group (odoo/addons/base/models/res_users.py (return self.with_user(uid)._has_group(group_ext_id)):832)
has_group (addons/mail/models/mail_message.py (if not self.env['res.users'].has_group('base.group_user'):):380)
check_access_rule (odoo/models.py (records.check_access_rule('create')):4321)
_create (odoo/models.py (records = self._create(data_list)):4087)
create (odoo/api.py (return create(self, arg)):410)
_model_create_multi (called at <decorator-gen-16> ():2)
create (odoo/addons/base/models/ir_fields.py (recs = super().create(vals_list)):613)
create (odoo/api.py (return create(self, arg)):410)
_model_create_multi (called at <decorator-gen-68> ():2)
create (addons/mail/models/mail_message.py (messages = super(Message, self).create(values_list)):599)
create (odoo/api.py (return create(self, arg)):410)
_model_create_multi (called at <decorator-gen-193> ():2)
create (addons/mail/models/mail_thread.py (return self.env['mail.message'].create(create_values_list)):2132)
_message_create (addons/mail/models/mail_thread.py (new_message = self._message_create(msg_values)):1874)
message_post (addons/rating/models/mail_thread.py (message = super(MailThread, self).message_post(**kwargs)):14)
message_post (addons/mail/wizard/mail_compose_message.py (ActiveModel.browse(res_id).message_post(**post_params)):321)
_action_send_mail (addons/mail/models/mail_thread.py (return composer._action_send_mail(auto_commit=auto_commit)):1964)
message_post_with_template (addons/mail/models/mail_thread.py (return record.message_post_with_template(False, **kwargs)):1929)
_message_compose_with_view (addons/mail/models/mail_thread.py (self._message_compose_with_view(views_or_xmlid, **kwargs)):1933)
message_post_with_view (addons/crm/models/crm_lead.py (opportunities_head.message_post_with_view():1336)
_merge_opportunity (addons/crm/models/crm_team.py (merged = lead_duplicates._merge_opportunity(user_id=False, team_id=False, auto_unlink=False, max_length=0)):578)
_allocate_leads_deduplicate (addons/crm/models/crm_team.py (assign_res = team._allocate_leads_deduplicate(candidate_lead, duplicates_cache=duplicates_lead_cache)):507)
_allocate_leads (addons/crm/models/crm_team.py (teams_data = self._allocate_leads(work_days=work_days)):304)
_action_assign_leads
test_assign_perf_duplicates (odoo/tools/misc.py (return func(*args, **kwargs)):792)
```
task-2796579
Part-of: odoo/odoo#85525
There were inconsistencies in the calls to `_render`.
* the view context could contain information that misled developers.
Indeed, the context and value of the view are not supposed to be found
in the rendering. Thus by calling `ir.qweb` with the name of the
template, we ensure that there is no unwanted information and in
addition the cache key is that of the name of the template which saves
a query.
* the context used for rendering was modified by a method on
`ir.ui.view`, except this is not information used by this model. There
is now a `_prepare_environment` method residing on `ir.qweb`. This
method allows to modify the value dictionary as well as the context in
which the rendering will be done. This preparation of the data as well
as my security check is done only once per rendering. This also saves
some queries
* Freeze options for rendering were inconsistent. It could be that
options on which rendering depends were not part of the cache key. Thus,
depending on the user who generated the generation of the rendering
function, there was or was not information in the template. For example
for automatic branding. This is no longer possible, because it is the
context that is used. The options serving as a cache key are only
recorded for information (for the profiling system for example). A
simplification of the `ir.qweb.field` models could be made.
The report rendering and call `ir.qweb` instead of `ir.ui.view`.
Part-of: odoo/odoo#85110
Currently when converting leads we may end up with False being compared to
a void partner recordset. Due to the use of != this leads to unnecessary
update of leads when no partner is involved in lead convert.
By comparing recordsets everytime we save queries and performance each
time a convert on a lead without customer is done. This leads to about
saving 150 queries in heavy duty tests.
Task-2722512 (Lead: performance in convert without customer)
Task-2722513 (Lead: performance master task)
closesodoo/odoo#81028
Related: odoo/enterprise#23094
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose
=======
When creating an opportunity, set the language of the Lead/Opportunity
to the partner's language if it is set instead of leaving it blank.
Also update tests to correctly test lang propagation.
Update event_crm so that lang of lead from registration is directly set
to False when there is no partner. Indeed we have no clue which lang we
should set and we can skip the field computation that otherwise triggers
some additional queries.
Task-2709436
Part-of: odoo/odoo#81028
In one week, this test failed about 4 times, seems mainly on nightly
enterprise. Let us update counter accordingly.
closesodoo/odoo#80164
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Seed is now fixed at beginning of each test to ensure random state is set
once for all and avoid random issues.
Task-2643740
X-original-commit: 85ebcaeaf909cd46743c20bf052360a65e60ea82
Part-of: odoo/odoo#79573
Purpose is to avoid randomly failing tess depending on how leads are allocated
through teams. We achieve that purpose by
* creating more leads, easing their repartition through teams;
* adding a lower and upper bound test for team allocation to ensure weights
are globally ok;
* changing member assign domain to fit more closely test lead data;
We also improve test by using the ``crm.assignment.commit.bundle`` config
parameter that has an impact on lead duplicates removing and therefore
performances.
LINKS
Task ID-2528906
COM PR odoo/odoo#70756
Followup of Task ID-2489951 (assign process improvements)
Followup of odoo/odoo#70172 (smoother assign process)
X-original-commit: 8fdf97b6498adc0a2fb15e360ebbbaa1672534e5
PURPOSE
All unassigned leads should be assigned to teams as soon as possible to ease
lead analysis.
Purpose of assign thresholds is to ensure sales people receive at least this
amount of leads within 30 days, counting lost and won leads. Giving them leads
regularly is also one goal of automatic assign.
SPECIFICATIONS
Counting every lead whatever its state may lead to an inconvenient situation.
If a salesman always reaches its maximum every days he will always receive the
number of lead he got 30 days ago. For example if the salesman goes in holidays
for few days and set the max to 0 he receives no lead during his vacation.
Then after a few days of assign he reaches its maximum and receive 0 leads
for a few days. This leads to having windows of leads that repeat themselves
every 30 days.
* Solution: do not limit at maximum capacity anymore. Compensation is voided
if limit is achieved. However asked assignment is done. Salesman could
receive more than its max capacity but 30 days window ensure old leads
are regularly going out of count.
``assignment_max`` is now more a mean target of leads to be assigned during
a 30 days window than a real maximum capacity. Field is renamed accordingly.
LINKS
Task ID-2444908 (assign fixes)
Task ID-2489951 (assign process improvements)
COM PR odoo/odoo#70172
X-original-commit: 3d1841f26ce64f0106122e48750d2c653ea9790f
Co-authored-by: Thibault François <tfr@odoo.com>
Co-authored-by: Thibault Delavallée <tde@odoo.com>
PURPOSE
All unassigned leads should be assigned to teams as soon as possible to ease
lead analysis.
Purpose of assign thresholds is to ensure sales people receive at least this
amount of leads within 30 days, counting lost and won leads. Giving them leads
regularly is also one goal of automatic assign.
SPECIFICATIONS: TEAM ALLOCATION
Team assignment has to be updated as we may have team domains that overlap.
We therefore remove maximum number of leads to allocate to teams. Instead all
available unassigned leads are allocated within teams.
- Solution: assign all available leads and not a count based on team's
capacity. This notably reverts the main goal of odoo/odoo@6df2f0cfc0
(see odoo/odoo#48422)
This assignment process is done proportionally to the team capacity. It is
computed as the sum of each member's maximum assignment counter. This means
that with a team having twice as much sale capacity than another team sharing
the same domain: first team should receive about 2/3 of leads while the second
one should receive the remaining 1/3.
- Solution: assign lead one by one. Choose a team randomly using a weighted
random algorithm, based on team's members capacity.
SPECIFICATIONS: MEMBER ASSIGN
Counting every lead whatever its state may lead to an inconvenient situation.
Moreover salespersons may opt-out from assign by setting their max capacity
to 0, for example when going on holidays.
When doing that lead assignment is not smooth and getting back to a full
pipe may take several days. To solve that issue a compensation is added in
assignment quota done to sales people. Salespersons having few leads will
get a boost in assign as soon as they get back in assign process. When being
near maximum compensation is nearing 0 and daily quota is given.
PERFORMANCE
During team assignation, assigning lead one by one may cause performance issue.
Since PLS is computed at each flush, so we want to flush after a bunch of lead.
To solve that we obviously need to not commit at each assignation but
also avoid to search for duplicate at each assignation, that why the
search for duplicates is done at the beginning of the process and stored
in memory.
LINKS
Task ID-2444908 (assign fixes)
Task ID-2489951 (assign process improvements)
COM PR odoo/odoo#70172
X-original-commit: 83f72711e7affa8b0ad997b9ee262326e9577d54
New leads are automatically assigned to sales team and salesman during assign
process. Some of those leads are duplicates of existing opportunities. If a
new lead is merged into an existing opportunity, the opportunity is the master
data and the lead is duplicated. However the opportunity should keep its sales
team and not be assigned to the team being under automatic assign.
Before this commit, the merge was forcing the crm team to the 'current' team
during auto assignment and thus it was changing the team of the opportunity.
In this commit we now assign team to lead to merge before the merge and do
not force the crm team during merge process.
LINKS
Task ID-2444908 (assign fixes)
Task ID-2489951 (assign process improvements)
COM PR odoo/odoo#70172
X-original-commit: 356a55053eb1cc49ae6eed0e58338cff08eddc42
This commit contains two main preparatory changes in tests
* 1) lead_month_count will be changed to count lost lead as well. So
archiving leads before running the test doesn't reset the counter.
Lead needs to be unlinked in order to make tests reproducible;
* 2) assertMemberAssign checked the validity of the team domain. Normally
all lead assigned to the member should match the domain of its team.
This assert is not always verified since we force the team after a
merge of duplicated leads. Final merged lead may not match team domain
anymore as final properties depend on all merged leads.
Some tests are also added :
* add tests for master lead team_id and user_id values when merging leads
in assign process;
* add tests using email_normalized check in duplicate finding to ensure that
normalized emails are used for duplicates and not only exact match;
* add tests for won / lost inclusion in ``_get_lead_duplicates``, as well
as different between probability=100 and stage.is_won = True . This is
going to change in upcoming commits, hence some test to highlight it;
* add tests counting existing leads for assignment purpose;
* add tests for assignment_max maximum leads allocation as well as max=0
meaning salesperson is opt-outed from assign;
* add tests for default user / team / stage computation for leads;
* add tests about leads taken into account in assign process (currently
failing with won unassigned leads);
We also add some flush to ensure some fields notably related to probabilities
are flushed before counting queries.
LINKS
Task ID-2444908 (assign fixes)
Task ID-2489951 (assign process improvements)
COM PR odoo/odoo#70172
X-original-commit: bd3444085974f4e12287c2f5da4fb33308cd6180
Co-authored-by: Thibault François <tfr@odoo.com>
Co-authored-by: Thibault Delavallée <tde@odoo.com>
Base: seems related fields take only 1 extra query instead of 3
Crm: assignment seems to have been slightly improved. Some randomness still
happens.
Hr Holidays: seems it was further improved beyond space frontier
Hr Work Entry Holidays: seems it was further improved
Test mail: those tests were a bit random, seems random is gone (hopefully)
Test mail full: updated local counters
Test mass mailing: seems we gained one query, updated local counters
closesodoo/odoo#70504
X-original-commit: 8c215b3d60929166b3e60e2a4dfb2a6c70439046
Related: odoo/enterprise#18194
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Those are not launched anymore by default. Indeed framework and code followup
is not ready for that level of details.
Task ID-2480727
closesodoo/odoo#67678
X-original-commit: f950399e6e4aaaf88b743abfeb37a2de4c99fa5c
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
According to last runbot state. Should probably be removed someday as actually
there is not much gain with those tests.
X-original-commit: dc05f873571ca60b7fbbb701a86760f13a6d3172
Probably linked to merge and forward of odoo/odoo#65688 that split some
computed fields. This may have an impact on some fields computation and
therefore queries.
Task ID-2455236
closesodoo/odoo#65748
X-original-commit: b1520ad39c7abd18f8af1c3b23e6e3e897916004
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Some tests sometimes require some more queries to be finished.
Task ID-2444908
closesodoo/odoo#65379
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Purpose is to split assign tests related to results (leads being assigned,
with deduplication, compared to team and member domains) from tests related
to performance (query counters).
Assign process is a random process: randomizing teams leads to searching,
assigning and de-duplicating leads in various order. As a lot of search
are implied during assign process query counters may vary from run to run.
"Heavy" performance test included here ranged from 6K to 6.3K queries. Either
we set high counters maximum which makes those tests less useful. Either we
avoid random if possible which is what we decided to do by setting the seed
of random in tests.
SPECIFICATIONS
For results oriented tests remove query counters. Those tests should simply
ensure results of assign process, notably that even with random choice between
team and users we always have leads assigned with right domain and counts.
For performance oriented tests using query counters set random seed to have
a reproducible scenario and have less random in query counters.
LINKS
Task ID-2446759 (lead merge priority field management)
Task ID-2446883 (query counters fix)
PR odoo/odoo#65016
william-andre