Fixes a traceback when generating work entries.
How to reproduce:
- Create a contract with a working schedule with 0 attendances
- Set the contract as running
- Make sure you have another contract for another employee running on
the same period, this one must have attendances
- Go to the work entries views, this should generate working entries for
the viewable dates
- If the work entries were already generated, go to a month where they
are not generated yet, you should have the traceback
Task ID: 2654797
closesodoo/odoo#77085
X-original-commit: 13f7f8c4e066b0a2fadef95cdff4a24a4f67f95c
Signed-off-by: Kevin Baptiste <kba@odoo.com>
Signed-off-by: William Braeckman (wbr) <Williambraecky@users.noreply.github.com>
Purpose
=======
The work entries generation is not scaling properly
for large employee datasets. A lot of improvements
have been made to reduce the number of query to the
database. However, one of the remaining bottleneck
was the new records insertion, made 1 by 1 on the
database.
Due to the fact that the model doens't have a lot
of columns, and doens't have columns containing
too much data (HTML fields for example), we could
imagine inserting the records by batch, for example
1000 by 1000, as it is done with SELECT.
The results weren't satisfying enough, so we prefered
using the new cron triggers mechanism, to re-trigger
the same cron at the end of its execution in another
transaction, to avoid exceeding the non return point
from which posgreSQL is pedaling in the semoule.
With this commit, it takes less than 30 seconds to
generate the work entries for 1000 employees, instead
of 3 min, and allows to generate the work entries in
a linear execution time instead of a exponential one.
Create method analysis:
=======================
Note, that only the call to the create method is
tracked, not the preprocessing time to retrieve
the work entries values.
When inserting the records 1 by 1:
----------------------------------
records - Elapsed Time (s) - AVG Time per record (s)
10 - 0.0045862197875976 - 0.00045862197875976
432 - 0.1101946830749511 - 0.00025508028489572
192 - 0.0525383949279785 - 0.00027363747358322
522 - 0.1418645381927490 - 0.00027177114596312
892 - 0.2803149223327636 - 0.00031425439723404
8800 - 3.4928441047668457 - 0.00039691410281441
88000 - 188.82338452339172 - 0.00214572027867490
176000 - 1172.4313135147095 - 0.00666154155406084
We observe that from 10.000 new record, the create
method is not scaling anymore before this commit.
For a company with 1000 employees, the mean work
entries by month is 1000*2*21=42.000 work entries,
and the time to create the records is not acceptable.
When inserting the records 1000 by 1000
---------------------------------------
records - Elapsed Time (s) - AVG Time per record (s)
10 - 0.003406763076782 - 0.0003406763076782
432 - 0.075028181076049 - 0.0001736763450834
192 - 0.030718326568603 - 0.0001599912842114
522 - 0.084228038787841 - 0.0001613563961452
892 - 0.145947217941284 - 0.0001636179573332
8800 - 2.204506397247314 - 0.0002505120905962
88000 - 181.9736533164978 - 0.0020678824240511
176000 - 1146.928646564483 - 0.0065166400372981
We observe a sligh improvement, but cleary not enough
and not worth the complexity of inserting the records
in batch on the database.
Note: The following explanation is speculative and could
be validated with a real analysis, but according to the
results we obtained with the cron, this is most likely
to be true.
In fact, this is a limitation of PosgreSQL. In the
implementation that manages the transactions, the number
of inserts becomes greater than certain memory limits,
suddenly it falls on a slower alternative storage (this
is the problem with SELECT and tuples of ids ).
When we inject 100,000 ids into a query, PosgreSQL parses
the query (already there it could have some trouble), then
it stores the ids in a data structure: hash if not too
large, other (on the file system) otherwise. Then it
executes the query and uses the data structure to check
the validity (membership).
Specification
=============
Instead of inserting the record in batch in the same
transaction, it would be more interesting to split the
transaction into several transactions:
Each transaction process N employees who have not yet been
processed, until there are no more. With the new cron
trigger mechanism, it is possible to:
- When the cron runs, it processes N employees
(N to be determined)
- If there is still some, he retriggers itself at the end
of his transaction
Like that, the cron is scheduled 1x / day, but it is
retriggered as many times as necessary each month.
Regarding the number of employees to process, with 100
employees, we can expect:
100 * 2 (morning / evening) * 21 (working days) = 4200
work entries to generate, which is manageable given
the above measures.
closesodoo/odoo#76488
Taskid: 2646056
Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
- hr_contract: Add kanban view on contract history action
- hr_contract: Display the avg wage, not the sum on aggregates
- hr_contract: Improve contract history list view
- hr_contract: Improve contract history tree view
- hr_contract: Improve contract history search view
- hr_contract: Display wage on contract kanban view
- hr_contract: Improve contract tree view
- hr_contract: Improve contract search view
- hr_contract: Make hr_responsible required
It is useful in multiple HR flows.
- hr_contract: Improve contract history form view
- hr_contract: Improve contract form view
- hr_work_entry: Improve work entry computed name
- hr_work_entry: Improve work entries tree view
- hr_work_entry: Improve work entry type form view
closesodoo/odoo#72402
Taskid: 2558171
Related: odoo/enterprise#19124
Related: odoo/upgrade#2583
Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
Before this commit, the hr_work_entry_contract contained enterprise views/features that could not be integrated in community.
These features are now provided by the hr_work_entry_contract_enterprise module.
taskid: 2222790
Before this commit, the module dependencies: hr_contract and hr_work_entry were defined in community.
As they could be integrated with some community applications, like hr_attendance or hr_holidays, it is more coherent to move them to community.
Taskid: 2222790
William Braeckman