Yannick Tivisse
f0d5c73c57
Purpose ======= The work entries generation is not scaling properly for large employee datasets. A lot of improvements have been made to reduce the number of query to the database. However, one of the remaining bottleneck was the new records insertion, made 1 by 1 on the database. Due to the fact that the model doens't have a lot of columns, and doens't have columns containing too much data (HTML fields for example), we could imagine inserting the records by batch, for example 1000 by 1000, as it is done with SELECT. The results weren't satisfying enough, so we prefered using the new cron triggers mechanism, to re-trigger the same cron at the end of its execution in another transaction, to avoid exceeding the non return point from which posgreSQL is pedaling in the semoule. With this commit, it takes less than 30 seconds to generate the work entries for 1000 employees, instead of 3 min, and allows to generate the work entries in a linear execution time instead of a exponential one. Create method analysis: ======================= Note, that only the call to the create method is tracked, not the preprocessing time to retrieve the work entries values. When inserting the records 1 by 1: ---------------------------------- records - Elapsed Time (s) - AVG Time per record (s) 10 - 0.0045862197875976 - 0.00045862197875976 432 - 0.1101946830749511 - 0.00025508028489572 192 - 0.0525383949279785 - 0.00027363747358322 522 - 0.1418645381927490 - 0.00027177114596312 892 - 0.2803149223327636 - 0.00031425439723404 8800 - 3.4928441047668457 - 0.00039691410281441 88000 - 188.82338452339172 - 0.00214572027867490 176000 - 1172.4313135147095 - 0.00666154155406084 We observe that from 10.000 new record, the create method is not scaling anymore before this commit. For a company with 1000 employees, the mean work entries by month is 1000*2*21=42.000 work entries, and the time to create the records is not acceptable. When inserting the records 1000 by 1000 --------------------------------------- records - Elapsed Time (s) - AVG Time per record (s) 10 - 0.003406763076782 - 0.0003406763076782 432 - 0.075028181076049 - 0.0001736763450834 192 - 0.030718326568603 - 0.0001599912842114 522 - 0.084228038787841 - 0.0001613563961452 892 - 0.145947217941284 - 0.0001636179573332 8800 - 2.204506397247314 - 0.0002505120905962 88000 - 181.9736533164978 - 0.0020678824240511 176000 - 1146.928646564483 - 0.0065166400372981 We observe a sligh improvement, but cleary not enough and not worth the complexity of inserting the records in batch on the database. Note: The following explanation is speculative and could be validated with a real analysis, but according to the results we obtained with the cron, this is most likely to be true. In fact, this is a limitation of PosgreSQL. In the implementation that manages the transactions, the number of inserts becomes greater than certain memory limits, suddenly it falls on a slower alternative storage (this is the problem with SELECT and tuples of ids ). When we inject 100,000 ids into a query, PosgreSQL parses the query (already there it could have some trouble), then it stores the ids in a data structure: hash if not too large, other (on the file system) otherwise. Then it executes the query and uses the data structure to check the validity (membership). Specification ============= Instead of inserting the record in batch in the same transaction, it would be more interesting to split the transaction into several transactions: Each transaction process N employees who have not yet been processed, until there are no more. With the new cron trigger mechanism, it is possible to: - When the cron runs, it processes N employees (N to be determined) - If there is still some, he retriggers itself at the end of his transaction Like that, the cron is scheduled 1x / day, but it is retriggered as many times as necessary each month. Regarding the number of employees to process, with 100 employees, we can expect: 100 * 2 (morning / evening) * 21 (working days) = 4200 work entries to generate, which is manageable given the above measures. closes odoo/odoo#76488 Taskid: 2646056 Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
Odoo
Odoo is a suite of web based open source business apps.
The main Odoo Apps include an Open Source CRM, Website Builder, eCommerce, Warehouse Management, Project Management, Billing & Accounting, Point of Sale, Human Resources, Marketing, Manufacturing, ...
Odoo Apps can be used as stand-alone applications, but they also integrate seamlessly so you get a full-featured Open Source ERP when you install several Apps.
Getting started with Odoo
For a standard installation please follow the Setup instructions from the documentation.
To learn the software, we recommend the Odoo eLearning, or Scale-up, the business game. Developers can start with the developer tutorials