The `t-cache` directive allows you to keep the rendered result
of a template part. The supplied key must be a tuple. This tuple
can contain recordset in this case the zone will be invalidated
each time the write_date of these records changes.
The `t-nocache` directive makes it possible to force rendering
of a part even if it is in a `t-cache`. The values available in
the `t-nocache` are the one provided when calling the template
(and therefore ignores any t-set that could have been done).
Part-of: odoo/odoo#88276
There were inconsistencies in the calls to `_render`.
* the view context could contain information that misled developers.
Indeed, the context and value of the view are not supposed to be found
in the rendering. Thus by calling `ir.qweb` with the name of the
template, we ensure that there is no unwanted information and in
addition the cache key is that of the name of the template which saves
a query.
* the context used for rendering was modified by a method on
`ir.ui.view`, except this is not information used by this model. There
is now a `_prepare_environment` method residing on `ir.qweb`. This
method allows to modify the value dictionary as well as the context in
which the rendering will be done. This preparation of the data as well
as my security check is done only once per rendering. This also saves
some queries
* Freeze options for rendering were inconsistent. It could be that
options on which rendering depends were not part of the cache key. Thus,
depending on the user who generated the generation of the rendering
function, there was or was not information in the template. For example
for automatic branding. This is no longer possible, because it is the
context that is used. The options serving as a cache key are only
recorded for information (for the profiling system for example). A
simplification of the `ir.qweb.field` models could be made.
The report rendering and call `ir.qweb` instead of `ir.ui.view`.
Part-of: odoo/odoo#85110
First step to stream QWeb templates: removing the two post
processing operations applied on rendered templates. This
should slightly speed up the rendering of every page.
1/ Don't remove empty lines after rendering, but fix the root
cause of: view inheritancies and QWeb compilation that don't
add extra empty lines.
2/ handle page break in the two reports that uses it, rather
than processing every view produced.
closesodoo/odoo#82244
Related: odoo/enterprise#23328
Signed-off-by: Fabien Pinckaers <fp@odoo.com>
By activating the profiler debugger, the two qweb options is added. The
qweb templates that need to be rendered are compiled into a new function
to add instructions for saving data.
`Add qweb directive context`
It's a sub-option of "Record sql" or "Record traces", add some context on
thread at current call stack level. This context stored by collector
beside stack and is used by Speedscope to add a level to the stack with
this qweb directive information.
```
directive=t-call='website.layout', xpath=/t/t
t_call_content
directive=t-foreach='5' t-as="'a', xpath=/t/t/div/div
directive=t-esc='website.search([])', xpath=/t/t/div/div/t
execute
```
`Record qweb`
Add profiling data used by ProfilingQwebView widget. In the `ir.profile`
form view, the widget display the duration and number of sql of every qweb
directives with the xml of templates. Every xml is recorded to be
consulted even if the user change the xml templates.
```xml
<t t-call="website.layout">
<div>
<div t-foreach="5" t-as="a">
<t t-esc="website.search([])"/> <!-- will display 5 separate requests -->
</div>
</div>
</t>
```
closesodoo/odoo#74712
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
Since de077243 the iter(False) will occur when trying to render a
speedscope output with `use_context=False` since the context given in
parameter of `stack_to_ids` won't be an iterable.
This simple fix avoid this by always giving an empty iterable
in this case.
closesodoo/odoo#75851
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The initial behaviour of ExecutionContext was to
- add the context for the current level of the stack on enter
- remove the context at this same level on exit.
When two ExecutionContext are nested at the same stack level:
- the first one adds it's context;
- the second one overrides the current context;
- the second one removes it at the end;
- the first one fails to remove the context at the same level a second time.
def call():
with ExecutionContext(foo=1):
with ExecutionContext(bar=2):
execute()
In this case we could imagine to combine both Execution managers in one:
def call():
with ExecutionContext(foo=1, bar=2)
execute()
But the semantics is different: ExecutionContext should add one level to
the stack, instead of two. In the first example above, we expect two
additionnal levels:
call
foo=1
bar=2
execute
A simple solution would be to transform the value of the context at some
level to a list of dict, but this would make the copy more difficult.
The decision was taken to change the context storage strategy, going
from a dict where the keys are the level to a tuple of tuples where the
first element is the level.
{3: {foo: 1}, 4: {bar: 2}} => ((3, {foo: 1}), (4, {bar: 2}))
The first benefit is to allow multiple contexts at the same level. The
second one is that we don't need to copy the data structure, since
tuples are immutables and the dicts are coming from kwargs. This means
that saving the context is faster, and the tuple is shared between
multiple samples. This is interesting if we assume that we do more
samples than exec_context mutations.
closesodoo/odoo#75687
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
This commit adds tooling to profile performance and save execution by
saving stack traces and queries to a file/database in specific format.
----------
Collectors
----------
For now, three different profiling modes (aka Collectors) are available
even if a last once should be introduced by @Gorash to profile qweb
execution.
- SQLCollector (or 'sql'): Saves the current stack trace and the query
every time Cursor.execute() is called. Any query executed on the thread
will be collected, no matter the cursor.
- PeriodicCollector (or 'traces_async'): Saves the stack trace every
'interval' seconds using a parallel thread to profile the caller thread.
The python implementation was optimized to minimize impact on
performance while remaining portable and easy to enable/disable
inside a odoo execution. Higher the frequency (lower the interval),
more impactful the profiling will become on the execution and increase
memory usage. From last experiments, 1ms looks to be a good minimum for
short executions.
- SyncCollector (or 'traces_sync'): Saves the stack trace every function
call/return. This collector is obviously quite impactful on performance
and can quickly overload the memory for long executions, but this is
quite useful to understand the precise path followed by some short
executions. Any time related information will be almost irrelevant with this
collector.
A base Collector defining minimal collectors features can easily be
extended to create custom collectors if needed.
----------------
Profiler & Usage
----------------
Collectors are not supposed to be used by themselves, but should be
given to a Profiler. The Profiler will synchronize collectors starts and
stop, and manage saving them to a file of in a ir_profile in the
database.
Exemple of usage:
```
with Profiler():
do_stuff()
```
This simple example will use the default collectors (sql and
traces_async) and save them to the database. The database is defined
automatically from current_thread 'dbname' if available.
Example of usage:
```
with Profiler(collectors=['sql'], db=False, path=/home/user/logs/do_stuff_profile/{time}):
do_stuff()
```
This more complex example disable the default behavior consisting
to save to the database, gives a path where the profile will be saved
and specify to only use the 'sql' collector. Note that
collectors=[SQLCollector()] would have the same behavior since
Collectors can be either a Collector instance or a string describing the
desired collector. This allows to define custom params for the
collectors and use custom collectors if needed.
Note that it is always possible to get results after execution without
saving it since they are available on the profiler.
```
with Profiler(collectors=['sql'], db=False) as p:
do_stuff()
print(len([None for entry in p.collectors[0].entries if ...]))
```
Profiler will also save the stack below the profiler start point, and
collectors will only collect the part of the stack over this stack.
This is a good way to reduce collectors CPU and memory usage.
Collected entries will be saved as follows:
```
[{
'start': 2.0,
'context': {},
'stack': [
['path_to_file', lno, 'func_name', 'line_content'],
...
],
},
...
]
```
SQLCollector will add three additional keys on each entry:
- query (query without parameters)
- full_query (mogrified query with parameters)
- time (the 'exact' execution time of the query)
----------------
ExecutionContext
----------------
A last tool, ExecutionContext, allows to define some context on some block of code:
Example of usage:
```
def process_modules(modules)
for module in modules:
with ExecutionContext(module=module): # note the 'not linter frienldy but still convenient' 2 spaces indentation
do_stuff(module):
```
This context will automatically be added in the stack as a virtual frame between
process_modules and do_stuff in order to split do_stuff from one single frame to
one frame per module.
----------
Speedscope
----------
The saved data are in a simple json format easy to analyze, but can't be visualized in
speedscope as they are. A utility class `Speedscope` can be used to generate a format
readable by speedscope. The used format is actually the format defined by speedscope,
meaning that all features should be available using it.
The output format is evented, meaning that we need to transform a list of samples
(a list of stack) to a list of event (going in/out a frame).
This is the main task of the Speedscope, as well as combining samples from different
sources, to display SQLCollector and PeriodicCollector results mixed together.
When stored on an ir_profile, the default speedscope generation can easily be generated
with the speedscope computed field.
This class can be used as it is but will mainly be useful for the next commit.
Special thanks to @rco-odoo for the in depth review and @Gorash for support.
Gorash