Unwittingly broken by the removal of `__implements__` in
a6d601dc4e, these lints don't run
correctly with the old pylint, allowing new errors to creep in since.
closesodoo/odoo#139605
X-original-commit: 99cec73f585c8759142a707b06934d9c23aa5478
Related: odoo/enterprise#49473
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
We introduce a new class of objects to wrap SQL code together with its
parameters. It is designed to be easily composable and to discourage
SQL injections. Its API is similar to the methods of module 'logging':
the code is a format string, and the positional parameters are meant to
be merged into it using the string formatting operator.
# default and increment are parameters of the SQL code in first argument
term = SQL("COALESCE(value, %s) + %s", default, increment)
# term can safely be injected into another SQL, besides regular parameters
query = SQL("SELECT %s FROM mytable WHERE id = %s", term, id_)
The SQL wrapper can return the final SQL code string as query.code, and
the corresponding parameters as query.params (list). The cursor method
execute() can now take an SQL object, and execute it just like
cr.execute(query.code, query.params)
It is quite easy to make SQL objects safe against SQL injections: if the
code is a string literal, then the SQL object is guaranteed safe,
provided the SQL objects within its parameters are themselves safe.
Part-of: odoo/odoo#134677
psycopg2.extras.execute_values was introduced in PR #101237
however it pypasses the override logic for cr.execute. As a result
1. --log-sql cannot log these queries
2. assertQueryCount cannot notice these queries
...
This commit create a new api cr.execute_values to support the same SQL feature
without losing the override logic for cr.execute
closesodoo/odoo#131190
Related: odoo/enterprise#47374
Signed-off-by: Rémy Voet (ryv) <ryv@odoo.com>
The dinosaur regular expressions[^1] used to count SELECT/INSERT queries in SQL
debugging mode were slowing down query execution. They can be replaced by
more efficient ones.
The `re.MULTILINE` flag is not necessary anymore, since there is no more
`$` or `^` in the expressions.
[^1]: f8471796a3closesodoo/odoo#129473
X-original-commit: 0730e432c470635baec962d16ca09e8423c2b3ba
Signed-off-by: Romain Derie (rde) <rde@odoo.com>
Signed-off-by: Olivier Dony (odo) <odo@odoo.com>
When exiting a flushing savepoint, the method flush() is invoked before
closing the savepoint (which releases it):
def _close(self, rollback):
if not rollback:
self._cr.flush()
super()._close(rollback)
If method flush() crashes at that point, the savepoint is not closed at
all (because of the exception) and is therefore not even rolled back.
The expected behavior is that the savepoint rollbacks and the exception
bubbles up.
closesodoo/odoo#126866
X-original-commit: 5f9b4f5c998388625de27fbd8fe0f7a0524c04fd
Signed-off-by: Rémy Voet (ryv) <ryv@odoo.com>
Signed-off-by: Raphael Collet <rco@odoo.com>
The Gevent worker has specifc needs in term of maximum concurrent
connections to the database and those needs are not compatible with the
default limit that is primerly set for http workers.
This PR makes it possible to supply a configuration dedicated to the
gevent worker.
task-id-3193565
task-id-2146565
closesodoo/odoo#125190
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
This commit introduced a way to log queried table on cursor without
having to be in `--log-sql` mode.
It will be used in next commit to add complete performances tests which
will ensure the correct tables are accessed when a page is requested.
This is useful as right now those performances tests are only doing a
check against the SQL query number without ensuring the queries are the
correct ones.
As a result, one could introduce an improvement which would reduce the
queries (-2) but break another previous improvement (-2) which would
add back those queries (+2). Since the query count would be the same
(-2+2), it would be undetected, instead of being fixed directly and
keep both improvements (-2-2).
See next commit for concrete example(s) and use of this improvement.
X-original-commit: 8ef6137a34aa35182399a2bd4b20dd57479ab2e7
Part-of: odoo/odoo#124939
Before this commit, a multiline SQL Query (crafted, as the ORM is one
lining the queries) would not be catched in the `sql_from_log` and
`sql_into_log` variables, despite being shown and counted in the request
queries (unless the `into` or `from` part of the query is on the first
line).
At the end, not only the query was not listed in this table report but
if you sum the table report queries, it would not match the shown query
count.
This is the case for the `website_visitor` UPSERT query:
```sql
WITH visitor AS (
INSERT INTO website_visitor (...)
VALUES (
..., (
SELECT id FROM res_country WHERE code = NULL
)
)
ON CONFLICT (access_token)
DO UPDATE SET
...
RETURNING ...
)
SELECT id, upsert from visitor;
```
This commit is allowing the `into` and `from` part of the select to be
detected even if it's not part of the very first line of the query.
As a result, since a SELECT and an INSERT could be part of a same query,
we exclude the SELECT to be considered as a SELECT query if an INSERT
was already found in this query.
This will allow to have table query count log be in par with the query
count number (except for the `base_registry_signaling`, see note below).
It will also be helpful in next commits which will introduce a way to
access those logged queried table in the testing suite without being in
`--log-sql` mode.
Note: the SQL query on `base_registry_signaling` (which occurs every
time to check if the cache should be invalidated) is not tracked
either due to how it is build, it doesn't seems really needed to
have this one shown in the table logs. See [1].
This commit also remove the `lower()` usage and use the `IGNORECASE`
flag from `RE` instead.
[1]: https://github.com/odoo/odoo/commit/4aeba0c12e0512e537178e6cba0993a11f001ec3
X-original-commit: 255aa168bef5d63480e88b4d34511e3aac910f49
Part-of: odoo/odoo#124939
Before this patch the naive dsn parser `_dsn_to_dict` would choke
on `application_name` containing spaces or the equal sign.
closesodoo/odoo#122212
X-original-commit: 3573fe0726f8dde1c5714b7d501f6c43d97ce731
Signed-off-by: Fabien Meghazi (fme) <fme@odoo.com>
This is an improvement over commit f6c13d7 in order to allow full
customization of postgresql connection's `application_name`.
In a PID linux namespace the containers processes IDs might not be relevant
for a system administrator and one might want to customize the content of the
`application_name` just like it was possible to do before commit f6c13d7
using the `PGAPPNAME` environment variable.
This commit allows to expose an `ODOO_PGAPPNAME` environment variable whose
content will be interpolated with the PID (optional)
closesodoo/odoo#117921
X-original-commit: a4108eee91f8e9bf82be5a4be4b91bab16baa0df
Signed-off-by: Olivier Dony (odo) <odo@odoo.com>
During revision 415525cecc
the code comparing the connections without password has been
dropped, for an unknown reason.
Because of the above, and the fact psycopg2 alter the password
with 'xxx' for security reasons, the connections recycling was
no longer working as expected, and was recycling less connections.
closesodoo/odoo#37005
Signed-off-by: Rémy Voet <ryv@odoo.com>
Before this commit, the connection pool works greedily regarding the
number of connections opened. Once a new connection is needed, it is
added to the pool and never closed unless the maximum number of
connections is reached.
We introduce 2 changes. First, we do not cycle on the available
connections anymore. We take the first one available and let it at its
position in the list. It opens the possibility to use
`idle_session_timeout` introduced in PostgreSQL 14.
Moreover, we introduce a garbage collection of the unused connections.
If the connection has been unused for more than `MAX_IDLE_TIMEOUT`
seconds, it is closed and removed from the pool.
This makes easier to overcommit `--db-maxconn` when running in worker
mode. We can set a value which is large enough for the websocket
requirements without the WorkerHttp having a large number of useless
opened connections.
closesodoo/odoo#113110
X-original-commit: 9fcf3de6a9f10d4171bb48feb65fd789db19d4c6
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
Start odoo on a specific database, e.g. 'db-example'. Drop it via JSON
or XML RPC. The database is successfully dropped but the RPC fails with
a traceback because it attempts to commit on a database that doesn't
exist anymore.
import requests
admin_passwd = ...
requests.post(
'http://127.0.0.1:8069/jsonrpc',
json={'params': {
'service': 'db',
'method': 'drop',
'args': [
admin_passwd, 'db-example'
]
}}
)
Closes odoo#104527
closesodoo/odoo#105710
X-original-commit: b6e195ccb3a6c37b0d980af159e546bdc67b1e42
Signed-off-by: Julien Castiaux <juc@odoo.com>
- Remove `_default_log_exceptions` of our `Cursor` class (unused except
in one test)
- Deprecated `serialized` args of `__init__` (Cursor class) and
`cursor()` (of Connection class), our cursor is always serialized.
- Remove `sql_log` attribute of Cursor class and replace it with
appropriate code to do the same stuff dynamically.
- Simplify some code
- Update some docstring
- Clean import
task-2766494
closesodoo/odoo#85078
Signed-off-by: Rémy Voet <ryv@odoo.com>
- Remove depreciated `after` method of `Cursor`
- Remove `unbuffer`/`flush_env`/`clear_env` methods
- Remove specific code for psycopg version < 2.7 (psycopg version is
always >= 2.7 in our requirements)
- Depreciated autocommit
task-2766494
Part-of: odoo/odoo#85078
Since e014cc88394fb9c8f6cff556ca73d5efafea030f, our proxy
`Cursor` object doesn't return a understandable error when we
try to use it when it is already closed
(`NoneType` or `Recursion` error).
Instead of reverting e014cc88394fb9c8f6cff556ca73d5efafea030f,
add a check in `__getitem__` raising a IterfaceError (like psycopg)
if we try to use psycopg cursor (`self._obj`)
task-2766494
Part-of: odoo/odoo#85078
Uses a dedicated logger (for easier filtering / silencing) for
results output, and provides rough (module-level) stats in INFO but
detailed (test-level) in DEBUG.
Also updates the global query counter (`odoo.sql_db.query_counter`) to
update after each query rather than on close: with test cursors the
actual underlying counter is only rarely flushed.
closesodoo/odoo#95420
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When freezegun is used, the profiler and sql_db time are freezed,
Making the profile and sql perf counters invalids.
A possible solution would be to black list some modules in freezegun but
this doens't look possible in the pinned version (0.3.x).
Saving the builtin time.time is not enough, it looks like freezegun will
find all occurences and replace them.
We need to get the __call__ instead.
closesodoo/odoo#95100
X-original-commit: 9ac5fdf1e6d00e6e4b4ff4b6e94a3cd28f8cae11
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
The call to `commit()` can fail during the flush of the environment
and raise an exception before the actual `COMMIT;`, leaving the
cursor unclosed.
This leaked cursor may hold locks that only be released when the GC
collects it. On low-traffic workers like the CronWorker, this can
lock usual database usages.
X-original-commit: 3cb3cec67c28ca147288103adb219e6da8694248
Part-of: odoo/odoo#93993
The `t-cache` directive allows you to keep the rendered result
of a template part. The supplied key must be a tuple. This tuple
can contain recordset in this case the zone will be invalidated
each time the write_date of these records changes.
The `t-nocache` directive makes it possible to force rendering
of a part even if it is in a `t-cache`. The values available in
the `t-nocache` are the one provided when calling the template
(and therefore ignores any t-set that could have been done).
Part-of: odoo/odoo#88276
Issue: The `web.tour` always uses the same date for record creation
or write date. This can add indeterminism regarding the order of
some records. It also prevents to make a comparison with the
write_date or the create_date.
Part-of: odoo/odoo#88276
This commit is the 14th commit of a comprehensive refactor of our HTTP
framework. See odoo/odoo#78857 for complete historic, discussions and
rationnals.
* `request.uid = x` => `request.update_env(user=x)`.
* `request.context = x` => `request.update_env(context=x)`.
* `request.context = dict(request.context, x=y)`
=> `request.update_context(x=y)`.
* `request.cr = None` => `request.cr.close()`.
* `http.mono_db()` => `request.db`.
* `http.dispatch_rpc()` => `service.dispatch_rpc()`.
* `@service.model.check` => `service.model.retrying()`.
* `request.endpoint`
=> `env['ir.http']._match(request.httprequest.path)[0].endpoint`.
* `request.routing_iteration `=> `removed`.
* `request.jsonrequest` => `request.dispatcher.jsonrequest`.
Note that `request.params` is now set much later in the process. If you
are in a situation where you values from the query string or the
http body you can use `request.get_http_params()`.
Note that using the new `request.future_response`, it is possible to
add headers and cookies on the response object before the response
object is initialized. Please note that headers/cookies saved on
the future response will NOT be injected in case of error.
PR: odoo#78857
Task: 2571224
The `check` decorator in `sql_db.py` was on a lot of `Cursor` methods.
It checks if the cursor is close before be using a the method.
Remove it because:
- It is completly redundant because `psycopg2` do already the job
to check the cursor before usage.
- It complicated the call stack and lead to a small overhead of
highly use methods (can be more than 1% of the execute call)
- Also the nature of
the Error isn't correct: raise `OperationalError`
(https://www.psycopg.org/docs/module.html#psycopg2.OperationalError)
instead of `InterfaceError`
(https://www.psycopg.org/docs/module.html#psycopg2.InterfaceError).
Part-of: odoo/odoo#80961
It's not entirely clear whether `@synchronized` is even useful, but
keep it for now. `locked` is just the default instance of
`@synchronised`.
- rewrite `@synchronized` using `decorator`, don't fold everything
into a single call as there's a potential for parametric conflict
- remove the independent `locked` in `sql_db.py`
- convert `lru` to `locked`
- move `Registry` over to `locked` where applicable
closesodoo/odoo#82718
Signed-off-by: Raphael Collet <rco@odoo.com>
It helps to debug queries executed in postgresql from Odoo
in order to know where they were called
Enabling the postgresql logs with the following `log_line_prefix`
log_line_prefix='%t [%p]: [%l-1] db=%d,user=%u,client=%h,app=%a '
You will see the following output in the postgresql.log:
... UTC [394452]: [371-1] db=odoo,user=odoo,client=127.0.0.1,app=odoo-740755 LOG: 00000: duration: 0.074 ms statement: SELECT 1
Notice `app=odoo-740755` it is the odoo pid that executed the query
and the postgresql PID `... UTC [394452]:`
Then you will be able to match the odoo.log and postgresql.log using the PIDs
740755 DEBUG odoo odoo.sql_db.connection: ConnectionPool(used=1/count=2/max=64) Create new connection backend PID 394452
740755 INFO odoo odoo.addons: Running SELECT 1
Notice the Odoo PID `740755 INFO` and the postgresql PID `backend pid 394452`
Note: It will require enable the sub-logger
- `--log-handler=odoo.sql_db.connection:DEBUG`
It will helps to debug what process is executing each query in the database
or if a postgressql PID is showing a error log related to connection (not even from a query)
e.g. The livechat stuck and you don't know what happen but you can see the postgresql.log the following message
for the same PostgreSQL backend_pid related to longpolling odoo pid
[394452]: [371-2] db=odoo,user=odoo,client=127.0.0.1,app=odoo-740755 LOG: XX00: Could not receive data from client: Connection time out
closesodoo/odoo#82857
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
* add configuration for `flake8[flake8-rst-docstring]`
* enable docstring-related checks
* fix invalid docstrings in odoo's core & `base`
* fix a few more bits (mostly missing or incorrect `:param:` info
fields) are out of scope for the lint but my editor catches
closesodoo/odoo#74604
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Because test cursors are implemented using savepoints, they should *at
most* be nested (ideally they would be strictly sequenced).
If upon being closed a test cursor finds a *different* un-closed
cursor at the top of the stack, one of its followers / children /
descendants was not closed before it, which is a problem.
The initial version of this would look for the cursor being closed in
the stack but this could lead to incoherent cursor stacks and errors
related to the management of the cursors stack, even though it only
exists for reporting reasons.
closesodoo/odoo#76243
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Initialize the savepoint semi-lazily (on-demand) as cycling a savepoint
generates 4~5 queries (depending whether the savepoint is explicitly
released on COMMIT or not):
SAVEPOINT
-- < do stuff>
-- commit
RELEASE SAVEPOINT -- or not
SAVEPOINT
-- close
ROLLBACK TO SAVEPOINT
RELEASE SAVEPOINT
With a lazy savepoint, this is just 0-1 queries (`SAVEPOINT` at the
first explicit query only, creating a cursor and then closing it
immediately is a no-op).
For reliability use a semi-lazy savepoint: always immediately emit a
`SAVEPOINT` on cursor creation, but don't automatically create one
after each `commit`. That limits the issues of overlapping (but
non-nested) savepoints.
Also fix the `generate` API to not use the request: when `website` was
converted to the new API, `cr`, `uid`, and `context` were dropped as
if it were a model... but it's not. So in order to recover an
execution environment, that was looked up on the session.
That, then, turns out to be an issue when `generate` is triggered from
an RPC call: the RPC layer creates its own cursor and environment
separate from the request's which may not have one at
all. Problematically during testing we're in `mono_db` mode, so the
request's cursor/env can be accessed and will be lazily initialized.
This then causes an issue with the `TestCursor`'s savepoints: rather
than be nested, the lifetimes of the request's and RPC's savepoints
only overlap[0]:
|-- rpc --|
|-- request --|
As a result, when the RPC's cursor is committed and released it
automatically released the request's, and the request's explicit
release then fails. This would break `/website:WithContext.test_search`.
By fixing the API of `generate`, it stops triggering the creation of a
request cursor, and therefore the overlap and resulting error.
[0] the laziness or eagerness of the savepointing in the test cursor
has no impact on this issue, as multiple requests have already been
issued on the RPC's test cursor before the request's is even
created
Part-of: odoo/odoo#76243
Ensure savepoints are *always* released when exiting the context:
rolling back to a savepoint does not release it, so the savepoint
would remain "active" forever (just possibly shadowed).
Also provide a `Savepoint` object to the user, with the following
facilities:
* `name`, in case there are useful things the user can do with a
savepoint name.
The name uses standard UUID representation (rather
than pure hex) because it's otherwise difficult to differentiate
savepoints: in a UUID1, fields 2, 3, and 5 almost certainly don't
change, and the changes between two UUIDs are the last 2-3 nibbles
of field 1 (`time_low`) and the content of field 4 (`seq`, 4 nibbles
at offset 16), with "properly" separated fields it's much easier to
notice the difference.
* `rollback` allows the user to rollback the savepoint to the
initialisation state at any moment.
* `close` allows the use of `contextlib.closing` as well as closing
the savepoint while in the covered span. Closing a savepoint rolls
it back by default (like cursors).
Because there's no such thing as committing a savepoint, it's also
possible to close *without* rolling back, which is similar (but not
identical).
This requires using the semantics of emitting the `SAVEPOINT` during
object initialisation: `closing` was designed to work with non-CMs so
it does not forward `__enter__` to the wrapped object.
Also introduce a CM type for the flushing:
* `_GeneratorContextManager` does not work well when used outside of a
`with`, especially when it yields something: if the CM itself goes
out of scope, the inner generator is `close`d, which raises a
`GeneratorExit` at the `yield` point, which `__exit__`s whichever CM
is held (and in our case would thus rollback and release the
savepoint)
* we probably want to clear() on `rollback`, so having the flushing CM
extend the savepoint one makes a lot of sense
* while it changes the semantics of `Cursor.savepoint` a bit (now
initialises the savepoint at call time instead of delaying until
`__enter__`), it enables the use of `closing` with flushing, and
without having to use the context manager objects directly:
`Cursor.savepoint()` is the only necessary interface, and should
work fine with and without flushing.
Part-of: odoo/odoo#76243
Enabling sql logging for a specific section of code is currently a
pain in the ass as it requires updating both the cursor and the
logger. This utility does that.
This CM is *not* thread-safe as it updates and resets the cursor
without using CAS or anything, so if cursor 1 gets enabled, then
cursor 2, then cursor 1 stops, then cursor 2, cursor 2 will reset the
logger to `logging.DEBUG` which cursor 1 had set, rather than the
probable `logging.NOTSET` it originally was.
Should be possible to fix by e.g. storing the levels in a static
stack (and setting whatever gets popped) buuut... the logging is
already a bit of a mess when multithreaded so I'm unsure it matters
much.
Part-of: odoo/odoo#76243
Refactor the Environments object into a Transaction object, which is
bound to one cursor, and is no longer shared among several cursors.
The following methods/properties have been changed:
- Environment.envs no longer works (because of the design change);
- Environment.manage() is deprecated (no longer useful);
- Environment.reset() is now an instance method;
- env.clear_upon_failure() is deprecated in favor of cr.savepoint().
closesodoo/odoo#75598
Related: odoo/enterprise#20451
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Xavier Dollé <xdo@odoo.com>
This commit adds tooling to profile performance and save execution by
saving stack traces and queries to a file/database in specific format.
----------
Collectors
----------
For now, three different profiling modes (aka Collectors) are available
even if a last once should be introduced by @Gorash to profile qweb
execution.
- SQLCollector (or 'sql'): Saves the current stack trace and the query
every time Cursor.execute() is called. Any query executed on the thread
will be collected, no matter the cursor.
- PeriodicCollector (or 'traces_async'): Saves the stack trace every
'interval' seconds using a parallel thread to profile the caller thread.
The python implementation was optimized to minimize impact on
performance while remaining portable and easy to enable/disable
inside a odoo execution. Higher the frequency (lower the interval),
more impactful the profiling will become on the execution and increase
memory usage. From last experiments, 1ms looks to be a good minimum for
short executions.
- SyncCollector (or 'traces_sync'): Saves the stack trace every function
call/return. This collector is obviously quite impactful on performance
and can quickly overload the memory for long executions, but this is
quite useful to understand the precise path followed by some short
executions. Any time related information will be almost irrelevant with this
collector.
A base Collector defining minimal collectors features can easily be
extended to create custom collectors if needed.
----------------
Profiler & Usage
----------------
Collectors are not supposed to be used by themselves, but should be
given to a Profiler. The Profiler will synchronize collectors starts and
stop, and manage saving them to a file of in a ir_profile in the
database.
Exemple of usage:
```
with Profiler():
do_stuff()
```
This simple example will use the default collectors (sql and
traces_async) and save them to the database. The database is defined
automatically from current_thread 'dbname' if available.
Example of usage:
```
with Profiler(collectors=['sql'], db=False, path=/home/user/logs/do_stuff_profile/{time}):
do_stuff()
```
This more complex example disable the default behavior consisting
to save to the database, gives a path where the profile will be saved
and specify to only use the 'sql' collector. Note that
collectors=[SQLCollector()] would have the same behavior since
Collectors can be either a Collector instance or a string describing the
desired collector. This allows to define custom params for the
collectors and use custom collectors if needed.
Note that it is always possible to get results after execution without
saving it since they are available on the profiler.
```
with Profiler(collectors=['sql'], db=False) as p:
do_stuff()
print(len([None for entry in p.collectors[0].entries if ...]))
```
Profiler will also save the stack below the profiler start point, and
collectors will only collect the part of the stack over this stack.
This is a good way to reduce collectors CPU and memory usage.
Collected entries will be saved as follows:
```
[{
'start': 2.0,
'context': {},
'stack': [
['path_to_file', lno, 'func_name', 'line_content'],
...
],
},
...
]
```
SQLCollector will add three additional keys on each entry:
- query (query without parameters)
- full_query (mogrified query with parameters)
- time (the 'exact' execution time of the query)
----------------
ExecutionContext
----------------
A last tool, ExecutionContext, allows to define some context on some block of code:
Example of usage:
```
def process_modules(modules)
for module in modules:
with ExecutionContext(module=module): # note the 'not linter frienldy but still convenient' 2 spaces indentation
do_stuff(module):
```
This context will automatically be added in the stack as a virtual frame between
process_modules and do_stuff in order to split do_stuff from one single frame to
one frame per module.
----------
Speedscope
----------
The saved data are in a simple json format easy to analyze, but can't be visualized in
speedscope as they are. A utility class `Speedscope` can be used to generate a format
readable by speedscope. The used format is actually the format defined by speedscope,
meaning that all features should be available using it.
The output format is evented, meaning that we need to transform a list of samples
(a list of stack) to a list of event (going in/out a frame).
This is the main task of the Speedscope, as well as combining samples from different
sources, to display SQLCollector and PeriodicCollector results mixed together.
When stored on an ir_profile, the default speedscope generation can easily be generated
with the speedscope computed field.
This class can be used as it is but will mainly be useful for the next commit.
Special thanks to @rco-odoo for the in depth review and @Gorash for support.
Since Odoo 13.0 and the refactor of the internal synchronisation between
the cache and the database, it is unsafe to use the ORM with the cursor
in autocommit mode. The function is deprecated for removal in the
future.
closesodoo/odoo#68313
Task: 2442905
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The goal of this change is to simplify the code managing `create_date`
and `write_date` in methods `create()` and `write()`, and also to remove
weird behaviors caused by the way those fields were updated.
Assume we update a simple field on a record. This adds pending updates
for the field and `write_date`. However, the value of `write_date` is
not known yet: it will be updated as `NOW() AT TIME ZONE 'UTC'` in SQL.
So `write_date` is actually given a dummy value in pending updates, and
it is invalidated from cache, until its value is flushed to the database
and fetched again.
Now assume we access another field on the record, and that field is not
in cache. The prefetching mechanism will read all column fields,
including `write_date`, and flush them first.
# this adds pending updates foo: 42, write_uid: 1, write_date: False
record.foo = 42
# assume 'bar' is not in cache; this prefetches all column fields,
# which flushes the pending updates above before reading them back
result = record.bar
We can avoid flushing pending updates if the values read from database
do not overwrite existing values in cache. If you assume that the value
of a pending update is in cache (in the example, `foo: 42`), you don't
need to flush the corresponding field. Indeed, the value of `foo` will
remain 42 in cache, whatever its value in the database. This assumption
(pending updates are in cache) is true for all fields *except* for
`write_date`: it is invalidated from cache, and given a dummy value in
pending updates. This branch actually makes this assumption true for
all fields. The avoidance of flushing pending updates will be done in
another commit.
In order to directly assign `write_date` its value, we use a cache for
the value `NOW() AT TIME ZONE 'UTC'` from the database. This costs at
most one query per transaction, and potentially saves a few queries.
Co-authored-by: Victor Feyens <vfe@odoo.com>
Before this change the rollback hooks were never called since rollback() was never explicitely called by the framework. At the end of a transaction, if no error occurs the famework call commit() on the odoo cursor. In all cases, the transaction ends with a call to close(). Into the implementation of _close() rollback() is called on the underlying connection to ensure that not committed changes are rollbacked. That's the reason why despite the fact that rollback() was not called on the Odoo cursor, changes are not committed into the db in case of exception. To keep the same behaviour and avoid to have to explicitely call rollback() on the odoo cursor to trigger the execution of registered rollback hooks, these hooks are now processed in _close(). Since the list of registered hooks is emptied if commit() is called, we are sure that rollback hooks are only executed in case of rollback.
OPW #2294911closesodoo/odoo#60339
X-original-commit: dce9a05f3a5d37fab711c8ca3c5444941f98e814
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
This fixes a crash in some controllers with `auth='None'` where some
updates are flushed with an environment where `uid=None`. When there is
an environment with a real uid, preferably use it.
closesodoo/odoo#59658
X-original-commit: 2795c86df54389b858a454cbc1782b1f34f3ba4f
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Python 3.8 changed the equality rules for bound methods to be based on
the *identity* of the receiver (`__self__`) rather than its *equality*.
This means that in 3.7, methods from different instances will compare
(and hash) equal, thereby landing in the same map "slot", but that isn't
the case in 3.8.
While it's usually not relevant, it's an issue for `GroupCalls` which is
indexed by a function: in 3.7, that being a method from recordsets
comparing equal will deduplicate them, but not anymore in 3.8, leading
to duplicated callbacks (exactly the thing GroupCalls aims to avoid).
Also, the API of `GroupCalls` turned out to be unusual and weird. The
bug above is fixed by using a plain list for callbacks, thereby avoiding
comparisons between registered functions. The API is now:
callbacks.add(func) # add func to callbacks
callbacks.run() # run all callbacks in addition order
callbacks.clear() # remove all callbacks
In order to handle aggregated data, the `callbacks` object provides a
dictionary `callbacks.data` that any callback function can freely use.
For the sake of consistency, the `callbacks.data` dict is automatically
cleared upon execution of callbacks.
Discovered by @william-andre
Related to odoo#56583
References:
* https://bugs.python.org/issue1617161
* python/cpython#7848
* https://docs.python.org/3/whatsnew/changelog.html#python-3-8-0-alpha-1
(no direct link because individual entries are not linkable, look for
bpo-1617161)
X-original-commit: d4b2e9224839aed8fc160ebe5a89e0f7d4c6a5bb
Upon cursor `commit()`, the pending computations are performed with
method `flush()`. However the latter method leaves fields to compute on
new records. As the `commit` ends the current transaction, we can clear
those pending computations.
X-original-commit: d27e98d9048d0cb1250522205286ebc7b7d5ea39
- A bug has been introduced by the PR https://github.com/odoo/odoo/pull/46719
This issue prevented flushes to database when no `uid` is set in the
current environment.
This happens when database changes are made in `auth="none"` routes.
In a MonoDB setup this was working because Odoo consider `None`, `False` as
`null` and was always bound to a database.
Nothing prevents to write `null` in the columns `create_uid` and
`write_uid` in database.
The PR that introduced the bug wanted to block the cases where
`uid` is an instance of the class `RequestUID` to avoid `Cannot adapt
type` errors.
So testing that `uid` is an integer isn't enough.
Two fixes were possible here, either check if `uid` is an instance
of the class `RequestUID` or if it is either an integer, `False` or `None`.
To avoid noise and imports from `odoo`, we test that `uid` is an instance
of `None`.
The case where it is `False` is already tested in the python code:
```python
isinstance(env.uid, int)
```
(manual) forward-port of odoo/odoo#48685closesodoo/odoo#48693
Signed-off-by: Toufik Benjaa (tbe) <tbe@odoo.com>
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
When routing an HTTP request, the dispatcher uses an environment with a
placeholder for the user (the object `RequestUID`). This environment is
not supposed to be used after routing has been done. However, the
flushing function takes any environment with a given cursor, and it
crashes when this environment is chosen (`psycopg2` cannot adapt the
type `RequestUID` for `write_uid`), and simply does not make sense.
closesodoo/odoo#48495
X-original-commit: da61a64bf6731a08557d23680eabf4d910035ebd
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
The `__closer` attribute is filled with the origin of a call to
`cr.close()`. That information is reused in the `check` decotator that
ensure the user is not using a closed cursor. In case he is using a
closed cursor, an error is raised with that optional origin in case of
`--log-sql`.
The attribute is a dundler so it is only accessible within its class, as
the decorator has been moved outside of the class by 058cf208a8 it is
not directly accessible thus `__getattr__` is called as a fallback. The
`__getattr__` itself is protected by the `check` decorator thus they
call each other in a infinite recursion.
As the related dundler is no more used internally, it has been decided
to remove it entirely.
closesodoo/odoo#47616
Task: 2199895
X-original-commit: 3ebf7cb9cca37b05f44aaffc0b1f289cc65f300f
Signed-off-by: Julien Castiaux <Julien00859@users.noreply.github.com>
Starting with Python 3, queries sent by psyocopg2 are stored as `bytes()` objects.
Logging those raw makes them appear unformatted, harder to read than in v10 or lower Odoo versions (i.e. `\n` instead of a raw newline character).
Decoding the query into unicode makes it easier to read in the logs.
closesodoo/odoo#38074
X-original-commit: 329accde2ed6d4dbfdde6d3d99d71e21d15396b7
Signed-off-by: Christophe Simonis <chs@odoo.com>
Xavier Morel