Before this PR:
The only way to neutralize the database is running cli command neutralize
After this PR:
There is a new checkbox "neutralize database" in Duplicate database and Restore database dialog that neutralize the database after duplication/restore.
I also moved the neutralization code to the external module so it can be called also outside the cli .
closesodoo/odoo#122185
X-original-commit: 616740e9d09b3d0376be43ed1489e390f6f5823e
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
We added trigram index for char fields since https://github.com/odoo/odoo/pull/83015.
But if `unaccent` is installed in the database (and isn't force to
`False` on field), these new trigram indexes are pointless and cost a
lot for nothing (almost nothing, it still can be used for equality
operator but in this case a btree will be far more efficient).
The simple way to fix it is to add `unaccent(<column>)` in the index
trigram definition, but unfortunately `unaccent` is not immutable and
may therefore not be indexed. In order to make `unaccent` indexable, we
must declare it as immutable (see
https://stackoverflow.com/questions/11005036/does-postgresql-support-accent-insensitive-collations/11007216#11007216
for more information and how to do that).
With this patch, trigram indexes are created with `unaccent(<column>)`
if the function `unaccent` is available in the database, and for the
fields that are not declared with `unaccent=False`. Moreover, we issue
a warning when `unaccent` is available but is not immutable, in which
case most trigram indexes will be useless.
odoo/upgrade#3736
task-2551518
closesodoo/odoo#95943
Signed-off-by: Rémy Voet <ryv@odoo.com>
Currently dbs can only be managed via the UI in order to take
filestores in account: while it's possible to load/copy/rename/drop
databases via `psql`, that will not manage the related filestores so
the result of the operation is incomplete DBs and leftover filestores
littering the disk.
Seems like a good idea to add a CLI to perform the same
tasks. Currently the CLI calls into the corresponding service, rather
than both calling into (possibly better designed) unified APIs, but
that seems fine for an initial version.
The top-level `db` command acts as a db manager, with git-style
sub-sub-commands for the various operations:
- `load` to load a dump file into a database (with a specified name or
not)
- `dump` to dump a local db to a zip dump (pg_dump can be created via
the corresponding command so not a concern)
- `duplicate` and `rename`
- `drop` in order to drop both the database itself and the
corresponding filestore
Notably, `create` is currently left out because a database can
trivially be created by invoking odoo using a dbname which doesn't
exist, so doesn't seem useful.
`list` is also left out, because `psql -l` generally does the job.
closesodoo/odoo#97365
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
They're not used much and they don't seem much of an advantage
compared to just calling `subprocess` with the discovery utility
functions.
While at it, fix `dump_db` to not unnecessarily have an open stdin to
`pg_dump`.
Part-of: odoo/odoo#98023
When setting the password in the database manager the config is saved
automatically in a odoorc file.
This can be problematic, especially when testing locally, with
specific options.
Going in the database manager and creating a new database or changing
admin pasword will save all those options.
This commit proposes to only save the admin_passwd when modified.
Part-of: odoo/odoo#82874
A cursor created by a Connection object is not expected to be used with
environments; use registry.cursor() instead.
closesodoo/odoo#78143
X-original-commit: 6aeb73f10953f3547b7cf830718c02a3933df9e0
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Creating a database from the Odoo CLI miserably fails with the error
"psycopg2.ProgrammingError: set_session cannot be used inside a
transaction".
Setting con.autocommit = True fails if some transaction is already
started, which is the case when creating a database. The fix consists
in rolling back the existing transaction (with only a SELECT) before
switching to autocommit.
closesodoo/odoo#68549
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Following 7a235c19ff, use an alternative
API to the method autocommit(). Several functions managing databases
use a connection in autocommit mode to execute some commands outside of
a transaction.
closesodoo/odoo#68491
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
odoo/odoo#53938 improved the SQL linter and used psycopg2.sql to
silence the linter where that still made sense. However I forgot to
mark table names (pretty much exclusively) as `sql.Identifier` in a
few somewhat rare callsites, which consequently break when invoked as
a simple string is not a Composable and psycopg2 therefore rejects it
when composing the query.
odoo/odoo#54556 fixed a few mis-updated ones, but apparently I still
managed to miss one here.
closesodoo/odoo#56191
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
odoo/odoo#53938 improved the SQL linter and used psycopg2.sql to
silence the linter where that still made sense. However I forgot to
mark table names (pretty much exclusively) as `sql.Identifier` in a
few somewhat rare callsites, which consequently break when invoked as
a simple string is not a Composable and psycopg2 therefore rejects it
when composing the query.
closesodoo/odoo#54556
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
The linter would miss / fail to warn on injection of *local variables*
in some cases.
Try to improve it to be stricter and more reliable, after discussion
with odo, sql which is "correctly" dynamic should use psycopg2's sql
package in order to bypass the linter (bonus: it should also properly
escape & quote identifiers).
closesodoo/odoo#53938
Related: odoo/enterprise#11718
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Avoid logging an error in the logs for an operation that is not an
closesodoo/odoo#48957
Error: if the extension already exists, this is a success.
X-original-commit: 2f884942c94a640ae5107ff2acb27ba4ecc60bc0
Signed-off-by: Damien Bouvy (dbo) <dbo@odoo.com>
The unaccent extension was installed only at restore, not at creation
of a database.
Make both consistent.
closesodoo/odoo#46534
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
* walksymlinks is useless, os.walk got followlinks in 2.6
* tempdir is redundant with tempfile.TemporaryDirectory added in 3.2
* listdir(recursive=False) is just os.listdir (I guess recursive=True
is somewhat useful)
* zip_dir is *not* redundant with shutil.make_archive:
- zip_dir handles the archived root differently
- zip_dir sorts files, make_archive sorts directories
- make_archive explicitly adds entries for directories (including
empty), sorts directories, doesn't sort files; zip_dir sorts
files (including with a custom key function) but ignores directories
- zip_dir filters out a bunch of trash files
closesodoo/odoo#39573
Signed-off-by: Christophe Simonis <chs@odoo.com>
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
Given a country code is specified and the country only has one timezone, it is
safe to assume that the admin user is operating from that timezone.
Set the default timezone to simplify the onboarding in such particular case.
If a country has more than one timezone, we can not make safe assumptions and
don't provide a default tz.
closesodoo/odoo#26365
Open the web/database/manager and create a db with a country set.
The country of the first company will be set to the country set on the form
Before this commit, the currency was not set, eventhough it is a field
on res.country
After this commit, we also set the currency of the first company
opw 1869435
closes#26097
As a follow-up to commit 276ea817f4,
we change the default template database used for CREATE DATABASE
operations, and we avoid setting the LC_COLLATE option unless
this default template is used (to avoid locale compatibility errors).
Rationale for the default template choice:
PostgreSQL uses `template1` by default, and recommends that users alter
`template1` if they need a custom template, while keeping template0
virgin. This is likely to be the reason why we had `template1` as the
default previously (from a1e5c645d9).
However, for new Odoo databases, it's actually better to use `template0`
as template, for multiple reasons:
- We do not customize the template database, and the system will work
fine when starting from a completely virgin database
- Since template0 does not allow connections by default, we have a
stronger guarantee that no user will be connected to the template
database (which would otherwise block database creations!)
- We use the same logic for creating new databases and for creating
placholder databases for restoring dumps. In the latter case,
PostgreSQL recommends using a virgin template, to obtain a perfect
copy of the database being restored (unaffected by any template
customization)
- Using `template0` as a template lets us choose any encoding and locale
settings (LC_COLLATE/LC_CTYPE) upon creation. Any other database
template might cause errors unless we select compatible encodings
and locale settings.
As the locale setting is system-dependent, we can't just hope
that there will be no compatibility problem, we must explicitly
default to template0. Therefore the change in
276ea817f4 was prone to causing
locale compatibility errors.
Finally, if users have a reason to prefer a different template, they
should select it explicitly with the --db_template config.
Note: existing environments will likely continue to use `template1`
as a default due to presence of older config files.
Complements #25196
This makes sorting and indexing independent of the cluster/machine
configuration, and allows simple b-tree indexes to be used for
prefix-searches on VARCHAR columns without needing special operator
classes (and therefore without needing duplicate indexes).
The 'C' collation is built-in and always available on any postgresql
installation. It also allows any encoding/locale to be used, contrary to
other LC_COLLATE values, so it should never conflict with custom db
templates.
Admins who want to apply a special collation can still do so by creating
the database manually, instead of letting the system do it.
Closes#25196
When trying to list db's, the db_user is searched and verified before
searching for the databases.
As it's enclosed in a db.cursor context manager, we can assume that
there is a db user at this very moment.
This commit uses a single query to find databases that belongs to this
cursor user.
Also, this commit adds a message in the logs when an exception occurs
to facilitate debugging instead of silently returning an empty list.
get_dsn_parameters may return in some edges cases (like on runbot)
and empty value for the user.
So it should not be used here, instead, a SQL query is able to provide
the currently connected user.
Also, get_dsn_parameters is new in psycopg 2.7.
When Odoo is started with a db_user using 'postgres', the process
exits with a status 1. However, this can be bypassed when the
PGUSER environment variable is used.
This commit will prevent the usage of the 'postgres' for the
environment variable too.
Also, when trying to list db's, various methods where used to find a
suitable postgres user to get this list.
In fact, as the db cursor is available, a user is already at work.
So, in order to avoid code duplication (e.g. verify user from
environment variables), this commit removes those various method
and get the db user from the cursor.
- The `--no-database-list` option will now also block access to database
management functions and screens.
Presumably this flag should only be used in production when all
databases have been provisioned, so the admin should like to block
access to the db manager at the same time.
- If no `--database` or `-d` parameter is provided, the system will be
unable to fetch a list of databases at all, so users will be blocked
with an error message.
- Hide the link on the login screen to the DB manager when it is
disabled, to prevent sending users to an error page.
- Weak attempt at updating the documentation
Note: the security check for RPC methods could have been done in the RPC
dispatcher, however that would not have protected service methods when
called directly, e.g. by a controller (e.g. the dump method).
- Add support for hashed master passwords (super-admin password) using a
strong scheme (PBKDF2_SHA512).
- Replace the password with a hash in memory (tools.config map), after
verifying it
- Automatically replace the plaintext master password with a hash when
saving it after a password change
- Preserve support for setting/using plaintext passwords when necessary
(e.g. as a temporary deployment thing)
In such a case the db filtering won't be applied on an empty regex so
we use the list of databases passed to `db_name` (--database) as
the database list (and we also avoid to list all the databases present
on the postgresql cluster)
As a side effect, this patch allows to strengthen the postgresql security
by preventing Odoo to list all the databases present on the cluster.
* remove references to basestring & unicode (use relevant pycompat
helpers)
* remove some str calls (either entirely or replaced by relevant
helper, either text or native)
* use better API to avoid unnecessary conversions
* remove some XML declarations in views
* StringIO removed from stdlib, replace with io
* try to correctly handle BytesIO/StringIO (one is for bytes the other
is for text)
* fix base64: Python 3 removed bytes-encoding and bytes-bytes
codecs (via #encode) so replace all calls to str.encode('base64'),
also b64encode is a bytes->bytes conversion so attempt to properly
handle that
issue #8530
Michele