Commit Graph
37 Commits
Author SHA1 Message Date
Merel Geens (mege) c65fb9d198 [IMP] odoo: prevent unsafe import from file
It's possible to specify an existing module as the xml id prefix when
importing records from a file. This will result in `ir_module_data`
records being created with `noupdate` set to False. When the module is
upgraded, these records will be deleted.

This can lead to undesired side effects like journal items for
accounts imported this way being deleted.

This change prevents creating new records linked to existing modules
when importing from a file. Instead, the user should either use no
prefix or the name of a non-existent module, like `__import__`.

opw-3231987

closes odoo/odoo#133263

X-original-commit: 8f12d14c4c4354a63c94f14e7a89abc6048525f9
Signed-off-by: Merel Geens <mege@odoo.com>
2023-08-28 09:34:05 +00:00
Adrien Widart (awt) f41c3b1d9e [FIX] base: rollback if broken transaction while importing
Suppose a user who has CRM module and wants to import some leads
thanks to this CSV file:
```csv
name,recurring_plan
Coca-Cola,Plan01
SAP,Plan02
```
And, because the recurring plans are not yet created on his database,
he enables the option "Create new values" for that field. An error
will occur and here is the only info the user will have:
> current transaction is aborted, commands ignored until end of
> transaction block

When importing, we convert the data encoded by the user ([1]). To do
so, we convert the provided values, field by field ([2]). Since
`recurring_plan` is a `many2one` field, we go (through
`_str_to_many2one`) in `db_id_for`. In this method, we `name_search`
the record and then, if it does not exist and if the feature is
enabled, we `name_create` it ([3]).

Back to the above use case. We start with the first line and there is
not any recurring plan called "Plan01" so we try to create it. But
here is the issue: we only have a name to create the record although
there is another required field: `number_of_months`. Therefore, it
leads to a `NotNullViolation` error. This error is caught (see [3])
and, later in the same method, we will raise a `ValueError`. This
error will be caught by one of the except in [2]: we will save the
error and will then  continue with the convertion of next values.
However, because of the `NotNullViolation`, the current SQL
transaction is broken. As a result, while trying to convert the
second line of the file, we will `name_search` "Plan02" and it will
simply lead to a `InFailedSqlTransaction`

[1]
https://github.com/odoo/odoo/blob/f3d7fdce608f692ecb08498ee158edf9dfbced5e/odoo/models.py#L1170-L1182
[2]
https://github.com/odoo/odoo/blob/b80d4294a000e748677a3a0c1849140bf46ddbe8/odoo/addons/base/models/ir_fields.py#L117-L118
[3]
https://github.com/odoo/odoo/blob/b80d4294a000e748677a3a0c1849140bf46ddbe8/odoo/addons/base/models/ir_fields.py#L472-L475

sentry-3969379125

closes odoo/odoo#132897

X-original-commit: 28373b9d261a48154b233fbbd895880680b0aed0
Signed-off-by: Adrien Widart (awt) <awt@odoo.com>
2023-08-24 18:38:17 +02:00
Xavier-Do 595aa24843 [IMP] registry: multiple ormcache
One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.

Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.

The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.

Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.

The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.

We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.

Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.

Note that this poc is inspired from the long term cache but not all
use case where applie yet.

Part-of: odoo/odoo#119813
2023-07-18 11:42:26 +02:00
Rémy Voet (ryv) 3c62ca1eb9 [REM] core: remove name_get API
Rationale
=========

Since v8, the `display_name` field is present on all models. By default,
`display_name` uses `name_get` which has pretty much the same purpose
(return record name used by the web client). Gradually, many (backend)
developers (and the ORM: https://github.com/odoo/odoo/commit/6da1c3ac4c036eac289597602976538e243cb939)
started using `display_name` (more convenient than
`record.name_get()[0][1]`) but it still had the `name_get` override.
It becomes more complex than necessary and poeple start to misunderstand
the two (and sometimes override both, leading to inconstiencies between
`display_name`/`name_get`).

To simplify the ORM and the API, we decided to keep only one of them,
the `display_name` field:
- It is much more convenient from a backend point of view
(`record.name_get()[0][1]` vs `record.display_name`)
- It is cached during the same transaction (and invalidated if
its dependencies change)
- It can be overridden like any other compute field (override
`_compute_display_name` with any extra dependencies)
- `name_get` is replaced by `read(['display_name'])`
(API perceptive), which can actually be more efficient
(if `display_name`'s depends are correct, the ORM will only fetch the
fields it needs instead of every prefetchable field)

Changes
=======

- Deprecates `name_get` for the v17 and based the method on
`display_name` (the opposite of before)
- Converts all usage of `name_get`
- Overrides of `name_get` are now overrides of `_compute_display_name`
- For `res.partner`, rename the field store `display_name` into
`complete_name` because `display_name` context-dependent and it makes
no sense to have a compute store that is context-dependent.
- Previously, it was possible to return multiple names for the same
record with `name_get`, but it was tricky and most of the usage of
this `name_get` didn't take this into account. The only example of
this is the `name_get` of `product.product`
(now use `", ".join(<names>)`).

Part-of: odoo/odoo#122085
2023-06-28 17:41:19 +02:00
william eaa2e3b452 [FIX] base: import related field with name containing %
To reproduce, import and invoice containing one line with a tax of
i.e. `21%` on it.

Because the message is recursively expanded to follow the relation of
the field, the interpolation symbol needs to be escaped every time.
Otherwise the message will contain something like
> No matching record found for name '21%' in field '%(field)s/Taxes'

Where `'21%'` will try to be mistakenly interpolated.

X-original-commit: ceb3ce3030b47baa2aca128d364a823becb173ab
Part-of: odoo/odoo#104764
2022-11-02 22:20:40 +01:00
ef00294e71 [IMP] core: store translated fields as JSONB columns
Translated fields no longer use the model ir.translation.  Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table.  The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages).  Empty text is allowed in translation values, but not NULL.

Here are examples for a field with translate=True:

    NULL
    {"en_US": "Foo"}
    {"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
    {"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}

Like before, writing False to the field makes it NULL, i.e., False in all
languages.  However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.

Here are examples for a field with translate=xml_translate:

    NULL
    {"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}

Change for callable(translate) fields: one can now write any value in any
language on such a field.  The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values.  Basically the
structure of the value must remain the same in all languages, like before.

Reading a translated field is now both simpler and faster than the former
implementation.  We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:

    SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...

The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages).  For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.

Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language.  Before this change, only the source value of the
field ('en_US') could be indexed.

Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages.  We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.

Code translations are no longer stored into the database.  They become static,
and are extracted from the PO files when needed.  The worker simply uses a cache
with extracted code translations for performance.  This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker.  Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).

Performance summary:
 (+) reading 'model' translated fields is faster
 (+) reading 'model_terms' translated fields is much faster (no need to inject
     translations into the source value)
 (+) searching translated fields with operator 'ilike' is much faster when the
     field is indexed with 'trigram'
 (+) updating translated fields requires less ORM flushing
 (-) importing translations from PO files is 2x slower

Some extra fixes:
 - make field 'name' of ir.actions.actions translated; because of the PG
   inheritance, this is necessary to make the column definition consistent in
   all models that inherit from ir.actions.actions.
 - add some backend API for the web/website client for editing translations
 - move methods get_field_string() to model ir.model.fields
 - move _load_module_terms to model ir.module.module
 - adapt tests in test_impex, test_new_api
 - because env.lang is injected into SQL queries, its returned value is
   now guaranteed to correspond to a valid active language or None
 - remove wizard to insert missing translations (no longer makes sense)

task-id: 2081307

Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2022-09-15 22:37:50 +02:00
Thibault Delavallée ef9429231c [IMP] various: update query counters
Notably after odoo/odoo@f9442f47eb query counters are heavily impacted
and a lot of them has lessened.

closes odoo/odoo#83832

Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2022-02-02 18:14:54 +00:00
Xavier Morel 11c17318be [IMP] core, test_impex: use Savepoint in load and import tests
The query count increases by one because the old code (with
hand-rolled savepoints) never released the `model_load` savepoint.

Part-of: odoo/odoo#76243
2021-10-01 15:26:51 +00:00
Xavier Morel 7ac3a391a1 [FIX] core: don't break import on files triggering UnicodeEncodeError
The import logging (ish) assumes that if an exception has at least 2
args the second arg is metadata added by the callee.

As it turns out, `UnicodeEncodeError` has *five* arguments, none of
which is added by us. So if encoding something fails during the
process (e.g. because the file contains a lone surrogate, which leads
to the database insert failing when psycopg2 tries to encode the query
to UTF8), then the `_log` function itself will fail, yielding a very
unhelpful error of:

   dictionary update sequence element #0 has length 1; 2 is required

(because we tried to update a dict using a string).

This issue occurs only during *field conversion* and most fields have
no need to interact with the database (so don't need to encode the
value, which is what fails), however it is a problem when the invalid
string is used as a record name to look for (e.g. an m2o).

Further improve the experience by converting the UnicodeEncodeError to
a ValueError using the stringified UEE: `_log` assumes the first
argument to the exception is an error message of some sort, but for
UnicodeError subclasses it's just the encoding involved in the
error (here `utf-8`), which doesn't really serve as an error message.

Stringifying the exception generates a complete error message which is
quite a bit more helpful.

Specific update notes:

* avoid modifying the exception in-place, doesn't seem useful
* not sure why `field_name` was added as part of the augmentation
  rather than up-front when `record` is created

Issue 2480064

closes odoo/odoo#72517

X-original-commit: 6c3c500929cd463cd3a1749f4ede8f3a8afd5748
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2021-06-22 09:33:43 +00:00
David Beguin c483f59b37 [REF] base_import: revamp import module
Purpose
=======

Propose an import tool that is more intuitive and allows more import options to
ease the import.

Specifications
==============

Update upload import file screen
--------------------------------

- Relabel import welcome screen
	- relabel the 'load file' button to 'upload file'
	- relabel the first (bold) line of the helper to 'Upload an Excel or CSV
	file to import'

- Move all import options in a left panel

- Invert rows and columns in the mapping view
	- One row for each column (header) to map in the imported file
	- Columns: File Column / Odoo Field / If a match cannot be found / Comments

Left panel details
------------------

The sidepanel to the left of the mapping view is composed as followed:
- Imported File
	- Import file name
	- Sheet: Dropdown to select the sheet in the file if there are more than 1.
	- 'use first row as header' checkbox: like the existing option, defines
	whether the first row of the file should be considered as the header.
	True by default. If False, display the first value only under 'File Column'
	(see above)
- Formatting: only available if the file is a .csv
	- list every option that already existed in previous import implementation.
- Batch Import
	- is only available in debug moide and if the file exceeds the batch limit
	- Batch limit : 2000
	- Allow to define the threshold and the batch size.
- Help
	- Download Template link
	- Go to FAQ link
- Advanced
	- Track history during import. (same as existing functionallity)
	- Show Fields of relation fields. (same as existing functionallity)

Columns details
---------------

- File Column:
	- Contains the column headers of the import file and the first non-empty
	value of the column as a 'subtitle' (in grey italic)
	- If 'use first row as header' is False, only display the first column value
	(without the grey italic)

- Odoo Field:
	- Contains dropdowns to the fields of the current model
	- Required fields are diplayed in bold in the dropdown
	- Icon: In the front of the field label, the field type (char, many2one,..)
	is indicated by an icon.
	- Tooltip: when hovering a field, display a tooltip with the following
	information: field label, technical name, type, related model (if any)
	- Placeholder: If no field is selected, display the placeholder
	'To import, select a field...' in text-warning bold (i.e. orange)
	- Allow clear: there is a close (fa-times) icon at the end of the dropdown
	body to remove the Odoo field (and prevent import of this column)
	- Multi Mapping: when setting a field already matched to another column,
	unset it from the 'old' column (NB: exception made for char/text fields)

- Comments:
	- Contains feedback from the system to the user, either import
	errors/warnings or any additional details (multi mapping comments,...)
	- For many2many field, display in related comment cell the following:
	"To import multiple values, separate them by a comma".
	- If there is an import error for a specific field, the error will be
	displayed in the comment cell of the related field.
	- If there is a mapping error, mapping options are displayed under the
	error div to let the user choose the best option to fix that error.

Import errors management
------------------------

- Errors/warnings that can be matched to a specific field are displayed as
alert-danger/warning in the corresponding 'Comments' cell of the field.
- For 'no match found' errors, if X values couldn't be found,
display an unique error box will all the errors.
	- Values beyond the first three are folded under a 'More' button.
	- No changes on the 'See possible values' button.
- If an unmatched value is present in one row only, display :
	'<value> at row X (<name of the row if the name field is matched>)'
- If an unmatched value is multiple rows, display:
	'<value> at multiple rows'
- Errors/warnings/infos that cannot be matched to specific field are displayed
above the mapping listview. Global errors are not regrouped by error types to
avoid too much code complexy just to handle the rare times where multiple errors
of same types cannot be linked to a specific mapped field.

BaseImportError class have been introduced to ease the formatting of the various
exceptions that can occur during the import.

After testing the import, display the following above the mapping table:
- No warning/error:
	'Everything seems valid' (alert-info)
- At least one error:
	'The file contains blocking errors (see below)' (alert-error)
- At least one warning but no error:
	'The file contains non-blocking warning (see below)' (alert-warning)

Mapping options
---------------

Mapping options are only displayed after testing or importing the file,
if there are import errors. The goal is to have a clean interface and to guide
the user step by step. The import process can therefore be a bit longer as it
needs to import -> choose solution for errors -> re-import but that is easier
for users to learn and understand this reworked import tool.

Possible values when a value cannot be matched:
- For many2one / many2many fields:
	- Prevent import: (selected by default) not finding a match is blocking the
	import
	- Skip unknown values: values that cannot be matched will be skipped.
	(hidden if the field is required)
	- Create new values: Create records for values that cannot be matched
- For selection fields:
	- Prevent import (selected by default)
	- Skip unknown values: (hidden if the field is required)
	- Set to <first value>: if cannot be matched, set it to <first value>
	- Set to <second value>
	- Set to <third value>
	- etc.
- For boolean fields:
	- Prevent Import (selected by default)
	- Set to True
	- Set to False
	Note: With this rework, boolean warnings where the system assumes the
	replacement value in case of matching error is removed and is replaced by a
	blocking error. The user now has to choose the value to set.

"Prevent import" is the default behaviour when testing or importing.

Automatic mapping proposal
---------------------------

When loading a file, an automated mapping is directly proposed to the user,
based on word distance (see below), and on mapping created on previous
imports.
- Priority is given for mapping created on previous imports, skip fuzzy mapping.
	- a distance of -1 is used to ensure priority during duplicates removal .
	- In case multiple headers are mapped on the same field, if the mapped field
	is already taken by another header, use fuzzy mapping instead.
- If no previous mapping for that header on that model, try an exact match on
every field id and field name of the model. (distance = 0)
- If no match is found, fuzzy mapping is applied (word distance).
The fuzzy mapping is executed only on the most likely fields (see below).
- Remove duplicates: keep the header-field couple that has the smallest
distance. In case of equality, keep the first.

Automatic mapping is therefore optimised for previous mapping or exact match,
as fuzzy mapping requires heaviest treatment. This is intended to prioritise the
import of files based on import templates.

Most likely fields
------------------

When parsing the import file, each header is analysed to guess what type of data
the column contains. For example, ff the column contains float, we suppose that
that header will most likely be matched on float or monetary fields.
The most likely fields are a subset of the model's fields that match the
header types.

Most likely fields are used for the fuzzy mapping, to propose the user a field
mapping based on the header types, if an exact match could not be found.

Most likely fields are also listed under "Suggested fields" in the mapping
dropdown in "Odoo Field" column of the import tool.
For now on, every fields (no mather their type) can be matched to any header,
but we prioritise the most likely fields. This way, we don't constrain the
mapping possibility based on what we suppose the user would do, but we instead
guide suggest the user the most likely mapping solutions.

Word distance mapping
----------------------------

In order to improve the mapping configuration, if an exact match cannot be found
between the file column and one of the odoo field:
- Use Word distance:
	Word distance return a indice between 0 and 1.
		- 0: exact match
		- 1: completely different
	- A: First try on field['name']
	- B: Then on field['string']
	- Keep the minimal distance between A and B for each odoo_field
	- C: Keep the field that has the minimal distance.
	- Match the column to the field by default if C['distance'] < 0.3.
	Note: 0.3 has been chosen to ensure proximity but still having a little
	error margin.

Multi mapping
-------------

When multiple file columns are matched to the same char/text/many2many field,
display the following alert-info box in the "Comments" column of the related
fields:

	"Those columns will be concatenated in field <field label>"

Multi mapping rule :
	- If it is a char field, separate the concatenated values by a space
	- If it is a text field, separate the concatenated values by a line break
	- If it is a many2many field, separate the concatenated values by a comma

Various improvements
--------------------

- During Import (or Test), the first waiting message displayed has been modified
to 'Importing...' or 'Testing...'. The progress (x record imported/tested) is
only displayed after the first batch of record have been imported/tested.
- Ease matching for Selection field: use case insensitive comparaison instead
of exact match.
- Add filename to import wizard.
- Add placeholder to search input of field mapping dropdown.

Tests have been adapted accordingly.

Links
=====

Task ID: 2352241

closes odoo/odoo#61948

Related: odoo/enterprise#17246
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2021-04-02 11:17:31 +00:00
Raphael Collet c1279cd52a [FIX] test_impex: cursor cache in tests
As the same cursor is used by all tests, make sure to clear its `cache`
upon setup.
2020-11-24 13:23:32 +00:00
Anh Thao Pham (pta) 7095a651a2 [IMP] test_impex: test for import with create enabled for m2o in o2m
closes odoo/odoo#54740

closes odoo/odoo#55267

X-original-commit: 9bf32f7be1402cd112d1b7729da242f7b1d3b991
Signed-off-by: Anh Thao PHAM <kitan191@users.noreply.github.com>
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
2020-07-31 14:44:58 +00:00
Xavier Morel ef709fd92b [IMP] core, base: avoid multiple xids when importing records w/ inherits
Before this change, we create an xid for every parent of a record
being imported, regardless of whether it already has an xid, or if
it's being created implicitly through the child.

This generates unnecessary extra xids on pre-existing objects
e.g. update 5 product variants -> the product gets 5 new xids despite
already having one.

We should *only* set a xid on parent records which are being
implicitly created by the creation of a child with a specified
xid. That is, we should never set a xid on the parent if it exists
before the child is created.

Update _process_end to try and see if "non-loaded" xids correspond to
an automatically generated "parent" xid: we're still setting a xid on
implicitly created records (if the child is created with a xid) so
they're properly removed if e.g. the module is uninstalled, but
because we're only doing so at creation these xids will not be visited
during update and _process_end will try to delete them.

A special case can be added to check that "unknown" parent xids don't
have children which _inherit them, in which case we want to protect them.

Task 2251039

closes odoo/odoo#53283

Related: odoo/enterprise#12023
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-07-27 06:32:59 +00:00
Xavier Morel 6b2dfcc8b0 [IMP] core: batching during import
By default importing is batched (or at least attempted to), however
because the import is functionally sequential it's possible for a
record to refer to a previously imported record of the same batch.

When records are referenced by xid this is easy, we can just keep
track of all the xids we *should* have created so far and when trying
to look up a xid if it's one of those flush all pending
creations (otherwise assume it's some other unrelated xid which
already exists in the system).

For "name_search" lookups however this is more complicated, so we'd
just flush the entire thing.

This, then, is an issue when importing records with an m2o which
is *generally* looked up by "name" rather than xid e.g. when importing
partners it's more likely the "state" will be specified by
name (e.g. AZ or Arizona) than xid (base.state_us_3 lol), because it
means more or less every line will be flushed, defeating all batching
and slowing down the import.

To fix this, allow passing *model names* to flush (not just xids), and
try to collate all the models we could be creating (or updating)
records for in the process of importing. If we see a name_search on
one of these models flush, otherwise don't because we couldn't have
impacted the search.

Task 1951307

While at it, f2a1618758 removed all
usage of model_load_save, but left the creation of the savepoint
itself.

The InternalError check should be left in place to catch things like
conversion issues which put the tnx in an invalid state (only invalid
states should trigger an InternalError according to psycopg's errors
documentation).

closes odoo/odoo#49289

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-05-28 11:40:35 +00:00
Jinal PatelandMohammed Shekha 8f8c93b68a [IMP] base: display full name of the field in import warning
Before this change, if an error occurs while trying to match / import
a sub-field e.g. order_line/product_id, only the name of the top-level
field is showing when displaying the error e.g. "Order Line", which
lacks precision and makes understanding and fixing the issue more
complicated.

After this change, the entire field path should be displayed, e.g.
"Order Line / Product" in the example above.

Task-1906700

closes odoo/odoo#33031

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Co-authored-by: Mohammed Shekha <msh@odoo.com>
2020-03-23 11:51:30 +00:00
Xavier Morel 012678bf27 [IMP] core: cutoff imports after some number of errors
Importing 100 records and getting 100 errors (often the same every
time because a required field was not mapped) is not super useful and
spams the logs a lot.

Cutting off at one point seems useful.

Note: integrates warnings-related fix from #47972 reported by Grzegorz
Marczyński in #47936.

closes odoo/odoo#48095

X-original-commit: 6b05f1d13c862dd66e89f35607d072315be80995
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-03-20 10:29:33 +00:00
Martin Trigaux 49fbab5329 [IMP] base: split and deprecate load_lang
load_lang was a kind of hybrid method trying to active or creating a
language if not found. This was error prone.
Instead rely on two methods with clear purpose:
ResLang._create_lang(lang, lang_name=None)
  - create a new res.lang entry using the locale of the server
    return the res.lang record to match the API of _activate_lang

ResLang._active_lang(code)
  - activate the given code lang

Most of the time, _active_lang is what is expected

tools.trans_load_data and IrTranslation._load_module_terms no longer
activate the language if not active.
Loading the translations should be explicit on an activated language,
it is too error prone to silently activate/create a language if not
found.
Remove lang_name from trans_load_data as no longer needed.
2019-11-19 10:37:01 +01:00
Yannick Tivisse 4aa5ba35af [IMP] base/test_*: Adapt tests to work with/without demo data 2019-11-05 13:08:02 +01:00
Wolfgang Taferner 5ac04570f1 [ADD] base: allow importing new O2M values with an xid
When importing an O2M value (sub-record), rather than raising an error
if the value has a xid set (foo_ids/id) and the xid does not exist in
the database, re-insert the xid as the "id" field of the sub-record,
then re-extract it at creation (the ORM will have ignored it) and
associate it with the newly created record.

Advantages:

* keeps o2m creation in the proper order during import (sub-records
  remain created after the parent) meaning o2ms with a required parent
  can take advantage of the feature
* avoids breaking batching when creating the objects
* no new command or smuggling through an existing command

closes odoo/odoo#35995

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2019-11-04 08:11:51 +00:00
Xavier Morel f58368210c [ADD] base_import: batching (& some other features)
various UI changes
------------------

* renamed "test import" button to "test"
* move relation fields thing to debug mode
* remove "Defer parent/child computation" option as it was deprecated
  / removed from the backend in
  80f1ac3599, turns out this checkbox
  existed inactive for longer than it's been of any use (added in
  68cb2ade09 on 2017-11-29, made
  non-operating on 2018-01-24, that's so sad)

batching
--------

* add support for batching imports (skip & limit parameters)
* modify client to use batched imports & properly adapt responses so
  it still looks like a single import for the client (more or less)
  e.g. update row numbers in error messages, etc...
* properly handle partial imports though
* disable usual loading throbber to have a single progress
  notification displayed continuously throughout all the batches: the
  normal throbber only shows after 3s of waiting for an RPC response,
  so it would keep flashing in and out (appear 3s into a batch's
  import then disappear at the end only to reappear 3s into the next
  batch's loading)

NOTE: the limit is row-wise. If a record straddles the limit (because
of nested O2M records), the record is imported in full and the "next
row" is whatever row follows the record. This means a limit of 10 can
lead to an import of 17 lines, and as the progress indicator is in
records# the increments can jump around.

Task 2059448
2019-09-13 09:45:29 +00:00
Yannick Tivisse 58b3557d6d [IMP] base_import: Improve import general UX (back2basics)
Purpose
=======

Improve error management UX on import.

TaskID: 2049992

closes odoo/odoo#35804

Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
2019-08-20 12:23:08 +00:00
Raphael Collet 9920f20e4c [IMP] models: ORM speedup
This branch is the combination of several optimizations in the ORM:

* store field values once in the cache: the cache reflects more
faithfully the database, only fields that explicitly depend on the
context have an extra indirection in the cache;

* delay recomputations by default: use method `recompute` to explicitly
flush out pending recomputations;

* delay updates in method `write`: updates are stored in a data
structure that can be flushed efficiently to the database with method
`flush` (which also flush out recomputations);

* make method `modified` take advantage of inverse fields to inverse
dependencies;

* filter records by evaluating a domain on records in Python;

* a computed field with `readonly=False` behaves like a normal field
with an onchange method;

* computed fields are computed in superuser mode by default.

Work done by Toufik Ben Jaa, Raphael Collet, Denis Ledoux and Fabien
Pinckaers.

closes odoo/odoo#35659

Signed-off-by: Denis Ledoux <beledouxdenis@users.noreply.github.com>
2019-08-20 12:43:59 +00:00
Martin TrigauxandRaphaël Collet 7593b887df [REF] fields: use ir.model.fields.selection
The selection values of a selection field are now stored in database in the
model ir.model.fields.selection

This will allow to have a modular approche on selections and each selection
is now linked to the module that declared it.
Previously to this change, the selections were linked to the field, meaning
uninstalling a module had no impact on the selections stored on database.

With this change, the selections will now be translated in the correct module
(having an external id) and the records having a used selection will now be
reset to null.

closes odoo/odoo#30228

Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>


Co-authored-by: Raphaël Collet <rco@odoo.com>
2019-08-19 11:44:34 +00:00
Sébastien Theys 1e9772889b [IMP] *: use image.mixin when appropriate
The following models are already using big images, or they might need big images
in the future:

- partner
- hr employee
- shop category
- lunch product
- gamification badge and karma rank

PR: #34925
2019-08-02 16:47:58 +00:00
Lucas LefèvreandYannick Tivisse 3e97cff779 [IMP] base: Remove customer and supplier fields from res.partner
Purpose
=======

Fields `customer` and `supplier` on `res.partner`
are mostly used in domains of many2x fields.

Those domains can confuse end users because they don't
see the partner they are looking for; and it's not obvious why.

Some identified problems:

1. It can lead to duplicated partners: the user does not find
   the partner, so he creates a new one.

2. The user imports supplier contacts in the Contacts app, so they
   don't get the `supplier` flag. Then the user wants to make a purchase order,
   and cannot find the new suppliers in the list

3. A user removes the customer flag on a prospect, because they don't think
   it's a customer yet - except now they can't make a quote for that customer...

Specification
=============

Remove the two mentioned fields.

Since fields `customer` and `supplier` have been removed, all partners
are now shown in many2one dropdowns.

But in some cases, not all partners are relevant or some are more likely
to be relevant than others. e.g. when creating a PO, top suppliers have a
higher priority than other partners.

So, adapt the places where those fields were used with the new mechanism to
display the searched the partners, according to the number purchase/sales
orders they made.

TaskID: 2031147

Co-authored-by: Yannick Tivisse <yti@odoo.com>
2019-08-01 12:42:03 +02:00
Hiral Bhavsar 3cd7ed07a2 [IMP] *: remove 'view_type' on window actions.
The old tree views don't really exist anymore, this odd pseudo-flag to
dispatch between "list" and "tree" tree views has no reason to remain.

Task 1937686

closes odoo/odoo#31243

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2019-06-17 11:34:17 +00:00
Xavier Morel a0e05e2ab9 [IMP] fields: selection fields only use strings
closes odoo/odoo#29039
2019-01-26 14:25:44 +00:00
TWA 983c7e8152 [IMP] mail,crm,...: Replace occurences of name_get()[0][1] by display_name
Purpose
=======

display_name is equal to doing name_get()[0][1]

Replacing name_get()[0][1] by display_name could be good for 2 things:
- Uniformization of the code
- Probable optimization if name_get is used inside a for loop on a recordset (as the values will be prefetched and put in the cache).

TaskID: 1849250

closes odoo/odoo#26341
2018-10-09 11:28:00 +00:00
Mohammed Shekha 8fbadb9575 [ADD] base_import: debug option to create M2O/M2M records
Adds a checkbox to import columns (in debug mode) allowing a user to
create records M2O and M2M records not found (via name_search).

Task ID: 1850633

* uses a context key to avoid altering basically all the import callstack
* attempted to lift the creation in the `_str_to_*` functions and create
  m2m via commands, but that doesn't really work out
2018-08-16 11:10:35 +02:00
Xavier Morel 24a8622b6a [CHG] base_import: UI/UX changes
Increased amount of magical autodetection of import
parameters (formats, delimiters, ...)

Task 1870663
2018-08-10 15:47:56 +02:00
Raphael Collet 960360afe4 [REF] *: use native date/datetime for Date/Datetime fields
From this commit onwards, Date fields will return datetime.date objects and Datetime fields will return datetime.datetime objects, this implies a number of things that are clearly explained both in the ORM API for master.

This commit also introduces a number of helper functions for dates and datetimes that are exposed in tools.date_utils and fields.Date[time], explained in the documentation as well.

Task-ID: 47189
2018-08-06 14:37:19 +02:00
Xavier Morel cb9c99b937 [IMP] base_import: parsing/cleanup of floats
* Handle a leading + in import data, some contexts (e.g. bank
  statements) will mark positive sums explicitly for clarity
* Add basic grouping/decimal separator inference for people importing
  data from many localisations or to avoid them *having* to configure
  their separators if we can handle it for them, currently very basic
2018-08-06 12:03:12 +02:00
Adrien Dieudonne bb1602b3ac [IMP] models: avoid empty string as module name
When you import a record in Odoo, you can specify an id without
prefixing it by the name of the module. e.g. 'project_project_x'
The result of this is an empty string as module name.
(This value is valid for a required field...)

Plus, when you check metadata, you see 'false.project_project_x'.

Now, the default module name is '__import__' instead of an empty string.
We already have similar behavior with '__export__' triggered by
'export_data' method.
2017-11-27 08:18:49 +01:00
Xavier Morel b93613066f [FIX] P3: JSON is an object<->text encoding
It doesn't load bytes, and it doesn't dump to bytes.
2017-08-20 23:25:54 +02:00
xmo-odoo ca74b5e1c8 [IMP] convert PG errors through pg93 diag info
Previously availability of meta-information (failing constraint,
impacted table and column, …) in pg errors (e.g. constraint check
failure) was limited and only available through formatted error
messages, which could be localised (by PG itself) so not did getting
that information require string extraction it was brittle in the face of
localised instances.

Postgres 9.3 adds this meta-information to error diagnostic data
(`PQresultErrorField`), allowing easy programmatic access to it.
Psycopg2 [added a Diagnostic
object](http://initd.org/psycopg/docs/extensions.html#psycopg2.extensions.Diagnostics)
around the time pg9.3 was itself released.

Assuming Odoo now requires pg >= 9.3 we can remove the old string
munging extraction for diagnostic.

* funny story, the handling of 23505 didn't actually work because
  "duplicate key value violates unique constraint" is a literal part
  of the error message, I'd misunderstood "value" as a field name
  because my test model's field was called "value". Makes sense too
  since you can set UNIQUE constraints on multiple fields (and UNIQUE
  indices on expressions)
* for both errors, it should be possible to use the table_name or
  constraint metadata to provide messages about sub-model issues (a
  constraint on an m2o), but that's not currently handled

Fixes #15323
2017-04-11 14:51:53 +02:00
Raphael Collet 0783318f97 [FIX] adapt logger names 2016-09-02 17:28:13 +02:00
Raphael Collet 9e64f9f951 [REF] openerp: move openerp to odoo 2016-09-02 17:28:12 +02:00