When a user tries to import the CSV file with a different separator at that
time, the values in mapper and rows_to_import are not correctly mapped. So
the traceback will be generated.
Steps to reproduce:
1. Click on import in the bank statement.
2. Select any CSV file for the bank statement line or can download and import
this file https://drive.google.com/file/d/1lnScw4RN6T01pOkyNON8vvb3FQOPiy1O/view?usp=drive_link
3. Select any separator other than a comma.
4. Click on the test or Import button.
5. Error will occur.
Error: IndexError: list index out of range.
To solve this issue, a row's length is checked with the
number of fields.
sentry-4021250095
closesodoo/odoo#134596
X-original-commit: 0eb30132c14420d42f88a5f54a81ba1dc51a867c
Signed-off-by: Achraf Ben Azzouz (abz) <abz@odoo.com>
Signed-off-by: Saurabh Choraria (sauc) <sauc@odoo.com>
Goal:
* Simplified modifiers to only have one way to define modifiers;
* Remove states attributes on python field;
* Use python expression in view `required`, `readonly`, `invisible`;
* More accurate validation of xml views.
This commit change the syntax to python expression. The next commit
will update/convert all xml views.
Before this commit:
* the `required`, `readonly` and `invisible` attributes can only have
values of `True`, `False`, 1, 0 or a python expression to use the
context;
* the `attrs` attribute define a dict. The key of this dict was
`required`, `readonly` and `invisible` and the values are the domain or
a string representing a domain to be evaluate as python expression.
This python expressions was evaluate by the javascript with view fields
and other contextual values as: context, uid, parent, active_id,
active_ids, active_model, allowed_company_ids, current_company_id.
* the `states` attribute in the view was a comma separated list of the
state. This list was combined with the `invisible` attribute;
* the `invisible` attribute on python field is used as default value;
* the `states` attribute on python field was dictionnary with state as
key and list of tuple. This structure was combined with `readonly` view
attribute.
* After combining, the resulting domains of the different attributes
`required`, `readonly` and `invisible` are evaluated with the values of
the fields. The `invisible` attributes is splitted into two use:
`invisible` and `column_invisible`.
After this commit:
* The attributes `required`, `readonly`, `invisible` and
`column_invisible` define python expression. This python expressions
are evaluate by the javascript with view fields and other contextual
values as: context, uid, parent, active_id, active_ids, active_model,
allowed_company_ids, current_company_id.
The domains can contains contextual value and will be evaluate by the
javascript.
```xml
<field name="field_a" readonly="not context.get('show_a')" attrs="{'readonly': [('field_b', '!=', False), ('field_c', '=', parent.c)]}"/>
<field name="field_b" states="draft"/>
```
will be replaced by
```xml
<field name="field_a" readonly="not context.get('show_a') or field_b and field_c == parent.c"/>
<field name="field_b" invisible="state != 'draft'"/>
```
Some inherited views will be modified differently in order to maintain
the previous behavior:
```xml
<field name="field_a" readonly="not context.get('show_a')" attrs="{'invisible': [('field_b', '!=', False)]}">
```
```xml
<field name="field_a" position="attributes">
<attribute name="attrs">{'readonly': [('field_c', '=', False)], 'invisible': [('field_d', '!=', '3')]}<attribute>
</field>
```
will be replaced by
```xml
<field name="field_a" readonly="not context.get('show_a')" invisible="field_b">
```
```xml
<field name="field_a" position="attributes">
<attribute name="readonly" add="(not field_c)" separator=" or "/>
<attribute name="invisible">field_d != 3<attribute>
</field>
```
Validation:
A stricter control is made on the level of the attributes (modifiers)
and the fields necessary for these. The use of the previous attributes
'attr' and 'states' triggers an error (these no longer exist after the
application of the migration script)
task-2495504
Part-of: odoo/odoo#104741
One of the main issue with ormcache is that the invalidation clears
everything, meaning that some value, slow to compute but with a long
lifetime, can be removed from the cache because an easy to invalidate
value is cleared, like after writting or creating a product has an
example.
Most example in the code will try to invalidate the cache of the models
doing something like `env['ir.qweb'].clear_caches()` but it is
finally equivalent to `env.registry.clear_cache()`, and cross worker.
The idea is to have multiple cache, maybe with specific sizes for a
specific purpose.
Having one per model is maybe a bad idea because it will be difficult
to size the LRU correcly, and it is too dynamic. Checking invalidation
may be expensive.
The proposed solution is closed allow a limited number of named caches,
using onse sequence per cache. This is actually close to the
cache_longterm.
We want to discourage using a specific cache for one use case in
the buisness code. Adding a cache shouldn't be something easy, doable
in stable.
Note that we could also change the invalisation mecanism using an
insert only table. We an check the sequence of this table, but also
fetch all invalidation messages.
Another possible improvement, especially if we have more than x cache is
to have a global sequence, checking signaling would mean to check the
main sequence, and only the other ones if the main one changed.
Note that this poc is inspired from the long term cache but not all
use case where applie yet.
Part-of: odoo/odoo#119813
When user tries to add non ascii characters in image field column in a file and
then when he tries to import it the error will occur.
Steps to reproduce:
1. Install contacts
2. export a contact and keep image field in fields to export.
3. Now in that file change the image data and add non ASCII characters
(for example: 'ô').
4. Now try to import this file in contacts.
5. The error will occur.
Applying this commit will fix this issue.
Currently, a ValueError is raised when we give non-ascii characters as input.
So I have reported an issue in python in which I have mentioned to edit the
documentation to note that it may raise ValueError for non-ascii content, or
to fix '_bytes_from_decode_data' function to raise binascii.Error instead of
ValueError.
To track the issue - https://github.com/python/cpython/issues/105193
sentry-4029823200
closesodoo/odoo#123572
X-original-commit: ceb67160e1747f0a886efb8ca0a60499e2b080d1
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Before this commit:
=====================
KeyError 'selection_values' that occur in base_import/_handle_fallback_values()
while importing a data file. If any field(s) is many2one or many2many and we
tried to import the value of that field(s) that is not created in the database.
In that case when we set the 'Prevent Import' option.
It will raise an error like KeyError: 'selection_values'.
After this commit:
=====================
Solved the issue when there is a many2one or many2many field(s) and the import
value of that field(s) which is not available in the database. also, the method
is only for the 'selection' field and 'boolean' field so the code works when
there is a selection field and their selection_values only.
sentry - 3958065223
closesodoo/odoo#115056
X-original-commit: 0ef930cd7598bd15227ff025d2662d4331f8e45b
Signed-off-by: Julien Castiaux (juc) <juc@odoo.com>
The main goal of this commit is to reduce the size of the registry by
removing the (almost) useless __last_update field.
Statistics # of fields with all modules installed:
before 30184 fields, 1299x last_update (4.30%)
Before this commit, the computed field __last_update was added on every model.
The idea behind this field was to have a computed field that had either
the write_date or the create_date if the write_date was empty. However,
the write_date is always written, even on creation, making it useless
to have the computed field __last_update
After this update, we completely remove from BaseModel:
* __last_update
* CONCURRENCY_CHECK_FIELD that was always defined as "__last_update"
* _compute_concurrency_field that was the compute function for __last_update
closesodoo/odoo#105739
Task-id: 3062140 (part of 3062137 improve registry load time)
Related: odoo/upgrade#4038
Related: odoo/enterprise#33939
Signed-off-by: Raphael Collet <rco@odoo.com>
Translated fields no longer use the model ir.translation. Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table. The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages). Empty text is allowed in translation values, but not NULL.
Here are examples for a field with translate=True:
NULL
{"en_US": "Foo"}
{"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
{"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}
Like before, writing False to the field makes it NULL, i.e., False in all
languages. However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.
Here are examples for a field with translate=xml_translate:
NULL
{"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}
Change for callable(translate) fields: one can now write any value in any
language on such a field. The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values. Basically the
structure of the value must remain the same in all languages, like before.
Reading a translated field is now both simpler and faster than the former
implementation. We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:
SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...
The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages). For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.
Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language. Before this change, only the source value of the
field ('en_US') could be indexed.
Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages. We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.
Code translations are no longer stored into the database. They become static,
and are extracted from the PO files when needed. The worker simply uses a cache
with extracted code translations for performance. This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker. Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).
Performance summary:
(+) reading 'model' translated fields is faster
(+) reading 'model_terms' translated fields is much faster (no need to inject
translations into the source value)
(+) searching translated fields with operator 'ilike' is much faster when the
field is indexed with 'trigram'
(+) updating translated fields requires less ORM flushing
(-) importing translations from PO files is 2x slower
Some extra fixes:
- make field 'name' of ir.actions.actions translated; because of the PG
inheritance, this is necessary to make the column definition consistent in
all models that inherit from ir.actions.actions.
- add some backend API for the web/website client for editing translations
- move methods get_field_string() to model ir.model.fields
- move _load_module_terms to model ir.module.module
- adapt tests in test_impex, test_new_api
- because env.lang is injected into SQL queries, its returned value is
now guaranteed to correspond to a valid active language or None
- remove wizard to insert missing translations (no longer makes sense)
task-id: 2081307
Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
How to reproduce the bug ?
- install the inventory app
- open the app and try to import a file where at least one cell has a
wrong value
What is the bug ?
When you try to import a file in the inventory app where at least one
cell has a wrong value, the file won't be imported. In addition to that,
you will get an error saying that one python module is missing while
this might not be the case.
opw-2715625
closesodoo/odoo#91482
X-original-commit: cafd8699ff8ec8eb93aa283a4cac57317ce40708
Signed-off-by: Adrien Minet <admi@odoo.com>
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Signed-off-by: Simon Goffin <sig@odoo.com>
Purpose
=======
Since v15, the time to import records in Odoo is much slower than in
v14. With a profiling, we know that the cause is `fields_get`. Indeed,
this method is heavy and call many times (#rows * #columns * ~#batches).
So the import time (in seconds) for ~1000 leads in CRM is;
Batch size | v14 | v15 | v15 fix
--------------------------------
2000 | 7 | 20 | 14
200 | 11 | 48 | 15
20 | 17 | 423 | 23
We can see that just by calling `._fields.get` instead of `fields_get`
we gain a lot of time. The reason for that is `fields_get` just read in
a python dictionary (which is really really fast), while `fields_get`
checks the access right, get the description of each fields (which
might make SQL queries to retrieve translation, etc).
Task-2687407
X-original-commit: 96e72a8b1657df00d5ed7ca6db7bd99e352d7096
Part-of: odoo/odoo#79913
PURPOSE
Improve import wizard as followup of recent improvements.
SPECIFICATIONS
Preview lines
- make the preview lines more visible
- set font-weight: 500 on o_import_header_name
- remove the text-muted class from the data preview line
- allow the user to preview the first 5 non-empty values
- when hovering on the data preview span, display a tooltip with the first
5 non-empty values of the file column (with 'Preview' as the tooltip title)
- In the 'when a value cannot be matched' dropdown:
add a new 'skip record' option for the following field types: boolean,
many2one, many2many, selection
add a new 'set empty' for the following field types: many2one, many2many,
selection hide this option if the field is required
When a value cannot be matched:
the 'skip record' option will make sure that lines with an unmatched value
will not be imported (i.e. skipped at import)
the 'set empty' option will set the field value to False when the value
cannot be matched
- Restore colored background on alert boxes by removing some css rules
Task-2504343
closesodoo/odoo#73713
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Co-authored-by: Mohammed Shekha <msh@odoo.com>
Purpose
=======
Propose an import tool that is more intuitive and allows more import options to
ease the import.
Specifications
==============
Update upload import file screen
--------------------------------
- Relabel import welcome screen
- relabel the 'load file' button to 'upload file'
- relabel the first (bold) line of the helper to 'Upload an Excel or CSV
file to import'
- Move all import options in a left panel
- Invert rows and columns in the mapping view
- One row for each column (header) to map in the imported file
- Columns: File Column / Odoo Field / If a match cannot be found / Comments
Left panel details
------------------
The sidepanel to the left of the mapping view is composed as followed:
- Imported File
- Import file name
- Sheet: Dropdown to select the sheet in the file if there are more than 1.
- 'use first row as header' checkbox: like the existing option, defines
whether the first row of the file should be considered as the header.
True by default. If False, display the first value only under 'File Column'
(see above)
- Formatting: only available if the file is a .csv
- list every option that already existed in previous import implementation.
- Batch Import
- is only available in debug moide and if the file exceeds the batch limit
- Batch limit : 2000
- Allow to define the threshold and the batch size.
- Help
- Download Template link
- Go to FAQ link
- Advanced
- Track history during import. (same as existing functionallity)
- Show Fields of relation fields. (same as existing functionallity)
Columns details
---------------
- File Column:
- Contains the column headers of the import file and the first non-empty
value of the column as a 'subtitle' (in grey italic)
- If 'use first row as header' is False, only display the first column value
(without the grey italic)
- Odoo Field:
- Contains dropdowns to the fields of the current model
- Required fields are diplayed in bold in the dropdown
- Icon: In the front of the field label, the field type (char, many2one,..)
is indicated by an icon.
- Tooltip: when hovering a field, display a tooltip with the following
information: field label, technical name, type, related model (if any)
- Placeholder: If no field is selected, display the placeholder
'To import, select a field...' in text-warning bold (i.e. orange)
- Allow clear: there is a close (fa-times) icon at the end of the dropdown
body to remove the Odoo field (and prevent import of this column)
- Multi Mapping: when setting a field already matched to another column,
unset it from the 'old' column (NB: exception made for char/text fields)
- Comments:
- Contains feedback from the system to the user, either import
errors/warnings or any additional details (multi mapping comments,...)
- For many2many field, display in related comment cell the following:
"To import multiple values, separate them by a comma".
- If there is an import error for a specific field, the error will be
displayed in the comment cell of the related field.
- If there is a mapping error, mapping options are displayed under the
error div to let the user choose the best option to fix that error.
Import errors management
------------------------
- Errors/warnings that can be matched to a specific field are displayed as
alert-danger/warning in the corresponding 'Comments' cell of the field.
- For 'no match found' errors, if X values couldn't be found,
display an unique error box will all the errors.
- Values beyond the first three are folded under a 'More' button.
- No changes on the 'See possible values' button.
- If an unmatched value is present in one row only, display :
'<value> at row X (<name of the row if the name field is matched>)'
- If an unmatched value is multiple rows, display:
'<value> at multiple rows'
- Errors/warnings/infos that cannot be matched to specific field are displayed
above the mapping listview. Global errors are not regrouped by error types to
avoid too much code complexy just to handle the rare times where multiple errors
of same types cannot be linked to a specific mapped field.
BaseImportError class have been introduced to ease the formatting of the various
exceptions that can occur during the import.
After testing the import, display the following above the mapping table:
- No warning/error:
'Everything seems valid' (alert-info)
- At least one error:
'The file contains blocking errors (see below)' (alert-error)
- At least one warning but no error:
'The file contains non-blocking warning (see below)' (alert-warning)
Mapping options
---------------
Mapping options are only displayed after testing or importing the file,
if there are import errors. The goal is to have a clean interface and to guide
the user step by step. The import process can therefore be a bit longer as it
needs to import -> choose solution for errors -> re-import but that is easier
for users to learn and understand this reworked import tool.
Possible values when a value cannot be matched:
- For many2one / many2many fields:
- Prevent import: (selected by default) not finding a match is blocking the
import
- Skip unknown values: values that cannot be matched will be skipped.
(hidden if the field is required)
- Create new values: Create records for values that cannot be matched
- For selection fields:
- Prevent import (selected by default)
- Skip unknown values: (hidden if the field is required)
- Set to <first value>: if cannot be matched, set it to <first value>
- Set to <second value>
- Set to <third value>
- etc.
- For boolean fields:
- Prevent Import (selected by default)
- Set to True
- Set to False
Note: With this rework, boolean warnings where the system assumes the
replacement value in case of matching error is removed and is replaced by a
blocking error. The user now has to choose the value to set.
"Prevent import" is the default behaviour when testing or importing.
Automatic mapping proposal
---------------------------
When loading a file, an automated mapping is directly proposed to the user,
based on word distance (see below), and on mapping created on previous
imports.
- Priority is given for mapping created on previous imports, skip fuzzy mapping.
- a distance of -1 is used to ensure priority during duplicates removal .
- In case multiple headers are mapped on the same field, if the mapped field
is already taken by another header, use fuzzy mapping instead.
- If no previous mapping for that header on that model, try an exact match on
every field id and field name of the model. (distance = 0)
- If no match is found, fuzzy mapping is applied (word distance).
The fuzzy mapping is executed only on the most likely fields (see below).
- Remove duplicates: keep the header-field couple that has the smallest
distance. In case of equality, keep the first.
Automatic mapping is therefore optimised for previous mapping or exact match,
as fuzzy mapping requires heaviest treatment. This is intended to prioritise the
import of files based on import templates.
Most likely fields
------------------
When parsing the import file, each header is analysed to guess what type of data
the column contains. For example, ff the column contains float, we suppose that
that header will most likely be matched on float or monetary fields.
The most likely fields are a subset of the model's fields that match the
header types.
Most likely fields are used for the fuzzy mapping, to propose the user a field
mapping based on the header types, if an exact match could not be found.
Most likely fields are also listed under "Suggested fields" in the mapping
dropdown in "Odoo Field" column of the import tool.
For now on, every fields (no mather their type) can be matched to any header,
but we prioritise the most likely fields. This way, we don't constrain the
mapping possibility based on what we suppose the user would do, but we instead
guide suggest the user the most likely mapping solutions.
Word distance mapping
----------------------------
In order to improve the mapping configuration, if an exact match cannot be found
between the file column and one of the odoo field:
- Use Word distance:
Word distance return a indice between 0 and 1.
- 0: exact match
- 1: completely different
- A: First try on field['name']
- B: Then on field['string']
- Keep the minimal distance between A and B for each odoo_field
- C: Keep the field that has the minimal distance.
- Match the column to the field by default if C['distance'] < 0.3.
Note: 0.3 has been chosen to ensure proximity but still having a little
error margin.
Multi mapping
-------------
When multiple file columns are matched to the same char/text/many2many field,
display the following alert-info box in the "Comments" column of the related
fields:
"Those columns will be concatenated in field <field label>"
Multi mapping rule :
- If it is a char field, separate the concatenated values by a space
- If it is a text field, separate the concatenated values by a line break
- If it is a many2many field, separate the concatenated values by a comma
Various improvements
--------------------
- During Import (or Test), the first waiting message displayed has been modified
to 'Importing...' or 'Testing...'. The progress (x record imported/tested) is
only displayed after the first batch of record have been imported/tested.
- Ease matching for Selection field: use case insensitive comparaison instead
of exact match.
- Add filename to import wizard.
- Add placeholder to search input of field mapping dropdown.
Tests have been adapted accordingly.
Links
=====
Task ID: 2352241
closesodoo/odoo#61948
Related: odoo/enterprise#17246
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
The German date format is the following: DD.MM.YYYY but when importing
such value, it is wrongly considered as a float value due to the '.'
being confused with the thousands separator in the regexp of method
_remove_currency_symbol.
We should check both date/time and float/monetary formats, so it will
correctly identify the German date format as well as float values.
closesodoo/odoo#62205
X-original-commit: 1bdd0cc247d36c869ffdee7d501cb65d36be47a6
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
Signed-off-by: Alex Tuyls <alt-odoo@users.noreply.github.com>
As part of QoL PR #26267 the way the recursion limit was
managed (where it is checked) was changed, effectively decreasing the
recursion limit by 1 (to the originally expected limit of 2).
According to #29877 this is an inconvenient change / regression in
e.g. accounting context: if an object has journal entries, updating a
field of a journal item requires 3 levels of indirection (entries ->
items -> field). The same would occur if an object is e.g. linked to
multiple projects (projects -> tasks -> field) or SO (-> lines ->
field).
Therefore bump the recursion limit up to 3.
Closes#29877closesodoo/odoo#61997
X-original-commit: 64ff41ce805c72eb700b45d55adfc7379de56862
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Column mappings are updated in-place, if multiple users are importing records
of the same model at the same time, this will trigger concurrency errors.
This is made worse by the error only being reported on commit (after having
processed the entire import) and being retried automatically, so it slows down
the user and the entire system, the more concurrent imports the slower.
Log except:
INFO dbname odoo.addons.base_import.models.base_import: done
ERROR dbname odoo.sql_db: bad query: UPDATE "base_import_mapping" SET "field_name"='name',"write_uid"=%s,"write_date"=(now() at time zone 'UTC') WHERE id IN (%s)
ERROR: could not serialize access due to concurrent update
INFO dbname odoo.service.model: SERIALIZATION_FAILURE, retry 1/5 in 0.8720 sec...
closesodoo/odoo#57655
X-original-commit: 3f5e9ca625b53021c100245f5786bda4069974ad
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit fixes all issues detected by the new pylint
gettext-variable test.
It converts some calls to the new syntax
_("Foo %s", bar)
to progressively migrate the code to the new syntax.
A few calls were not technically incorrect but still detected by the
linter.
_("Foo" +
"Bar")
has been converted to
_("Foo"
"Bar")
as it has the same effect and make sure the argument is of type
asteroid.Const instead of BinOp).
closesodoo/odoo#53683
Related: odoo/enterprise#11467
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Using a few regex like
\((_\(.*%s.*)(\) % )([\w\[\]][\w .\[\]\(\)'"]*)\)
($1, $3))
Old syntax is still compatible but starts the migration to the new
syntax that catches error.
Steps to reproduce:
-install sales
-try to import a product file with volumes set to scientific notation
(9.2e-05 for example)
Previous behavior:
scientific notation is not recognized by the base_import module
and raises a small warning
Current behavior:
scientific notation is converted to decimal notation on the fly
when possible
opw-2162353
closesodoo/odoo#42591
X-original-commit: 9acd76c5d116abd07207af3b548658ad983ff82d
Signed-off-by: Jorge Pinna Puissant (jpp) <jpp@odoo.com>
Excel can store data in multiple tables, each is called a sheet. Odoo
was only importing the first sheet making the process to import a file
containing multiple sheets cumbersome.
It is now possible to select the sheet in the import options. By default
the first sheet is selected.
closesodoo/odoo#40728
Task: 2043768
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Test import of a file with xml ids.
At the first dry run, the xml id X is associated with id R for each such record.
If some records refer to X via a relational field, then in SQL it reduces
to a query using R as id; but R does not exist since it was a dry run.
As a result, subsequent runs fail.
The cache should be cleared after a dry run, since the xml ids are not reliable.
opw 2068446
closesodoo/odoo#37396
X-original-commit: 1b35294d7b8d07f373eca24977e2952b9cfb2c7d
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
* If the item is not an acceptable URL, check if it's base64 and
return an error if it's not, otherwise it just ends up blowing later
when the content of the field is assumed to be base64 and explodes
* The URL having to contain png or jpg or tiff or gif or bmp doesn't
make much sense (especially in this modern world where
placekitten.com doesn't add extensions, not to mention this doesn't
actually check for extensions). After discussing with odo it doesn't
seem to have any security impact, the only thing this filter does is
annoy people for no reason.
various UI changes
------------------
* renamed "test import" button to "test"
* move relation fields thing to debug mode
* remove "Defer parent/child computation" option as it was deprecated
/ removed from the backend in
80f1ac3599, turns out this checkbox
existed inactive for longer than it's been of any use (added in
68cb2ade09 on 2017-11-29, made
non-operating on 2018-01-24, that's so sad)
batching
--------
* add support for batching imports (skip & limit parameters)
* modify client to use batched imports & properly adapt responses so
it still looks like a single import for the client (more or less)
e.g. update row numbers in error messages, etc...
* properly handle partial imports though
* disable usual loading throbber to have a single progress
notification displayed continuously throughout all the batches: the
normal throbber only shows after 3s of waiting for an RPC response,
so it would keep flashing in and out (appear 3s into a batch's
import then disappear at the end only to reappear 3s into the next
batch's loading)
NOTE: the limit is row-wise. If a record straddles the limit (because
of nested O2M records), the record is imported in full and the "next
row" is whatever row follows the record. This means a limit of 10 can
lead to an import of 17 lines, and as the progress indicator is in
records# the increments can jump around.
Task 2059448
- Change the date format of the users language to "%b %d, %Y"
- Export any res.partner as .csv and include at least 'Display Name' in
the fields to Export
- Import the file exported
The import fail with error: "Import preview error failed due to: 'b'".
This was introduced with commit 32c2666d18 where the format of the
user was added in the list of patterns to match. Since we use a limited
version of TimeRE (see Python module `_strptime`), only the patterns
defined in `_P_TO_RE` are supported. If an unsupported format is used, a
`KeyError` is raised.
We simply skip the pattern if it cannot be parsed. Another solution
would be to instanciate TimeRE, but that would also require to set the
locale correctly. Long story short: things will start to get messy for a
nice-to-have feature, a.k.a being able to import dates such as
`January 1, 2019`.
Fixes#35868
opw-2055971
opw-2055140
closesodoo/odoo#35883
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
Multi is the default api for methods, it is not necessary to explicitly
decorate methods with it, adds clutter and most people use it because
they see that the rest of the code uses it.
Done with `find . -type f -name '*.py' | xargs sed -i '/@api.multi/d'`
Some distributions still bundle chardet 2.3, which have some guessing
divergences / incompatibilities with python (resolved in 3.x):
1. UTF-{16,32} with BOM is guessed as LE/BE, which when used to decode
the string doesn't strip out the BOM. Handle this by checking if
the BOM is present and converting the encoding name to the
non-marked version in that case.
2. The ISO-8859-1 test string is guessed as ISO-8859-2 (TBF the
decoding does make some sense). Allow multiple targets/guesses to
"fix" that.
closesodoo/odoo#33179
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When an import failed because of a specific cell, the error was
Error cell found while reading XLS/XLSX file: #N/A
Add the line and column index in the error for better debugging
Translate error message
Cherry-pick of odoo/odoo#30729 in master
Closes#30729closesodoo/odoo#31677
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Before this commit, when a user imports an Excel file, with date as
strings. The _try_match_date_time function will try to find the best
date parsing to use in the import without taking into account the
language of the user.
Now, the function will take into account the language of the user when
trying to find the best date parsing pattern for the import.
opw-2046217
closesodoo/odoo#35540
Signed-off-by: Jorge Pinna Puissant (jpp) <jpp@odoo.com>
Create a Binary field through the interface.
Try to import records through CSV like file, with a url as the value of that new image field
Before this commit, the url was saved in DB, leading to an error when trying to get
the image at read time (/web/image)
This is due to the fact that before 66f0e26f6f (saas-12.2) , Binary fields
were not attachments by default, thus did not enter the condition that db403e6dd7
introduced
After this commit, the special case of manual fields with url at import is correctly handled
OPW 2024822
closesodoo/odoo#34489
Signed-off-by: Lucas Perais (lpe) <lpe@odoo.com>
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
Check that it makes custom binary fields into attachment as that's the
main reason for the change: when users create binary fields via Studio,
they're necessarily db-stored (as the interface doesn't allow altering
the attachment attribute and it's unclear how we'd handle users
switching it on/off every time), which significantly bloats their
database (and burns storage & backup space), especially as the primary
use case for binary fields is adding images and documents to records.
* check that binary fields are properly created as attachment=True
* add attachment=False on fields where that seems relevant (most but not
all of the fields previously using the default)
* remove occurrences of attachment=True
closesodoo/odoo#29308
* actually pass the flags as flags in re.sub, passing re.IGNORECASE
as *count* doesn't actually ignore case
* ensure the entire string is matched by the pattern, or we'll get
shortest-matching-pattern which is *not* what the later strptime
will do
closesodoo/odoo#27925
Do not propagate cache invalidations to other workers when changes must be
discarded, because of an import error or a dry run, which are both handled as
successful transactions.
* reduce number of patterns: using localised month names is useless
since we're not setting the locale so it always uses the server's own
* avoid going through the entire strptime machinery: lift the bits of
TimeRE we're interested in to get regex bits out of strptime patterns
and just to an re.match to check whether value & pattern match
Further possible optimisation: cache REs and only compile user-provided
patterns dynamically.
Purpose of this commit is to give description more "business oriented"
because those descriptions appears in Odoo Studio which is supposed to be used by end users, not only by developers.
Related Task ID : 37311
Adds a checkbox to import columns (in debug mode) allowing a user to
create records M2O and M2M records not found (via name_search).
Task ID: 1850633
* uses a context key to avoid altering basically all the import callstack
* attempted to lift the creation in the `_str_to_*` functions and create
m2m via commands, but that doesn't really work out
Turns out %s is not the pattern for sub-minute seconds, that'd be
%S (note the casing). The former is the number of seconds since
epoch, which tends not to match sub-minute second patterns (sounds
like bull to me but there you are), and so importing datetimes was a
bit broken since the previous improvements.
Fix that.
Saurabh Choraria