PURPOSE
Improve import wizard as followup of recent improvements.
SPECIFICATIONS
Preview lines
- make the preview lines more visible
- set font-weight: 500 on o_import_header_name
- remove the text-muted class from the data preview line
- allow the user to preview the first 5 non-empty values
- when hovering on the data preview span, display a tooltip with the first
5 non-empty values of the file column (with 'Preview' as the tooltip title)
- In the 'when a value cannot be matched' dropdown:
add a new 'skip record' option for the following field types: boolean,
many2one, many2many, selection
add a new 'set empty' for the following field types: many2one, many2many,
selection hide this option if the field is required
When a value cannot be matched:
the 'skip record' option will make sure that lines with an unmatched value
will not be imported (i.e. skipped at import)
the 'set empty' option will set the field value to False when the value
cannot be matched
- Restore colored background on alert boxes by removing some css rules
Task-2504343
closesodoo/odoo#73713
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Co-authored-by: Mohammed Shekha <msh@odoo.com>
Purpose
=======
Propose an import tool that is more intuitive and allows more import options to
ease the import.
Specifications
==============
Update upload import file screen
--------------------------------
- Relabel import welcome screen
- relabel the 'load file' button to 'upload file'
- relabel the first (bold) line of the helper to 'Upload an Excel or CSV
file to import'
- Move all import options in a left panel
- Invert rows and columns in the mapping view
- One row for each column (header) to map in the imported file
- Columns: File Column / Odoo Field / If a match cannot be found / Comments
Left panel details
------------------
The sidepanel to the left of the mapping view is composed as followed:
- Imported File
- Import file name
- Sheet: Dropdown to select the sheet in the file if there are more than 1.
- 'use first row as header' checkbox: like the existing option, defines
whether the first row of the file should be considered as the header.
True by default. If False, display the first value only under 'File Column'
(see above)
- Formatting: only available if the file is a .csv
- list every option that already existed in previous import implementation.
- Batch Import
- is only available in debug moide and if the file exceeds the batch limit
- Batch limit : 2000
- Allow to define the threshold and the batch size.
- Help
- Download Template link
- Go to FAQ link
- Advanced
- Track history during import. (same as existing functionallity)
- Show Fields of relation fields. (same as existing functionallity)
Columns details
---------------
- File Column:
- Contains the column headers of the import file and the first non-empty
value of the column as a 'subtitle' (in grey italic)
- If 'use first row as header' is False, only display the first column value
(without the grey italic)
- Odoo Field:
- Contains dropdowns to the fields of the current model
- Required fields are diplayed in bold in the dropdown
- Icon: In the front of the field label, the field type (char, many2one,..)
is indicated by an icon.
- Tooltip: when hovering a field, display a tooltip with the following
information: field label, technical name, type, related model (if any)
- Placeholder: If no field is selected, display the placeholder
'To import, select a field...' in text-warning bold (i.e. orange)
- Allow clear: there is a close (fa-times) icon at the end of the dropdown
body to remove the Odoo field (and prevent import of this column)
- Multi Mapping: when setting a field already matched to another column,
unset it from the 'old' column (NB: exception made for char/text fields)
- Comments:
- Contains feedback from the system to the user, either import
errors/warnings or any additional details (multi mapping comments,...)
- For many2many field, display in related comment cell the following:
"To import multiple values, separate them by a comma".
- If there is an import error for a specific field, the error will be
displayed in the comment cell of the related field.
- If there is a mapping error, mapping options are displayed under the
error div to let the user choose the best option to fix that error.
Import errors management
------------------------
- Errors/warnings that can be matched to a specific field are displayed as
alert-danger/warning in the corresponding 'Comments' cell of the field.
- For 'no match found' errors, if X values couldn't be found,
display an unique error box will all the errors.
- Values beyond the first three are folded under a 'More' button.
- No changes on the 'See possible values' button.
- If an unmatched value is present in one row only, display :
'<value> at row X (<name of the row if the name field is matched>)'
- If an unmatched value is multiple rows, display:
'<value> at multiple rows'
- Errors/warnings/infos that cannot be matched to specific field are displayed
above the mapping listview. Global errors are not regrouped by error types to
avoid too much code complexy just to handle the rare times where multiple errors
of same types cannot be linked to a specific mapped field.
BaseImportError class have been introduced to ease the formatting of the various
exceptions that can occur during the import.
After testing the import, display the following above the mapping table:
- No warning/error:
'Everything seems valid' (alert-info)
- At least one error:
'The file contains blocking errors (see below)' (alert-error)
- At least one warning but no error:
'The file contains non-blocking warning (see below)' (alert-warning)
Mapping options
---------------
Mapping options are only displayed after testing or importing the file,
if there are import errors. The goal is to have a clean interface and to guide
the user step by step. The import process can therefore be a bit longer as it
needs to import -> choose solution for errors -> re-import but that is easier
for users to learn and understand this reworked import tool.
Possible values when a value cannot be matched:
- For many2one / many2many fields:
- Prevent import: (selected by default) not finding a match is blocking the
import
- Skip unknown values: values that cannot be matched will be skipped.
(hidden if the field is required)
- Create new values: Create records for values that cannot be matched
- For selection fields:
- Prevent import (selected by default)
- Skip unknown values: (hidden if the field is required)
- Set to <first value>: if cannot be matched, set it to <first value>
- Set to <second value>
- Set to <third value>
- etc.
- For boolean fields:
- Prevent Import (selected by default)
- Set to True
- Set to False
Note: With this rework, boolean warnings where the system assumes the
replacement value in case of matching error is removed and is replaced by a
blocking error. The user now has to choose the value to set.
"Prevent import" is the default behaviour when testing or importing.
Automatic mapping proposal
---------------------------
When loading a file, an automated mapping is directly proposed to the user,
based on word distance (see below), and on mapping created on previous
imports.
- Priority is given for mapping created on previous imports, skip fuzzy mapping.
- a distance of -1 is used to ensure priority during duplicates removal .
- In case multiple headers are mapped on the same field, if the mapped field
is already taken by another header, use fuzzy mapping instead.
- If no previous mapping for that header on that model, try an exact match on
every field id and field name of the model. (distance = 0)
- If no match is found, fuzzy mapping is applied (word distance).
The fuzzy mapping is executed only on the most likely fields (see below).
- Remove duplicates: keep the header-field couple that has the smallest
distance. In case of equality, keep the first.
Automatic mapping is therefore optimised for previous mapping or exact match,
as fuzzy mapping requires heaviest treatment. This is intended to prioritise the
import of files based on import templates.
Most likely fields
------------------
When parsing the import file, each header is analysed to guess what type of data
the column contains. For example, ff the column contains float, we suppose that
that header will most likely be matched on float or monetary fields.
The most likely fields are a subset of the model's fields that match the
header types.
Most likely fields are used for the fuzzy mapping, to propose the user a field
mapping based on the header types, if an exact match could not be found.
Most likely fields are also listed under "Suggested fields" in the mapping
dropdown in "Odoo Field" column of the import tool.
For now on, every fields (no mather their type) can be matched to any header,
but we prioritise the most likely fields. This way, we don't constrain the
mapping possibility based on what we suppose the user would do, but we instead
guide suggest the user the most likely mapping solutions.
Word distance mapping
----------------------------
In order to improve the mapping configuration, if an exact match cannot be found
between the file column and one of the odoo field:
- Use Word distance:
Word distance return a indice between 0 and 1.
- 0: exact match
- 1: completely different
- A: First try on field['name']
- B: Then on field['string']
- Keep the minimal distance between A and B for each odoo_field
- C: Keep the field that has the minimal distance.
- Match the column to the field by default if C['distance'] < 0.3.
Note: 0.3 has been chosen to ensure proximity but still having a little
error margin.
Multi mapping
-------------
When multiple file columns are matched to the same char/text/many2many field,
display the following alert-info box in the "Comments" column of the related
fields:
"Those columns will be concatenated in field <field label>"
Multi mapping rule :
- If it is a char field, separate the concatenated values by a space
- If it is a text field, separate the concatenated values by a line break
- If it is a many2many field, separate the concatenated values by a comma
Various improvements
--------------------
- During Import (or Test), the first waiting message displayed has been modified
to 'Importing...' or 'Testing...'. The progress (x record imported/tested) is
only displayed after the first batch of record have been imported/tested.
- Ease matching for Selection field: use case insensitive comparaison instead
of exact match.
- Add filename to import wizard.
- Add placeholder to search input of field mapping dropdown.
Tests have been adapted accordingly.
Links
=====
Task ID: 2352241
closesodoo/odoo#61948
Related: odoo/enterprise#17246
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Steps to reproduce:
-install sales
-try to import a product file with volumes set to scientific notation
(9.2e-05 for example)
Previous behavior:
scientific notation is not recognized by the base_import module
and raises a small warning
Current behavior:
scientific notation is converted to decimal notation on the fly
when possible
opw-2162353
closesodoo/odoo#42591
X-original-commit: 9acd76c5d116abd07207af3b548658ad983ff82d
Signed-off-by: Jorge Pinna Puissant (jpp) <jpp@odoo.com>
various UI changes
------------------
* renamed "test import" button to "test"
* move relation fields thing to debug mode
* remove "Defer parent/child computation" option as it was deprecated
/ removed from the backend in
80f1ac3599, turns out this checkbox
existed inactive for longer than it's been of any use (added in
68cb2ade09 on 2017-11-29, made
non-operating on 2018-01-24, that's so sad)
batching
--------
* add support for batching imports (skip & limit parameters)
* modify client to use batched imports & properly adapt responses so
it still looks like a single import for the client (more or less)
e.g. update row numbers in error messages, etc...
* properly handle partial imports though
* disable usual loading throbber to have a single progress
notification displayed continuously throughout all the batches: the
normal throbber only shows after 3s of waiting for an RPC response,
so it would keep flashing in and out (appear 3s into a batch's
import then disappear at the end only to reappear 3s into the next
batch's loading)
NOTE: the limit is row-wise. If a record straddles the limit (because
of nested O2M records), the record is imported in full and the "next
row" is whatever row follows the record. This means a limit of 10 can
lead to an import of 17 lines, and as the progress indicator is in
records# the increments can jump around.
Task 2059448
Some distributions still bundle chardet 2.3, which have some guessing
divergences / incompatibilities with python (resolved in 3.x):
1. UTF-{16,32} with BOM is guessed as LE/BE, which when used to decode
the string doesn't strip out the BOM. Handle this by checking if
the BOM is present and converting the encoding name to the
non-marked version in that case.
2. The ISO-8859-1 test string is guessed as ISO-8859-2 (TBF the
decoding does make some sense). Allow multiple targets/guesses to
"fix" that.
closesodoo/odoo#33179
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
* expand auto-detected date and time patterns (e.g. %b, %I, ...)
* try to make date-pattern-detection clearer
* add a select2 dropdown for date patterns (with a bunch of
preselected patterns) rather than just an input
* also try to improve other column-matching bits (e.g. less reliance
on exceptions, attempts to avoid redundant work)
It would probably be even better to iterate the file content and get
the non-quoted non-alphanumeric characters as separator
candidates (instead of a hard-coded list) however Python does not seem
to have a decoding iterator (taking bytes and yielding an iterator of
codepoints or even grapheme clusters) — incidentally uniseg seems to
require up-front decoding as well — so that's not really convenient as
we may be dealing with large-ish files and not want to load it
entirely in memory.
An alternative would be to use TextIOWrapper and iterate the file by
buffers of a few ks, and classify that based on either codepoints or
grapheme clusters.
* if an encoding is explicitly specified, use it and don't guess
* otherwise guess and return the guessed encoding so it can be
displayed in the configuration UI
* fix less-than-stellar configuration & behaviour of select2 inputs
to properly reflect underlying values as they get modified, to
correctly handle future configuration guesses
* Handle a leading + in import data, some contexts (e.g. bank
statements) will mark positive sums explicitly for clarity
* Add basic grouping/decimal separator inference for people importing
data from many localisations or to avoid them *having* to configure
their separators if we can handle it for them, currently very basic
Task 40692
Various changes to import/export (mainly) UIs:
* default to excel & "full" (non-import-compatible) export
* auto-detect encoding of CSV using chardet
* remember column -> field mapping after having imported a file (useful
for repeated imports where auto-matching failed)
* better handle localised booleans & column names
* automatically select source list view's fields when exporting
* better integrate import templates feature and add a number of templates
The commit 1ba4fbe640 did not match the specs of the task, but
was merged anyway. Reverting the commit, and check by default so the
user is not confused with missing fields.
opw-1824074
Use case to reproduce:
-> Import a file with a float using a coma related to the model
(for example the product.supplierinfo/price)
It happens because the method that parse the date and float field
only compare them to the main imported model.
This commit complete the method by using a recursion on relational
model in order to also parse the relational subfields.
- Import a CSV file without header
- Untick the option 'The first row contains the label of the column'
An error is reported.
If no header, the method `_match_headers` should return consistent
object types.
opw-769117
* StringIO removed from stdlib, replace with io
* try to correctly handle BytesIO/StringIO (one is for bytes the other
is for text)
* fix base64: Python 3 removed bytes-encoding and bytes-bytes
codecs (via #encode) so replace all calls to str.encode('base64'),
also b64encode is a bytes->bytes conversion so attempt to properly
handle that
issue #8530
In Python 3:
* various builtins and dict methods were changed to return
view/iterable objects rather than lists
* and the separate Python 2 view/iterable builtins and methods were
removed altogether
This is problematic when using these items as list (which the happens
repeatedly in Odoo), but more viciously when iterating *multiple times*
over them (which also happens, which I've messed up multiple times while
writing this, and which is a pain to debug even when you've just created
the issue).
Convert all code using these to semantics-matching cross-version
helper functions to get the LCD behaviour between P2 and P3, and
forbid the builtins via lint.
issue #8530
normal mode: Importing only regular/basic fields from source file
advanced mode: Importing any related fields from source file(i.e. if detected pattern in header like '<relational_fieldname>/<fname>')
- Rename following checkbox "Show all fields for completion (advanced)" in "Show fields of relation fields (advanced)"
- If the import file contains a column with a field of a relation field, show the column label in the preview, even if "Show fields of relation fields (advanced)" is not checked > to make it less confusing when importing advanced files (e.g. products with attributes)
This is a backport of 1ba4fbe640, which was performed due to several
tickets mentioning confusion with the new import system.
normal mode: Importing only regular/basic fields from source file
advanced mode: Importing any related fields from source file(i.e. if detected pattern in header like '<relational_fieldname>/<fname>')
- Rename following checkbox "Show all fields for completion (advanced)" in "Show fields of relation fields (advanced)"
- If the import file contains a column with a field of a relation field, show the column label in the preview, even if "Show fields of relation fields (advanced)" is not checked > to make it less confusing when importing advanced files (e.g. products with attributes)
After this commit,
- chinese custom date are supported
- Possibility to specify the custom datetime format and not only dateformat.
Withtout it, impossible to import a date column and datetime column at
sametime with custom format.
- Add test to check import date with custom format
- Fix translation where error messages was in the source.
This commit fixes#13783 and closes#13803
Courtesy of @odony for the review ;)
automatically try to detect type of columns from the first 10 rows of file:
If column only contains integer, only show, int, float, monetary, m2o, o2m, m2m fields in wizard
If column only contains true/false values, only show boolean fields in wizard
etc
add an advanced mode which is the same as previous import (show all fields)
Support different date format and float value with currency as well as float value with parenthesis to reprensent negative value.
CSV export was added in f146e2a and since then newline were not
exported.
But new line should be allowed if the string is quoted by " characters
which is done in Odoo.
closes#11005
opw-667853
If the user provides a known mimetype which doesn't actually match the
file as far as we're concerned, warn and fallback on extension rather
than blow up.
This comes up when a user installs Excel on Windows: it associates CSV
files with itself and sets their mime type to application/vnd.ms-excel,
which is not correct and leads to us trying to load a CSV with xlrd (an
excel-parsing library).
Fixes#9779
* added intermediate ``_read_file`` step dispatching between CSV, ODS
and XLS(X) parsing
- primary dispatch on mime type, secondary on file extension:
+ OS may not provide a relevant mime type if no software locally
installed for the filetype (e.g. Windows sends excel files as
application/octet-stream if excel is not installed, and ODS files as
zip if OpenOffice/LibreOffice isn't installed)
+ applications may re-register extensions to non-standard mimetypes
breaking the dispatcher
So if the mimetype is found trust it, otherwise try with the
filename's extension (if any)
- all readers skip lines with only empty cells in their output
- ODS and XLS content are "CSVified", rows are converted to arrays of
unicode strings
* UI altered to not assume CSV files everywhere, and avoid returning
garbage preview data for non-CSV imports
Various:
* added a ``can_import`` utility function to tests.common, can be used
to skip tests if an optional Python dependency is not installed (but
one would like tests to run if the dependency is available)
* fixed datetime issue in ir_fields
* added converter for monetary fields (iso float)
* spreadsheet don't have integers, twiddling required to ensure integral
values won't be serialized as floats (breaking conversion back to
Python)
* improved some tests by asserting no error is generated (bonus: logs
the error message if there is one, rather than just saying the result
is blown)
* because some systems only provide elderly versions of
XLRD (e.g. current debian stable provides 0.9.2 from April 2013)
- keep using xldate_as_tuple instead of 0.9.3's xldate_as_datetime,
workaround is simple
- check for xlrd.xlsx in case of pre-0.8 XLRD
Authorship:
Ronak Baxi <rba@odoo.com>
Mohammed Shekha <msh@openerp.com>
Task 10792
Closes#7285
- display_name uses name_get and not the other way around:
name_get should not call _compute_display_name, _compute_display_name should call name_get.
The previous behaviour was not backward-compatible with the old api.
All the models redefining name_get would have 2 different behaviors between name_get and display_name.
- Do not set an inverse function to display_name:
In most cases, writing on display_name writes on _rec_name (if any, not mandatory).
If the display_name computation is redefined, we need to redefine as well the inverse method to avoid unexpected behaviour
This required to also modify tests in base_import as readonly fields are avoided.
- Remove search method on display_name:
For the same reason as for the first point, it could be good that searching on display_name use name_search (and not the other way around).
However doing this would be very inefficiant (need to do the search, without limit, extract the ids of the name_get result just to generate
a subdomain ('id', 'in', [...]). As in most cases it would anyway mean to search on the _rec_name it's better to directly do so.
- Changing label to avoid mismatch:
In view displaying the list of fields or when a match is made on the label of a field (e.g. when importing csv file,
matching is made on both label and technical name), the fact that display_name field has '
Calling it 'Display Name' will avoid most errors.
- remove display_name definition from website_forum_doc,ir_model:
These fields are doing the same thing as the display_name of the new api, we can remove them.
We need to keep the one for res.partner as it's a stored field.
A squashed merge is required as the conversion of the apiculture branch from
bzr to git was not correctly done. The git history contains irrelevant blobs
and commits. This branch brings a lot of changes and fixes, too many to list
exhaustively.
- New orm api, objects are now used instead of ids
- Environements to encapsulates cr uid context while maintaining backward compatibility
- Field compute attribute is a new object oriented way to define function fields
- Shared browse record cache
- New onchange protocol
- Optional copy flag on fields
- Documentation update
- Dead code cleanup
- Lots of fixes
