Column mappings are updated in-place, if multiple users are importing records
of the same model at the same time, this will trigger concurrency errors.
This is made worse by the error only being reported on commit (after having
processed the entire import) and being retried automatically, so it slows down
the user and the entire system, the more concurrent imports the slower.
Log except:
INFO dbname odoo.addons.base_import.models.base_import: done
ERROR dbname odoo.sql_db: bad query: UPDATE "base_import_mapping" SET "field_name"='name',"write_uid"=%s,"write_date"=(now() at time zone 'UTC') WHERE id IN (%s)
ERROR: could not serialize access due to concurrent update
INFO dbname odoo.service.model: SERIALIZATION_FAILURE, retry 1/5 in 0.8720 sec...
closesodoo/odoo#57655
X-original-commit: 3f5e9ca625b53021c100245f5786bda4069974ad
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
This commit fixes all issues detected by the new pylint
gettext-variable test.
It converts some calls to the new syntax
_("Foo %s", bar)
to progressively migrate the code to the new syntax.
A few calls were not technically incorrect but still detected by the
linter.
_("Foo" +
"Bar")
has been converted to
_("Foo"
"Bar")
as it has the same effect and make sure the argument is of type
asteroid.Const instead of BinOp).
closesodoo/odoo#53683
Related: odoo/enterprise#11467
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
Using a few regex like
\((_\(.*%s.*)(\) % )([\w\[\]][\w .\[\]\(\)'"]*)\)
($1, $3))
Old syntax is still compatible but starts the migration to the new
syntax that catches error.
Steps to reproduce:
-install sales
-try to import a product file with volumes set to scientific notation
(9.2e-05 for example)
Previous behavior:
scientific notation is not recognized by the base_import module
and raises a small warning
Current behavior:
scientific notation is converted to decimal notation on the fly
when possible
opw-2162353
closesodoo/odoo#42591
X-original-commit: 9acd76c5d116abd07207af3b548658ad983ff82d
Signed-off-by: Jorge Pinna Puissant (jpp) <jpp@odoo.com>
Excel can store data in multiple tables, each is called a sheet. Odoo
was only importing the first sheet making the process to import a file
containing multiple sheets cumbersome.
It is now possible to select the sheet in the import options. By default
the first sheet is selected.
closesodoo/odoo#40728
Task: 2043768
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
Test import of a file with xml ids.
At the first dry run, the xml id X is associated with id R for each such record.
If some records refer to X via a relational field, then in SQL it reduces
to a query using R as id; but R does not exist since it was a dry run.
As a result, subsequent runs fail.
The cache should be cleared after a dry run, since the xml ids are not reliable.
opw 2068446
closesodoo/odoo#37396
X-original-commit: 1b35294d7b8d07f373eca24977e2952b9cfb2c7d
Signed-off-by: Nans Lefebvre (len) <len@odoo.com>
* If the item is not an acceptable URL, check if it's base64 and
return an error if it's not, otherwise it just ends up blowing later
when the content of the field is assumed to be base64 and explodes
* The URL having to contain png or jpg or tiff or gif or bmp doesn't
make much sense (especially in this modern world where
placekitten.com doesn't add extensions, not to mention this doesn't
actually check for extensions). After discussing with odo it doesn't
seem to have any security impact, the only thing this filter does is
annoy people for no reason.
various UI changes
------------------
* renamed "test import" button to "test"
* move relation fields thing to debug mode
* remove "Defer parent/child computation" option as it was deprecated
/ removed from the backend in
80f1ac3599, turns out this checkbox
existed inactive for longer than it's been of any use (added in
68cb2ade09 on 2017-11-29, made
non-operating on 2018-01-24, that's so sad)
batching
--------
* add support for batching imports (skip & limit parameters)
* modify client to use batched imports & properly adapt responses so
it still looks like a single import for the client (more or less)
e.g. update row numbers in error messages, etc...
* properly handle partial imports though
* disable usual loading throbber to have a single progress
notification displayed continuously throughout all the batches: the
normal throbber only shows after 3s of waiting for an RPC response,
so it would keep flashing in and out (appear 3s into a batch's
import then disappear at the end only to reappear 3s into the next
batch's loading)
NOTE: the limit is row-wise. If a record straddles the limit (because
of nested O2M records), the record is imported in full and the "next
row" is whatever row follows the record. This means a limit of 10 can
lead to an import of 17 lines, and as the progress indicator is in
records# the increments can jump around.
Task 2059448
- Change the date format of the users language to "%b %d, %Y"
- Export any res.partner as .csv and include at least 'Display Name' in
the fields to Export
- Import the file exported
The import fail with error: "Import preview error failed due to: 'b'".
This was introduced with commit 32c2666d18 where the format of the
user was added in the list of patterns to match. Since we use a limited
version of TimeRE (see Python module `_strptime`), only the patterns
defined in `_P_TO_RE` are supported. If an unsupported format is used, a
`KeyError` is raised.
We simply skip the pattern if it cannot be parsed. Another solution
would be to instanciate TimeRE, but that would also require to set the
locale correctly. Long story short: things will start to get messy for a
nice-to-have feature, a.k.a being able to import dates such as
`January 1, 2019`.
Fixes#35868
opw-2055971
opw-2055140
closesodoo/odoo#35883
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
Multi is the default api for methods, it is not necessary to explicitly
decorate methods with it, adds clutter and most people use it because
they see that the rest of the code uses it.
Done with `find . -type f -name '*.py' | xargs sed -i '/@api.multi/d'`
Some distributions still bundle chardet 2.3, which have some guessing
divergences / incompatibilities with python (resolved in 3.x):
1. UTF-{16,32} with BOM is guessed as LE/BE, which when used to decode
the string doesn't strip out the BOM. Handle this by checking if
the BOM is present and converting the encoding name to the
non-marked version in that case.
2. The ISO-8859-1 test string is guessed as ISO-8859-2 (TBF the
decoding does make some sense). Allow multiple targets/guesses to
"fix" that.
closesodoo/odoo#33179
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
When an import failed because of a specific cell, the error was
Error cell found while reading XLS/XLSX file: #N/A
Add the line and column index in the error for better debugging
Translate error message
Cherry-pick of odoo/odoo#30729 in master
Closes#30729closesodoo/odoo#31677
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
Before this commit, when a user imports an Excel file, with date as
strings. The _try_match_date_time function will try to find the best
date parsing to use in the import without taking into account the
language of the user.
Now, the function will take into account the language of the user when
trying to find the best date parsing pattern for the import.
opw-2046217
closesodoo/odoo#35540
Signed-off-by: Jorge Pinna Puissant (jpp) <jpp@odoo.com>
Create a Binary field through the interface.
Try to import records through CSV like file, with a url as the value of that new image field
Before this commit, the url was saved in DB, leading to an error when trying to get
the image at read time (/web/image)
This is due to the fact that before 66f0e26f6f (saas-12.2) , Binary fields
were not attachments by default, thus did not enter the condition that db403e6dd7
introduced
After this commit, the special case of manual fields with url at import is correctly handled
OPW 2024822
closesodoo/odoo#34489
Signed-off-by: Lucas Perais (lpe) <lpe@odoo.com>
This commit replaces calls to pycompat helpers that were intended for
python 2 <-> python 3 interoperability for python 3 builtins, as python
2 is no longer officially supported by Odoo.
This includes:
* calls to imap/izip/ifilter replaced by map/zip/filter
* uses of text_type replaced by str
* uses of unichr replaced by chr
* calls to implements_to_string, implements_iterator removed
* string_types and integer_types replaced by str, int respectively
* calls to to_native replaced by calls to to_text
This is done in preparation to the removal of these deprecated helpers
in the following commit.
Check that it makes custom binary fields into attachment as that's the
main reason for the change: when users create binary fields via Studio,
they're necessarily db-stored (as the interface doesn't allow altering
the attachment attribute and it's unclear how we'd handle users
switching it on/off every time), which significantly bloats their
database (and burns storage & backup space), especially as the primary
use case for binary fields is adding images and documents to records.
* check that binary fields are properly created as attachment=True
* add attachment=False on fields where that seems relevant (most but not
all of the fields previously using the default)
* remove occurrences of attachment=True
closesodoo/odoo#29308
* actually pass the flags as flags in re.sub, passing re.IGNORECASE
as *count* doesn't actually ignore case
* ensure the entire string is matched by the pattern, or we'll get
shortest-matching-pattern which is *not* what the later strptime
will do
closesodoo/odoo#27925
Do not propagate cache invalidations to other workers when changes must be
discarded, because of an import error or a dry run, which are both handled as
successful transactions.
* reduce number of patterns: using localised month names is useless
since we're not setting the locale so it always uses the server's own
* avoid going through the entire strptime machinery: lift the bits of
TimeRE we're interested in to get regex bits out of strptime patterns
and just to an re.match to check whether value & pattern match
Further possible optimisation: cache REs and only compile user-provided
patterns dynamically.
Purpose of this commit is to give description more "business oriented"
because those descriptions appears in Odoo Studio which is supposed to be used by end users, not only by developers.
Related Task ID : 37311
Adds a checkbox to import columns (in debug mode) allowing a user to
create records M2O and M2M records not found (via name_search).
Task ID: 1850633
* uses a context key to avoid altering basically all the import callstack
* attempted to lift the creation in the `_str_to_*` functions and create
m2m via commands, but that doesn't really work out
Turns out %s is not the pattern for sub-minute seconds, that'd be
%S (note the casing). The former is the number of seconds since
epoch, which tends not to match sub-minute second patterns (sounds
like bull to me but there you are), and so importing datetimes was a
bit broken since the previous improvements.
Fix that.
Before, once reaching level 0 all of an O2M's fields would still be
selected, so we would really be selecting depth+1 levels of fields,
which makes for an extremely busy list as it's a combinatorial
explosion.
Change this so at level 0 the only field selected from a field is an
xid. Not sure that's even necessary though, do xids really make sense
for o2ms?
* expand auto-detected date and time patterns (e.g. %b, %I, ...)
* try to make date-pattern-detection clearer
* add a select2 dropdown for date patterns (with a bunch of
preselected patterns) rather than just an input
* also try to improve other column-matching bits (e.g. less reliance
on exceptions, attempts to avoid redundant work)
It would probably be even better to iterate the file content and get
the non-quoted non-alphanumeric characters as separator
candidates (instead of a hard-coded list) however Python does not seem
to have a decoding iterator (taking bytes and yielding an iterator of
codepoints or even grapheme clusters) — incidentally uniseg seems to
require up-front decoding as well — so that's not really convenient as
we may be dealing with large-ish files and not want to load it
entirely in memory.
An alternative would be to use TextIOWrapper and iterate the file by
buffers of a few ks, and classify that based on either codepoints or
grapheme clusters.
Various bits would lead to odd tracebacks or a complete lack of
actionable feedback.
Discover that a binary field set to b'' will return None when read back,
handle that, then handle reading an empty CSV file, then handle the
resulting iterator having no lines whatsoever when trying to match
headers.
At least for CSVs, we're now properly telling users that their file
seems to have no content when they literally upload an empty file.
* if an encoding is explicitly specified, use it and don't guess
* otherwise guess and return the guessed encoding so it can be
displayed in the configuration UI
* fix less-than-stellar configuration & behaviour of select2 inputs
to properly reflect underlying values as they get modified, to
correctly handle future configuration guesses
* Handle a leading + in import data, some contexts (e.g. bank
statements) will mark positive sums explicitly for clarity
* Add basic grouping/decimal separator inference for people importing
data from many localisations or to avoid them *having* to configure
their separators if we can handle it for them, currently very basic
Task 32132
Currently, importing image requires either uploading them
one-by-one (after the initial import), scripting that upload, or
converting all images to base64 and putting that in the import
file (leading to a huge unreadable file) before actually doing the
import.
This is problematic for non-technical users, especially when they're
deploying or synchronising e-commerce products.
This change matches URLs in image fields and goes to read the URL &
load that as image data. It performs the read inline which is reactive
but means the imports will be very slow (wallclock) as each URL will
have to be fetched and encoded.
Configuration: via config file only
* import_image_timeout (seconds, default 3), initial connection timeout (not
overall file download)
* import_image_maxbytes (bytes, default 10MB): maximum image size to
download, checked before download (Content-Length) and validated
during (in case Content-Length is missing)
* import_image_regex (str), matches field content as "probably an
image", by default checks that the field value starts with
http|https and ends with an image-type extension (e.g. bmp, png, jpg)
Hard Limits:
* only works on fields actually called "icon", "image", "logo" or
"picture"
* the image can't be more than 42MP (checked after all data is
downloaded)
Hooks:
* _can_import_remote_urls (on res.users) checks whether the current
user is allowed to import via URLs
* _import_image_by_url (on base_import.import) performs the actual
download, could be overridden to store urls to import and
batch/queue the actual imports
Task 40692
Various changes to import/export (mainly) UIs:
* default to excel & "full" (non-import-compatible) export
* auto-detect encoding of CSV using chardet
* remember column -> field mapping after having imported a file (useful
for repeated imports where auto-matching failed)
* better handle localised booleans & column names
* automatically select source list view's fields when exporting
* better integrate import templates feature and add a number of templates
The commit 1ba4fbe640 did not match the specs of the task, but
was merged anyway. Reverting the commit, and check by default so the
user is not confused with missing fields.
opw-1824074
When a user imports a file with a datetime field set to a blank space,
the value is computed as 1900-01-01 00:00:00. However, when converted in
the user's TZ, this can lead to a value < 1900-01-01, which is not
recognized at the JS level as a valid value.
This makes the access to the record impossible, and possibly makes the
access to an app impossible. It is for example the case with Purchase,
where the order date is displayed on the default's action view (list
view).
We strip the value before testing if the value exists to avoid this
case.
opw-1826344
Moisés López