Do not propagate cache invalidations to other workers when changes must be
discarded, because of an import error or a dry run, which are both handled as
successful transactions.
* actually pass the flags as flags in re.sub, passing re.IGNORECASE
as *count* doesn't actually ignore case
* ensure the entire string is matched by the pattern, or we'll get
shortest-matching-pattern which is *not* what the later strptime
will do
closesodoo/odoo#27925
* reduce number of patterns: using localised month names is useless
since we're not setting the locale so it always uses the server's own
* avoid going through the entire strptime machinery: lift the bits of
TimeRE we're interested in to get regex bits out of strptime patterns
and just to an re.match to check whether value & pattern match
Further possible optimisation: cache REs and only compile user-provided
patterns dynamically.
Purpose of this commit is to give description more "business oriented"
because those descriptions appears in Odoo Studio which is supposed to be used by end users, not only by developers.
Related Task ID : 37311
Adds a checkbox to import columns (in debug mode) allowing a user to
create records M2O and M2M records not found (via name_search).
Task ID: 1850633
* uses a context key to avoid altering basically all the import callstack
* attempted to lift the creation in the `_str_to_*` functions and create
m2m via commands, but that doesn't really work out
Turns out %s is not the pattern for sub-minute seconds, that'd be
%S (note the casing). The former is the number of seconds since
epoch, which tends not to match sub-minute second patterns (sounds
like bull to me but there you are), and so importing datetimes was a
bit broken since the previous improvements.
Fix that.
Before, once reaching level 0 all of an O2M's fields would still be
selected, so we would really be selecting depth+1 levels of fields,
which makes for an extremely busy list as it's a combinatorial
explosion.
Change this so at level 0 the only field selected from a field is an
xid. Not sure that's even necessary though, do xids really make sense
for o2ms?
* expand auto-detected date and time patterns (e.g. %b, %I, ...)
* try to make date-pattern-detection clearer
* add a select2 dropdown for date patterns (with a bunch of
preselected patterns) rather than just an input
* also try to improve other column-matching bits (e.g. less reliance
on exceptions, attempts to avoid redundant work)
It would probably be even better to iterate the file content and get
the non-quoted non-alphanumeric characters as separator
candidates (instead of a hard-coded list) however Python does not seem
to have a decoding iterator (taking bytes and yielding an iterator of
codepoints or even grapheme clusters) — incidentally uniseg seems to
require up-front decoding as well — so that's not really convenient as
we may be dealing with large-ish files and not want to load it
entirely in memory.
An alternative would be to use TextIOWrapper and iterate the file by
buffers of a few ks, and classify that based on either codepoints or
grapheme clusters.
Various bits would lead to odd tracebacks or a complete lack of
actionable feedback.
Discover that a binary field set to b'' will return None when read back,
handle that, then handle reading an empty CSV file, then handle the
resulting iterator having no lines whatsoever when trying to match
headers.
At least for CSVs, we're now properly telling users that their file
seems to have no content when they literally upload an empty file.
* if an encoding is explicitly specified, use it and don't guess
* otherwise guess and return the guessed encoding so it can be
displayed in the configuration UI
* fix less-than-stellar configuration & behaviour of select2 inputs
to properly reflect underlying values as they get modified, to
correctly handle future configuration guesses
* Handle a leading + in import data, some contexts (e.g. bank
statements) will mark positive sums explicitly for clarity
* Add basic grouping/decimal separator inference for people importing
data from many localisations or to avoid them *having* to configure
their separators if we can handle it for them, currently very basic
Task 32132
Currently, importing image requires either uploading them
one-by-one (after the initial import), scripting that upload, or
converting all images to base64 and putting that in the import
file (leading to a huge unreadable file) before actually doing the
import.
This is problematic for non-technical users, especially when they're
deploying or synchronising e-commerce products.
This change matches URLs in image fields and goes to read the URL &
load that as image data. It performs the read inline which is reactive
but means the imports will be very slow (wallclock) as each URL will
have to be fetched and encoded.
Configuration: via config file only
* import_image_timeout (seconds, default 3), initial connection timeout (not
overall file download)
* import_image_maxbytes (bytes, default 10MB): maximum image size to
download, checked before download (Content-Length) and validated
during (in case Content-Length is missing)
* import_image_regex (str), matches field content as "probably an
image", by default checks that the field value starts with
http|https and ends with an image-type extension (e.g. bmp, png, jpg)
Hard Limits:
* only works on fields actually called "icon", "image", "logo" or
"picture"
* the image can't be more than 42MP (checked after all data is
downloaded)
Hooks:
* _can_import_remote_urls (on res.users) checks whether the current
user is allowed to import via URLs
* _import_image_by_url (on base_import.import) performs the actual
download, could be overridden to store urls to import and
batch/queue the actual imports
Task 40692
Various changes to import/export (mainly) UIs:
* default to excel & "full" (non-import-compatible) export
* auto-detect encoding of CSV using chardet
* remember column -> field mapping after having imported a file (useful
for repeated imports where auto-matching failed)
* better handle localised booleans & column names
* automatically select source list view's fields when exporting
* better integrate import templates feature and add a number of templates
The commit 1ba4fbe640 did not match the specs of the task, but
was merged anyway. Reverting the commit, and check by default so the
user is not confused with missing fields.
opw-1824074
When a user imports a file with a datetime field set to a blank space,
the value is computed as 1900-01-01 00:00:00. However, when converted in
the user's TZ, this can lead to a value < 1900-01-01, which is not
recognized at the JS level as a valid value.
This makes the access to the record impossible, and possibly makes the
access to an app impossible. It is for example the case with Purchase,
where the order date is displayed on the default's action view (list
view).
We strip the value before testing if the value exists to avoid this
case.
opw-1826344
Empty cells containing only spaces may have unexpected results (not
shown traceback and broken import) when being imported and they were
intended for a float or monetary field.
To avoid the error we could either, ignore left/right whitespace when
parsing a float value (so same behavior as en emtpy cell) or prevent the
error and get an friendly import report error:
' ' does not seem to be a number for field 'Name of Field'
This change choose the first way, so a float ' ' (just whitespace) will
be considered as '' (empty).
opw-1816784
closes#23105closes#23355
Co-authored-by: Humberto Arocha <hbto@vauxoo.com>
Currently when importing files, the parent left and right are recomputed on each record creation/update, which could lead to a lot a request.
This commit will defer the parent field recomputation to the end on the `load` function by recomputing the values from the MPTT and inserting directly the new values on the records.
Tested on a customer instance, this will reduce the execution time when importing a thousand stock.location from 15 minutes to several seconds.
Use case to reproduce:
-> Import a file with a float using a coma related to the model
(for example the product.supplierinfo/price)
It happens because the method that parse the date and float field
only compare them to the main imported model.
This commit complete the method by using a recursion on relational
model in order to also parse the relational subfields.
Steps to reproduce the bug:
1. Import a bank statement with the shown dateformad DDMMYYYY (without headers)
2. Specify this date format in the advanced options
3. Try to find the date column to assign.
Bug:
The date format is not suggested for the fields with format DDMMYYYY.
opw:769118
- Import a CSV file without header
- Untick the option 'The first row contains the label of the column'
An error is reported.
If no header, the method `_match_headers` should return consistent
object types.
opw-769117
Now that we're closer to switching to P3 for good, these helpers have
outlived their usefulness, and mostly add noise.
All remaining dict.iter*() or dict.view*() must be converted to the
normal keys(), values() or items() calls.
Whenever the result is likely to be used for more than the scope of a
loop, or when the dict needs to be modified during iteration, the calls
must be wrapped in a ``list()``, to protect the new P3 semantics.
Those cases are very exceptional.
Also removed some dead code or improved the API to remove unnecessary
conversions.
* remove references to basestring & unicode (use relevant pycompat
helpers)
* remove some str calls (either entirely or replaced by relevant
helper, either text or native)
* use better API to avoid unnecessary conversions
* remove some XML declarations in views
* StringIO removed from stdlib, replace with io
* try to correctly handle BytesIO/StringIO (one is for bytes the other
is for text)
* fix base64: Python 3 removed bytes-encoding and bytes-bytes
codecs (via #encode) so replace all calls to str.encode('base64'),
also b64encode is a bytes->bytes conversion so attempt to properly
handle that
issue #8530
Currently when importing files, the parent left and right are recomputed on each record creation/update, which could lead to a lot a request.
This commit will defer the parent field recomputation to the end on the `load` function by recomputing the values from the MPTT and inserting directly the new values on the records.
Tested on a customer instance, this will reduce the execution time when importing a thousand stock.location from 15 minutes to several seconds.
Raphael Collet