Commit Graph
644 Commits
Author SHA1 Message Date
Julien Mougenot 669c2ebaec [IMP] web: improve eval context dates
This commit handles 2 things:

1) the merging of the `py_utils.context()` into the evaluation context
returned by `BasicModel._getEvalContext` to access the same time-related
keys,

2) the addition of 2 new keys into the `py_utils.context` (thus
transmitted to the basic model): `today` (an alias for the already
present `current_date`) and `now` which represents the current date and
time value.

Task 2269697
2020-07-14 09:10:32 +00:00
william bffb3b7fce [REF] base: clean dosctring and remove deprecated function
The function check_with_xsd has been deprecated for more than 3 years.
Docstring is now compliant with PEP 257

closes odoo/odoo#54338

X-original-commit: 29938397ee17835645e89ee0dadb26e14ef45927
Signed-off-by: Josse Colpaert <jco@openerp.com>
2020-07-10 14:18:13 +00:00
william 048c7fef67 [IMP] base: _check_with_xsd from ir.attachment
Search the xsd files from in the database.
To enable this option, the Environment should be passed to the optional
`env` parameter. Both the XSD root and the XSD imported by the root and
the recusrively imported files will be searched in the database.

X-original-commit: 06a35f2e11230db81b8c21696d228097b31cf649
2020-07-10 14:18:12 +00:00
Martin Trigaux 110263df56 [IMP] tools: use same syntax for _lt than _
A new gettext syntax was added at odoo/odoo#52155 but only for _, not
for _lt
Allow to have the same syntax
2020-06-30 10:19:50 +00:00
Martin Trigaux d12813f7ee [IMP] tools: fallback on empty translation
Avoid returning False if, for any reason, the return is falsy (False,
None,...)

Following the discussion in odoo/odoo#53254, the root cause was a call in the form
  _(foo)

where foo was equal False

While nothing else than a string should be the argument of _, it's
still technically possible to pass something else and get a bad return
value.
Fallback on empty string to be consistent.
2020-06-30 08:53:29 +00:00
Nasreddin (bon) 3c169a415b [FIX] tools: Return input_str when removing accents if empty
Issue

	The misc feature remove_accents(input_str) return a
	string 'False' if the param is a boolean.

Solution

	Return input_str if equal '' or False.

opw-2278959

closes odoo/odoo#53572

X-original-commit: 3fc7ce3b5599355933734e5e3f59d14a7a9b199c
Related: odoo/enterprise#11391
Signed-off-by: Nicolas Martinelli (nim) <nim@odoo.com>
Signed-off-by: bon-odoo <nboulif@users.noreply.github.com>
2020-06-24 11:06:01 +00:00
Benjamin Frantzen (bfr) ae15f89f73 [FIX] tools.pdf: do not crash when PDF has no attachments
In the PDF, `DictionaryObject` can be wrapped in `IndirectObject`. For nested
dictionaries, using `get` on a `DictionaryObject` will unwrap the result if it's
an `IndirectObject`, but not `__getitem__` so when trying to futher call `get`
on the result will cause an error: we need to unwrap the object first.
Instead, we patch `DictionaryObject.get()` for it to unwrap the object in case
it's an `IndirectObject`.

This is a follow-up commit of fece5ab1bf2fef043b131f1bd0886f0841116103

closes odoo/odoo#53269

X-original-commit: 189a0b28ec7fdb7472f936450cfd7b4bfd6912ed
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Signed-off-by: bfr-o <bfr-o@users.noreply.github.com>
2020-06-18 16:49:32 +00:00
Martin Trigaux ba244cef01 [IMP] *: replace to new _() syntax
Using a few regex like
\((_\(.*%s.*)(\) % )([\w\[\]][\w .\[\]\(\)'"]*)\)
($1, $3))

Old syntax is still compatible but starts the migration to the new
syntax that catches error.
2020-06-18 13:03:34 +02:00
Martin Trigaux 3c3b88f676 [IMP] tools: add placeholder support to _
Mimic the syntax of logging to use placeholders in translatable
message.
The main advantage is to be able to fallback on the source term if the
translation can not be formatted properly.
It is very common to have errors in translations with missing
placeholders or badly translated (ie. the source
     "%(subject)s" -> "%(sujet)s").

Instead of blocking the execution of the code, fallback on source
string.

Task-id: 1853119
2020-06-18 13:03:34 +02:00
Dharmraj Jhala 5af661da7d [FIX] tools: fix tool method to check empty html content
Since a recent commit[1], we have a utility method in tools named
'is_html_empty'  that checks whether the given html content is
void(containing only formatting tags) or not. However, the re from
this method does not consider the case of self closing tags, fox ex
`<br/>`. In such cases, the method returns Falsy value even if the
content is void.

This commit fixes the issue by considering self-closing void tags
in the regular expression.

commit[1] - https://github.com/odoo/odoo/commit/974f512f5f5d3b9f80a8c3fcde290e4f55cf1230

Task - 2267689

closes odoo/odoo#53167

X-original-commit: 0fd6c9388dc959a9b6b7828850683175edb05b9d
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2020-06-17 14:19:09 +00:00
std-odoo b16843230d [IMP] mass_mailing: send a statistics email to the mailing responsible
Purpose
=======
Provide a follow-up/overview of the Mailing 24 hours after its been sent.

Specifications
==============
Send an email to the responsible of the mailing 24 hours the last email
of the mass mailing.

During the link trackers creation, extract the button label if exists
(so we can display them in the statistics email).

Technical remarks
=================
The statistics email is sent to the responsible of each mailing in the
CRON of mass mailing.

Task-2227411

closes odoo/odoo#49836

Related: odoo/upgrade#1094
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2020-06-12 11:49:45 +00:00
Nicolas Lempereur eaef67323b [FIX] tools: add bdi tag as safe tag
We already have BDO tag as allowed tag in sanitized HTML that is
similar to BDI tag, so this commit also adds BDI in allowed tags.

BDI* tags make sense in email since email client can have different
orientation than Odoo (alternative is to use `<div dir="auto"></div>`).

*: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/bdi

opw-2256982
closes #52787

closes odoo/odoo#52801

X-original-commit: 68edebb0d862acb3dc14f839aaef744306c5e269
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
2020-06-11 06:39:11 +00:00
Benjamin Frantzen (bfr) b1d19c61b1 [FIX] tools.pdf: do not crash when PDF has no attachments
Depending on the PDF version and the presence of embedded files, the PDF
document catalog `trailer["/Root"]` may not have any Name Dictionary
`["/Names"]`, and the latter may not have any `EmbeddedFiles` entry.

In that case `getAttachments()` should return an empty list rather than
raising a `KeyError`.

closes odoo/odoo#52312

Ref: Section 7.7.2, 7.7.4 and 7.11.4 of [PDF 1.7 spec](https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/PDF32000_2008.pdf)
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
2020-06-03 14:08:37 +00:00
1c522ee778 [ADD] odoo-bin cloc cli subcommand.
Odoo cloc is a tool to count the number of relevant lines written in Python,
Javascript or XML. This can be used as rough metric for pricing maintenance of
customizations.

It has two modes of operation, either by providing a path:

    odoo-bin cloc -p module_path

Or by providing the name of a database:

    odoo-bin cloc --addons-path=dirs -d database

In the latter mode, only the custom code is accounted for.
Both modes can be used simultaneously.

Files that cannot be parsed are shown at the end of the report.
Parsing can fail due to syntax errors or excessive file size.

closes odoo/odoo#52635

X-original-commit: ae858c3ac66267b4726db459032b91a6be1cc1d6
Related: odoo/enterprise#11018
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
Co-authored-by: Thibault Francois <tfr@odoo.com>
Co-authored-by: Antoine Vandevenne (anv) <anv@odoo.com>
2020-06-09 00:30:55 +00:00
Martin Trigaux c9f93eb639 [FIX] tools: remove max language length constraint
There is no reason for this constraint to be. Languages with a code
longer than 5 do exists in standard, (e.g. ar_001, sr@latin)

Introduced at 004a0b996f

Fixes odoo/odoo#52558

closes odoo/odoo#52569

X-original-commit: 54340487adaed962750c9719fb50fc7e000da476
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2020-06-08 09:00:02 +00:00
Benjamin Frantzen (bfr) 0226901ec7 [IMP] base, tools: manage multiple attachments on PDF files 2020-05-29 07:16:30 +00:00
Stéphane Debauche 2244961314 [IMP] mail,tools: clean tags to kill list in HTML sanitizer
Some tags are not necessary in the HTML template and we can remove them to
have a proper HTML output.

Task 2171753

closes odoo/odoo#51757

Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2020-05-25 07:28:19 +00:00
Martin Trigaux c142f1628d [FIX] tools: reimport translations in csv
Before this commit trying to reimport csv of a translation failed.
The reason was that before 632fa044c3 the parsing was common
between po and csv while now it's split in two different parsers.
The res_id column of the CSV is the external id of the record and is
handled by IrTranslationImport.

Use DictReader instead of csv_reader to easily add new keys and still
be flexible on the given csv files.

Match the POReader format with a imd_model and imd_name column.

As for the PoFileReader, the code translations are unique and must be
discarded in case of duplicate.

Correct the error message if the imported file is not correct (was
missing an argument)

Fixes odoo/odoo#50975

closes odoo/odoo#51468

X-original-commit: 3cedadb5ba606e156f52e70ed08b25ac2df8d58b
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2020-05-19 08:01:17 +00:00
std-odoo 1c7c837a10 [FIX] mass_mailing: improve opened traces tracking
Purpose of this commit is to improve tracking of opened traces. Notably
a token is added to ensure we do not mess with traces and have unique
tracking URLs.

MIGRATION REMARK

Emails sent before the migration will not be marked as opened anymore after
migration. We recommend to avoid sending statistically important mass mailings
about one week before migrating database. Indeed statistics show that most of
open emails happen within the first week after being sent.

Task 2223146

closes odoo/odoo#49139

Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
2020-05-18 08:30:59 +00:00
Stéphane Debauche 55cb0de1db [IMP] tools: remove unnecessary lxml Cleaner override
Purpose
=======

Currently a custom override of Cleaner.allow_element()
exists in tools to allow object tags for SVG images.
This was due to the first prototype of website builder
and its first implementation of image and svg management.

This is not necessary anymore as using the tag is sufficient.
We can safely clean code in the cleaner, allowing to speedup
its performances. As it is used in most html fields and email
parsing each unnecessary code removed is time gainged.

Task 2215228

closes odoo/odoo#51294

X-original-commit: 4a0b09fd3b0d3b63f2df00f002e4d6825cac4eef
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2020-05-14 18:51:22 +00:00
Martin Trigaux d9287caf94 [IMP] *: convert to private methods
render, render_template, load, activity_schedule_with_view,
get_website_pages should all be private:
It should not be possible to render an aribtrary template only with
its name or id

Still need to render some qweb views from js so the method
render_template is kept public.
This explains why the website editor still need read access on
ir.ui.view as we want to allow any snippet to be rendered.
2020-05-14 13:59:10 +02:00
Victor Feyens d2652e971a [IMP] doc: add information on the new populate feature.
* Cmdline interface: how to trigger database population
* Testing: how to implement database population on a given model.
  * autodocumentation of the population methods

+ improve population methods docstrings.

closes odoo/odoo#50596

Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
2020-05-13 15:47:04 +00:00
Julien Castiaux afcb734908 [IMP] ir_mail_server: IDNA and SMTPUTF8 capabilities
It has been a recurrent request from customers to be able to send email
messages to email addresses containing non-ascii characters. [IDNA] is a
domain extension to allow unicode characters in domain names. [SMTPUTF8]
is a SMTP extension to allow unicode in any header.

IDNA defines the [punycode] encoding which translates unicode to an
ascii representation. This encoding MUST be used to encode domains.

SMTPUTF8 is an SMTP extension that allow utf-8 in all headers on the
envelope.

[IDNA] https://tools.ietf.org/html/rfc5890
[SMTPUTF8] https://tools.ietf.org/html/rfc6531
[punycode] https://tools.ietf.org/html/rfc3492

Task: 2116928
opw-2229906
opw-2248251

closes odoo/odoo#47709

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-05-05 09:17:15 +00:00
Jeremy KerstenandRomain Derie 9247c5e8fd [IMP] tools: avoid useless query on res.users
Without this commit, if `lang_code` or `env.context.get('lang')` was set, we
would still access `env.user.company_id.partner_id.lang` to fill the loop array
even if we break and don't need that value.

Switching from loop to if conditions prevents that.

task-2211013

Co-authored-by: Romain Derie <rde@odoo.com>
Co-authored-by: Jeremy Kersten <jke@odoo.com>
2020-04-27 13:49:01 +00:00
Nicolas Lempereur b781fb588e [FIX] mail.py: escape plaintext email
A plaintext email is displayed in a `<pre/>` tag to conserve spacing.

But since there is no escaping, if in this text there was XML tags or
HTML entities, they would appear as HTML in browser which is not wanted.

Do note that this was not a security issue since the content will still
be subjected to the checks and foundling of HTML emails.

Without the change, the added test would fail because character &,<,>
were not escaped.

opw-2242323
closes #50003

closes odoo/odoo#50123

X-original-commit: 932532b5b59e0b71c8e16dadfb2ff36c38764208
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2020-04-24 09:08:08 +00:00
Christophe Monniez eec563c2f1 [FIX] packaging: include _vendor in packaging
Since a0f9ef56, a _vendor module was added without `__init__.py` file,
following the PEP 420 specifications.

Unfortunately, as the `setuptools.find_packages()` only recognize
packages if they have such an init file, the _vendor module is not
packaged. This leads to an import error when running Odoo from the
src,deb and rpm packages.

The `setuptools.find_namespace_packages()`, which is compliant with PEP
420, could have been used instead. But this function does not exists in
setuptools 39.0.1 which is the one packaged in Ubuntu Bionic.

Finally, this commit simply adds the missing dunder init file.

closes odoo/odoo#50051

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
2020-04-24 07:43:14 +00:00
Xavier Morel 9d4ee88263 [FIX] core: vendored sessions.py so it actually works
* remove SessionMiddleware we don't need as it's responsible for most
  imports
* fix relative imports to absolute imports from werkzeug
* remove py2/py3 compatibility imports, shims and conditions as we're
  P3 only
* remove deprecation warning (duh)
2020-04-22 09:28:14 +00:00
Xavier Morel 39fb7f9d31 [ADD] core: vendor werkzeug 0.16's sessions.py
In Werkzeug 1.0, sessions support was moved out and into a separate
package (secure-cookies). Issue is distros are already starting to
update werkzeug to 1.0, without necessarily adding
secure-cookies (e.g. arch, debian experimental). Plus secure-cookies
has some changes e.g. different filename & al. So just vendor the
"continuity" version which is the last werkzeug before removal.

This commit copies the original file as-is so we can track eventual
changes if necessary.
2020-04-22 09:28:14 +00:00
Xavier Morel 0655974d5a [IMP] safe_eval: opcodes blacklist & cleanup
There are bytecode operations we don't support because we've never
needed them, and there are bytecode operations we don't support
because they're vectors of security issues.

Currently no difference is being made between the two, so when looking
to add a new opcodes it's hard to know whether it's been considered
and rejected or whether it just hasn't been considered (or found
useful) yet.

Add an explicit blacklist, which is explicitly subtracted to all
opcodes lists, to allow motivating the bans of certain
opcodes. Opcodes listed in no lists are the "graylists" of opcodes we
either haven't yet considered or have not had a use for.

Also cleanup the existing lists of opcodes to remove long-gone (or
even never-existing) opcodes, and add a few missing opcodes:

* DUP_TOPX was removed from Python 3, replaced by DUP_TOP_TWO
* STORE_MAP was removed in Python 3.5
* BINARY_DIVIDE and INPLACE_DIVIDE were removed from Python 3 (they
  were used to invoke P2's integer division)
* the SLICE+<X> were removed from Python 3, which only uses
  BUILD_SLICE
* CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW were removed in Python
  3.6 and functionally replaced by CALL_FUNCTION_EX (bpo-27213)
* JUMP_IF_FALSE and JUMP_IF_TRUE were removed back in 2.7, replaced by
  POP_JUMP_* and JUMP_*_OR_POP (bpo-4715)
* various finally-related bytecode instructions were added to Python
  3.8 to improve and speed up the handling of return, break and
  continue (bpo-17611).
* INPLACE_REMAINDER, INPLACE_LEFTSHIFT and INPLACE_RIGHTSHIFT were
  mentioned in PEP 202 but never actually implemented: INPLACE_
  instructions were intended to mirror the BINARY_ instructions, these
  mnemonics didn't match the BINARY_ ones, so the actual mnemonics are
  INPLACE_MODULO, INPLACE_LSHIFT and INPLACE_RSHIFT
* while at it, BINARY_ and INPLACE_ mnemonics: as noted above they're
  mirrors of one another with different prefixes (except for SUBSCR
  which doesn't have an INPLACE version), we only supported a fraction
  of the INPLACE codes for unknown reason.

  Initially looked at simply aligning the BINARY and INPLACE mnemonics
  e.g.

       'BINARY_POWER',  'BINARY_MULTIPLY',  'BINARY_FLOOR_DIVIDE',
      'INPLACE_POWER', 'INPLACE_MULTIPLY', 'INPLACE_FLOOR_DIVIDE',
      ...

  so it would be easier to notice discrepancies, but it seems simpler
  and clearer to just move the suffixes to a single list and generate
  the BINARY_ and INPLACE_ mnemonics from that
* moved STORE_SUBSCR from EXPR to SAFE, I don't see how `foo[...] = 3`
  could be in an expression
* remove pseudo-graylist `_POSSIBLE_OPCODES_P3`, was intended as a
  list of new P3 opcodes we might want to add, but the purpose was
  hard to understand and a proper blacklist makes way more sense

closes odoo/odoo#41085

Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
2020-03-23 08:56:21 +00:00
Lucas Perais (lpe) 69b51ce05e [FIX] translate: do not export OWL directive's attribute
Have a static template of the form:

```xml
<div t-name="ComponentA">
 <ComponentB title="no export" />
</div>
```
To be used by OWL (github.com/odoo/owl)
In this context, the title in the attributes of ComponentB
should not be considered as *the* title attribute of HTML
but as a sort of variable declaration

Before this commit, the string contained in the value of this
title attribute was exported. It was not translated at runtime,
but we want to export as less stuff as possible in PO files

So, after this commit, none of the attributes held by a OWL directive
are exported

This commit relies on OWL Syntax, which makes mandatory
for a component directive's first letter to be capitalized
https://github.com/odoo/owl/blob/master/doc/reference/component.md#composition

Also, it relies on the good practice and widespread convention to have
every HTML node lower cased
https://www.w3schools.com/html/html5_syntax.asp

closes odoo/odoo#46191

Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>
2020-04-17 12:55:32 +00:00
Romeo Fragomeli fbd1d7584a [FIX] tests,tools: remove unused code
Due to this commit: odoo/odoo@0ea67467b1 ( https://github.com/odoo/odoo/blob/0ea67467b1301b39a263a386a622e931784eb0de/odoo/tools/config.py#L521 )

the screencasts value is not a string anymore but a valid PATH.

So now you can't set '1', 'true' or 't' to force to have the
same directory as the screenshot dir.

Steps to reproduce:
odoo-bin ... --screencasts 1 (with a failed JS test to produce a screencast)

closes odoo/odoo#49665

X-original-commit: ab1bf59a3b78f3c0857755cb2439636b6df297c3
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
Signed-off-by: rfr-odoo <rfr-odoo@users.noreply.github.com>
2020-04-16 13:28:07 +00:00
Xavier Morel 1ecb0641ef [FIX] core: calling read_group / name_search over xmlrpc
Also non-browser jsonrpc (as it goes through a similar process): for
internal performance reasons, name_search and read_group have been
converted to a *lazy* name_get, so the "display name" is not
unnecessarily computed.

However this is an issue for the RPC endpoints (/xmlrpc and /jsonrpc)
as they have no support for `lazy` and thus tend to blow up and / or
do the wrong thing when trying to output a lazy:

* xmlrpc has no way to handle lazy at all and straight blows up
* jsonrpc falls back to `json_default` so they try to stringify the
  lazy, which might have worked except

*Problematically* both endpoints delegate the actual work to
`dispatch_rpc` which handles dispatching between various services and
ultimately creates a *new* cursor before calling model
methods (`object` service and `execute`/`execute_kw`).

This means by the time the result is serialized to be output, the
lazy's cursor has long been closed, and thus any access to an
unevaluated `lazy` errors out when trying to fetch the underlying
item.

This also means we can't just add a hook to serialize the lazy
in the xmlrpc marshaller, though we do have to do that. We *also* (for
both xmlrpc and jsonrpc) have to force evluation of lazy values before
our cursor is closed, meaning it has to be done right after the method
is invoked, iterating the entire response.

Related to task 2170343

closes odoo/odoo#49286

X-original-commit: e2b5a359c1d5eccbe725c1c3169b4130d7bca49b
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-04-09 09:06:34 +00:00
Julien Castiaux ab4000fb3c [REF] base: Remove deprecated exceptions and osv
TL;DR: remember `osv` and `except_orm` ? You can forget about them.

* Deprecated `except_orm` dropped.
* `UserError` elevated as super type of all user-related
  errors.
* Unused `DeferredException` dropped.
* Unused `QWebException` dropped (real one is in `qweb.py`).
* `MailDeliveryException` made a python exception.
* `name` legacy exception attribute made an alias of the python standard
  `args[0]` attribute and deprecated.
* `value` legacy exception attribute dropped.
* `exception_type` RPC error response key dropped.
* Deprecated `osv` module dropped.
* `--osv-memory-age-limit` cli option made an alias of
  `--transient-age-limit` and deprecated.

The `odoo.exceptions.Warning` have long been a deprecated alias to
`UserError`. It is going to be removed in a future version but first we
explicitly deprecate it with a warning.

The `odoo.exceptions.DeferredException` was a very old internal
exception, it has been removed without deprecation notice as it is never
raised.

The `odoo.exceptions.except_orm` has been a deprecated exception type
with deprecation warning for 5 years, it has been removed in favor of
UserError which becomes the super class of all user-related errors.

The `odoo.base.models.ir_mail_server.MailDeliveryException` was
inheriting `except_orm`. As it is not related to a user error but is
more of a problem an admin much take care of, the exception has been
made a Python error.

The `exception_type` JSON key in RPC error responses was holding an
hardcoded value derived from the exception type. Its usage has been
dropped in favor of the `name` JSON key that holds the precise exception
name. Again as it was hardly used in the source code (beside the crash
manager) it has been dropped without deprecation warning.

Since we are here trying to clean odoo custom exceptions, we are also
deprecating the `name` exception attribute in favor of the more standard
`args[0]` attribute.

The `name` (along with `value`) were two attributes used to raise
`except_orm` exceptions before the introduction of `UserError`,
`AccessError` and related exceptions. The `name` attribute, at the time,
was holding the exception type/title. Nowadays it contains the error
message. The `value` attribute, at the time, was holding the error
message. Nowadays it is no more used.

The `osv` module contains very old deprecated aliases. There is no
simple way to log a deprecation warning for osv, osv_memory and
osv_abstract but as they have not been in use for ages, they have been
removed too. To be consistent, the `--osv-memory-age-limit` cli option
has been made a deprecated alias to the `--transient-age-limit`.

closes odoo/odoo#45723

Task: 2187728
Related: odoo/enterprise#9162
Signed-off-by: Raphael Collet (rco) <rco@openerp.com>
2020-04-08 08:41:17 +00:00
Xavier Morel 9d37cecfca [FIX] core: iterations on LRU
Iteration methods on LRU were removed because they were not
thread-safe and it's not clear that making them thread-safe is the
correct thing to do, so not providing them seems saner.

I thought I'd looked for usages of the LRU but apparently didn't look
hard enough as I missed that it's used by the cron workers (apparently
using the threaded server we only run crons for dbs currently living
in the registry cache, the more you know).

Convert these to iterating on the LRU's internal mapping, and also
don't iterate on the LRU to clear its entries one by one when we can
just clear the entire thing safely, although Registry.delete_all
really seems completely unused.

closes odoo/odoo#49023

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-04-06 09:45:29 +00:00
Xavier Morel 1b30a78beb [IMP] core: rebuild LRU on OrderedDict
That's pretty much the entire point of OrderedDict's existence.

Also removed the iteration methods: they're not used and they're not
thread-safe, since they're written as generators they "return"
instantaneously when called immediately unlocking, then the iteration
itself is unprotected, so the entire thing is useless.

They could be reimplemented with explicit locking (a `with self._lock`
internally) if there's ever a use case for it, but that doesn't seem
to be the case at the moment.

closes odoo/odoo#46061

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-04-03 13:16:46 +00:00
Raphael Collet 3ecb6fdcb0 [IMP] core: optimize model._auto_init() on a new model
Create the table with all the columns from scratch, with the NOT NULL
constraint when required.  Also do not call `_check_removed_columns()`
on a new table.

This saves 0.5% of the total installation time.

X-original-commit: 0727cacf5194a143b15ab4cb9893f3035a67be1f
2020-04-03 11:40:28 +00:00
Xavier Morel 6c4e4981bc [IMP] core: remove odoo.tools.safe_eval._get_opcodes
It's not necessary anymore as all supported Python versions implement
get_instructions, and inlining the usage of that is as readable as
calling _get_opcodes.

Also use the subset/superset predicate for validity testing instead of
difference as it's a fair bit faster:

    ❯ python3.8 -mtimeit -s 's1 = set(range(10)); s2 = set(range(5))' 's2 - s1'
    5000000 loops, best of 5: 88.3 nsec per loop
    ❯ python3.8 -mtimeit -s 's1 = set(range(10)); s2 = set(range(5, 15))' 's2 - s1'
    2000000 loops, best of 5: 159 nsec per loop
    ❯ python3.8 -mtimeit -s 's1 = set(range(10)); s2 = set(range(5))' 's1 >= s2'
    5000000 loops, best of 5: 71.1 nsec per loop
    ❯ python3.8 -mtimeit -s 's1 = set(range(10)); s2 = set(range(5, 15))' 's1 >= s2'
    5000000 loops, best of 5: 53.6 nsec per loop

we're paying double in the failure case but that doesn't super duper matter
because we're raising an exception and bailing out, the 24% gain on the
happy path seems more relevant.

closes odoo/odoo#48948

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-04-03 09:35:08 +00:00
Adrian Torres 5952928b42 [REM] *: remove various unused import shims
Before this commit, a lot of leftover import shims existed in the
codebase for py2-py3 compatibility, these are no longer needed since
Odoo 13.0+ doesn't support Python 2 anymore and is (finally) in EOL.

With this commit, these shims are dropped, making the code cleaner,
easier to read and with one less dependency.

Queue -> queue -> py2-py3 compatibility
xmlrpclib -> xmlrpc.client -> py2-py3 compatibility
ConfigParser -> configparser -> py2-py3 compatibility
itertools.izip_longest -> itertools.zip_longest -> py2-py3 compatibility
urllib -> urllib.request -> py2-py3 compatibility
__builtins__ -> builtins -> py2-py3 compatibility
_winreg -> winreg -> py2-py3 compatibility

mock -> unittest.mock -> merged into CPython

The debian/fedora packages and requirements.txt have been updated accordingly

closes odoo/odoo#44601

Related: odoo/enterprise#8141
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-04-01 12:45:40 +00:00
fja-odoo 388c222c6c [IMP] base, *: allow html fields not to sanitize forms
* = event, website_event_track, website_sale, website_hr_recruitment,
website_livechat, website_sale, website_slides

The option to sanitize or not the forms was not available, this will
allow better flexibility on whether forms should be sanitized or not on
an HTML field.
Also we use this new param to allow forms to be added on some already
existing html fields where forms where sanitized out.

task-2209554

closes odoo/odoo#47318

Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
2020-03-31 14:22:43 +00:00
Christophe Monniez 748cc7e501 [IMP] base, lunch: populate res.users and lunch models
closes odoo/odoo#48636

X-original-commit: f3f2a1e4723240b6396cea943fc86291c16f4c9d
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>
2020-03-30 21:30:15 +00:00
Xavier-Do f545fd274d [IMP] core, base: add tooling to populate database
Some use case like testing performance or upgrade scripts required a database
with prefilled data, covering basic corner cases. A solution can be to
create data a procedural way.

This commit proposes an API to easily populate a model, usually by giving a
list of possible values for each field or by giving a compute method that will
be based on raw values of other fields.

The basic way to define how to populate a new field is to override `_populate_factories`,
a method that returns a sequence of pairs `(field_name, factory)`.

The definition of a field is a "factory", a function that returns a neverending iterator
combining its value(s) with the values of the iterator given in parameter.
Some factory helpers are given in `tools.populate.py`:
- `iterate(vals, weighs)` ensures that one record is created for each value
by iterating on them, then resumes as `random.choice` on those vals following weights
once the first iteration is finished.
- `cartesian(vals, weights)` makes a cartesian product of its own values with the values
of its input iterator, then resumes as a randomized generator.
- `compute(function)` calls the given function with the current values dict and a random object,
and assigns the current field to the returned value.
- ...

Each iterator yields dictionaries of field values, and the factory should add a
value for the current field(s).  The yielded dictionaries also contain a pseudo_field
`"__complete"`, that indicates whether this step is some randomized data
to reach the expected count of records.  A falsy value indicates that the iterator
is still covering mandatory cases.  This indicates whether a cartesian product is
finished, or an `iterate` has consumed all its values.

The order of the factories is quite important, since some computed fields may need
other fields to be defined, and `cartesian` factories should always be at the beginning
to avoid having too many combination.  That is why the factories are given as a list of
pairs instead of a dictionary; this makes it easier to insert elements at any place.

Example:
field A: cartesian([T, F])
field B: cartesian([0, 1])
field C: iterate([a, b, c, d, e])
field D: compute(1-B)

_c is shortcut for __complete
_ is a random value, or result of a random value

```
iter | root  | field A  | field B | field C     | field D   | result
0     {_c:F}  {... A:T}  {... B:0} {...C:a}      {...D:1}    T,0,a,1 complete:False
                         {... B:1} {...C:b}      {...D:0}    T,1,b,0 complete:False
              {... A:F}  {... B:0} {...C:c}      {...D:1}    F,0,c,1 complete:False
                         {... B:1} {...C:d}      {...D:0}    F,1,d,0 complete:False
1     {_c:T}  {... A:_}  {... B:_} {...C:e,_c:F} {...D:_}    _,_,e,_ complete:False
2     {_c:T}  {... A:_}  {... B:_} {...C:_}      {...D:_}    _,_,_,_ complete:True
```

X-original-commit: 4c0182dafa584853ed83a166096f45c33c06a245
2020-03-30 21:30:13 +00:00
Nicolas Lempereur f3182c6035 [FIX] translate.py,web: aria-label qweb translation and export
In 12.0 9de1bc0e aria-label translation was added for python views, but:

- aria-label attribute are not exported

- aria-label are not translated for JS view rendering

So by default, aria-label would only be translated for python views that
had the same aria-label term in a translatable element (eg. "title"
attribute).

opw-2222504
closes #48255

closes odoo/odoo#48275

X-original-commit: 6b6b11ea2f8391f1f195bd1650531f77b6bed729
Signed-off-by: Nicolas Lempereur (nle) <nle@odoo.com>
2020-03-24 14:46:30 +00:00
Thibault Delavallée aaefa35e29 [MOV] link_tracker, various: more link shortening tools to mail.render.mixin
PURPOSE

Move links shortening tools on mail.render.mixin to have rendering and
post procesing tools available on that mixin.

SPECIFICATIONS

Move html and text links shortening methods on mail.render.mixin class.
That way rendering and post processing is done in that mixin instead of
being specific to link.tracker model. Indeed those methods are tools and
not really business related methods.

Future commits will improve the way mail and sms composer use the render
mixin. This commit is therefore both a cleaning and preparing commit.

LINKS

Task ID 1963529
Community PR odoo/odoo#32397
2020-03-24 10:25:04 +00:00
Xavier Morel 43f10313c6 [IMP] *: automatically brand PDFs (from PyPDF2) as Odoo
To avoid having to fixup half a dozen places where we're creating PDF
writers, and possibly ending up with new ill-configured writers in
the future, patch PyPDF2's own writer with a subclass setting /Creator
and /Producer.

Note that this will not affect non-post-processed PDFs generated by
wkhtmltopdf. wkhtmltopdf does not allow setting these properties[0][1], so
to fix this issue we'd have to alter _run_wkhtmltopdf to pass the
result through PyPDF2 in order to alter its metadata.

[0] https://github.com/wkhtmltopdf/wkhtmltopdf/issues/2000
[1] https://bugreports.qt.io/browse/QTBUG-44451

closes odoo/odoo#29460

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-03-23 14:41:52 +00:00
Xavier Morel fe376d50d0 [FIX] core, base: leftover deprecation warnings
* from collections import <ABC> is deprecated, unclear why the
  deprecation warning didn't appear before (possibly only appears in
  3.7/3.8?) either way `collections.abc` should be 3.3+ so switch
  everything to it.
* add some more ignores on third-party packages deprecation
  warnings (meh)
* while at it, mitigate generation of non-breaking space on some
  versions of Babel (in the french locale used by our tests anyway)

closes odoo/odoo#47581

Related: odoo/enterprise#9214
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>
2020-03-19 09:32:21 +00:00
Romain Derie 96d3fa4e01 [IMP] base, website: add ir.ui.view action to compare arch (wizard)
This commit adds the possibility to compare a view arch to another one.
The result will be shown in a diff viewer (github like).

This is following what was done at #32009

task-2190072

closes odoo/odoo#44646

Signed-off-by: Jérémy Kersten (jke) <jke@openerp.com>
2020-03-17 15:58:45 +00:00
Xavier-Do b2b36524c2 [IMP] core: improve module loading logs
Performances from a general point of view can be difficult to track.
This commit proposes to improve logs in two ways:

The current logs only use the sql_counter, wich will only be updated
when a cursor is closed. In a test-enable install, this counter
is actually the queries of the tests wince the install cursor is
open untill the end. The first fix is to use bot sql_counter and
sql_log_count to have total queries untill now on closed cursor,
but also the current number of queries of the current cursor.

This means that the new log format will be
{nb} modules loaded in {time}, {loading_querie} (+{test_cr_queries}) queries
instead of
{nb} modules loaded in {time}, {tests_cr__queries}queries

Nothe that in the current version, {nb} is actually the total number of
loaded modules until now.

This commit also add an equivalent end log by module and change the
loglevel of module start on install (mainly usefull if an error occurs
before anything else is logged hidding the module causing this error.)

A cleaner runbot logger is also added, in order to be abble to call
_logger.runbot( instead of _logger.log(25. This will clarify the purpose
of such a log level.

closes odoo/odoo#47283

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>
2020-03-16 13:21:57 +00:00
Rémy Voet (ryv) c5e6511e8d [IMP] mail: ease bounce email content update based on alias security
PURPOSE

When incoming emails bounce due to alias security bounce email is quite
generic. Purpose of this task is to ease its customization and update.

SPECIFICATIONS

In order to improve the flexibility of alias, add a customizable html field
on the alias model. This html content will be send as bounce email core content
in case of bounced/unauthorized mail received for this alias,

Obviously it has no effect on 'everyone' security setting as no email will
bounce due to that issue.

If it is not set a default generic mail will be send depending on security
setting. It allows to keep void html fields when no specific bounce content
is required

In HR, an old template allowing some light customization for employee based
security option is removed as it is completely replaced by the new feature.

Also add references message-id of the mail received to the answer so that
threads are correctly set.

LINKS

Task ID 2126509
2020-03-12 13:54:32 +00:00
Rémy Voet 974f512f5f [IMP] tools: add a tool method to find if an html content is empty
In odoo empty html is not always completely empty. This tool method allow
to consider html holding void tags as empty html. This happens notably when
using the editor that may store <p><br><p> even if displayed content is void.

Task ID 2126509
PR #41827
2020-03-12 13:54:32 +00:00
jbm-odoo 568b258c43 [IMP] tools: rotate PDF
Task ID 1251

closes odoo/odoo#45764

Related: odoo/enterprise#8569
Signed-off-by: Yannick Tivisse (yti) <yti@odoo.com>
2020-03-11 09:57:55 +00:00