Files
odoo_source/addons/mail_plugin/controllers/mail_plugin.py
T
ef00294e71 [IMP] core: store translated fields as JSONB columns
Translated fields no longer use the model ir.translation.  Instead they store
all their values as JSON, and store them into JSONB columns in the model's
table.  The field's column value is either NULL or a JSON dict mapping language
codes to text (the field's value in the corresponding language), and must
contain an entry for key 'en_US' (as it is used as a fallback for all other
languages).  Empty text is allowed in translation values, but not NULL.

Here are examples for a field with translate=True:

    NULL
    {"en_US": "Foo"}
    {"en_US": "Foo", "fr_FR": "Bar", "nl_NL": "Baz"}
    {"en_US": "Foo", "fr_FR": "", "nl_NL": "Baz"}

Like before, writing False to the field makes it NULL, i.e., False in all
languages.  However, writing "" to the field makes its value empty in the
current language, but does not discard the values in the other languages.

Here are examples for a field with translate=xml_translate:

    NULL
    {"en_US": "<div>Foo<p>Bar</p></div>", "fr_FR": "<div>Fou<p>Barre</p></div>"}

Change for callable(translate) fields: one can now write any value in any
language on such a field.  The new value will be adapted in all languages, based
on the mapping of terms between languages in the old values.  Basically the
structure of the value must remain the same in all languages, like before.

Reading a translated field is now both simpler and faster than the former
implementation.  We fetch the value of the field in the current language by
coalescing its value with the 'en_US' value of the field:

    SELECT id, COALESCE(name->>'fr_FR', name->>'en_US') AS name ...

The raw cache of the field contains either None or a dict which is conceptually
a subset of the JSON value in database (except for missing languages).  For the
sake of simplicity, most cache operations deal with the dict and return the text
value in the current language.

Trigram indexes have been adapted to the new storing strategy, and should enable
to search in any language.  Before this change, only the source value of the
field ('en_US') could be indexed.

Computed stored translated fields are not supported by the framework, because of
the complexity of the computation itself: the field would need to be computed in
all active languages.  We chose to not provide any hook to compute a field in
all languages at once, and the framework always invokes a compute method once to
recompute it.

Code translations are no longer stored into the database.  They become static,
and are extracted from the PO files when needed.  The worker simply uses a cache
with extracted code translations for performance.  This is reasonable, since
fr_FR code translations for all modules takes around 2MB of memory, and the
cache can be shared among all registries in the worker.  Changing code
translations requires to update the corresponding PO file and reloading the
worker(s).

Performance summary:
 (+) reading 'model' translated fields is faster
 (+) reading 'model_terms' translated fields is much faster (no need to inject
     translations into the source value)
 (+) searching translated fields with operator 'ilike' is much faster when the
     field is indexed with 'trigram'
 (+) updating translated fields requires less ORM flushing
 (-) importing translations from PO files is 2x slower

Some extra fixes:
 - make field 'name' of ir.actions.actions translated; because of the PG
   inheritance, this is necessary to make the column definition consistent in
   all models that inherit from ir.actions.actions.
 - add some backend API for the web/website client for editing translations
 - move methods get_field_string() to model ir.model.fields
 - move _load_module_terms to model ir.module.module
 - adapt tests in test_impex, test_new_api
 - because env.lang is injected into SQL queries, its returned value is
   now guaranteed to correspond to a valid active language or None
 - remove wizard to insert missing translations (no longer makes sense)

task-id: 2081307

Co-authored-by: Fabien Pinckaers <fp@openerp.com>
Co-authored-by: Raphael Collet <rco@odoo.com>
2022-09-15 22:37:50 +02:00

446 lines
18 KiB
Python

# -*- coding: utf-8 -*-
# Part of Odoo. See LICENSE file for full copyright and licensing details.
import base64
import json
import logging
import requests
from werkzeug.exceptions import Forbidden
from odoo import http, tools, _
from odoo.addons.iap.tools import iap_tools
from odoo.exceptions import AccessError
from odoo.http import request
_logger = logging.getLogger(__name__)
class MailPluginController(http.Controller):
@http.route('/mail_client_extension/modules/get', type="json", auth="outlook", csrf=False, cors="*")
def modules_get(self, **kwargs):
"""
deprecated as of saas-14.3, not needed for newer versions of the mail plugin but necessary
for supporting older versions
"""
return {'modules': ['contacts', 'crm']}
@http.route('/mail_plugin/partner/enrich_and_create_company',
type="json", auth="outlook", cors="*")
def res_partner_enrich_and_create_company(self, partner_id):
"""
Route used when the user clicks on the create and enrich partner button
it will try to find a company using IAP, if a company is found
the enriched company will then be created in the database
"""
partner = request.env['res.partner'].browse(partner_id).exists()
if not partner:
return {'error': _("This partner does not exist")}
if partner.parent_id:
return {'error': _("The partner already has a company related to him")}
normalized_email = partner.email_normalized
if not normalized_email:
return {'error': _('The email of this contact is not valid and we can not enrich it')}
company, enrichment_info = self._create_company_from_iap(normalized_email)
if company:
partner.write({'parent_id': company})
return {
'enrichment_info': enrichment_info,
'company': self._get_company_data(company),
}
@http.route('/mail_plugin/partner/enrich_and_update_company', type='json', auth='outlook', cors='*')
def res_partner_enrich_and_update_company(self, partner_id):
"""
Enriches an existing company using IAP
"""
partner = request.env['res.partner'].browse(partner_id).exists()
if not partner:
return {'error': _("This partner does not exist")}
if not partner.is_company:
return {'error': 'Contact must be a company'}
normalized_email = partner.email_normalized
if not normalized_email:
return {'error': 'The email of this contact is not valid and we can not enrich it'}
domain = tools.email_domain_extract(normalized_email)
iap_data = self._iap_enrich(domain)
if 'enrichment_info' in iap_data: # means that an issue happened with the enrichment request
return {
'enrichment_info': iap_data['enrichment_info'],
'company': self._get_company_data(partner),
}
phone_numbers = iap_data.get('phone_numbers')
partner_values = {}
if not partner.phone and phone_numbers:
partner_values.update({'phone': phone_numbers[0]})
if not partner.iap_enrich_info:
partner_values.update({'iap_enrich_info': json.dumps(iap_data)})
if not partner.image_128:
logo_url = iap_data.get('logo')
if logo_url:
try:
response = requests.get(logo_url, timeout=2)
if response.ok:
partner_values.update({'image_1920': base64.b64encode(response.content)})
except Exception:
pass
model_fields_to_iap_mapping = {
'street': 'street_name',
'city': 'city',
'zip': 'postal_code',
'website': 'domain',
}
# only update keys for which we dont have values yet
partner_values.update({
model_field: iap_data.get(iap_key)
for model_field, iap_key in model_fields_to_iap_mapping.items() if not partner[model_field]
})
partner.write(partner_values)
partner.message_post_with_view(
'iap_mail.enrich_company',
values=iap_data,
subtype_id=request.env.ref('mail.mt_note').id,
)
return {
'enrichment_info': {'type': 'company_updated'},
'company': self._get_company_data(partner),
}
@http.route(['/mail_client_extension/partner/get', '/mail_plugin/partner/get']
, type="json", auth="outlook", cors="*")
def res_partner_get(self, email=None, name=None, partner_id=None, **kwargs):
"""
returns a partner given it's id or an email and a name.
In case the partner does not exist, we return partner having an id -1, we also look if an existing company
matching the contact exists in the database, if none is found a new company is enriched and created automatically
old route name "/mail_client_extension/partner/get is deprecated as of saas-14.3, it is not needed for newer
versions of the mail plugin but necessary for supporting older versions, only the route name is deprecated not
the entire method.
"""
if not (partner_id or (name and email)):
return {'error': _('You need to specify at least the partner_id or the name and the email')}
if partner_id:
partner = request.env['res.partner'].browse(partner_id).exists()
return self._get_contact_data(partner)
normalized_email = tools.email_normalize(email)
if not normalized_email:
return {'error': _('Bad Email.')}
# Search for the partner based on the email.
# If multiple are found, take the first one.
partner = request.env['res.partner'].search(['|', ('email', 'in', [normalized_email, email]),
('email_normalized', '=', normalized_email)], limit=1)
response = self._get_contact_data(partner)
# if no partner is found in the database, we should also return an empty one having id = -1, otherwise older versions of
# plugin won't work
if not response['partner']:
response['partner'] = {
'id': -1,
'email': email,
'name': name,
'enrichment_info': None,
}
company = self._find_existing_company(normalized_email)
can_create_partner = request.env['res.partner'].check_access_rights('create', raise_exception=False)
if not company and can_create_partner: # create and enrich company
company, enrichment_info = self._create_company_from_iap(normalized_email)
response['partner']['enrichment_info'] = enrichment_info
response['partner']['company'] = self._get_company_data(company)
return response
@http.route('/mail_plugin/partner/search', type="json", auth="outlook", cors="*")
def res_partners_search(self, search_term, limit=30, **kwargs):
"""
Used for the plugin search contact functionality where the user types a string query in order to search for
matching contacts, the string query can either be the name of the contact, it's reference or it's email.
We choose these fields because these are probably the most interesting fields that the user can perform a
search on.
The method returns an array containing the dicts of the matched contacts.
"""
normalized_email = tools.email_normalize(search_term)
if normalized_email:
filter_domain = [('email_normalized', 'ilike', search_term)]
else:
filter_domain = ['|', '|', ('display_name', 'ilike', search_term), ('ref', '=', search_term),
('email', 'ilike', search_term)]
# Search for the partner based on the email.
# If multiple are found, take the first one.
partners = request.env['res.partner'].search(filter_domain, limit=limit)
partners = [
self._get_partner_data(partner)
for partner in partners
]
return {"partners": partners}
@http.route(['/mail_client_extension/partner/create', '/mail_plugin/partner/create'],
type="json", auth="outlook", cors="*")
def res_partner_create(self, email, name, company):
"""
params email: email of the new partner
params name: name of the new partner
params company: parent company id of the new partner
"""
# old route name "/mail_client_extension/partner/create is deprecated as of saas-14.3,it is not needed for newer
# versions of the mail plugin but necessary for supporting older versions
# TODO search the company again instead of relying on the one provided here?
# Create the partner if needed.
partner_info = {
'name': name,
'email': email,
}
#see if the partner has a parent company
if company and company > -1:
partner_info['parent_id'] = company
partner = request.env['res.partner'].create(partner_info)
response = {'id': partner.id}
return response
@http.route('/mail_plugin/log_mail_content', type="json", auth="outlook", cors="*")
def log_mail_content(self, model, res_id, message, attachments=None):
"""Log the email on the given record.
:param model: Model of the record on which we want to log the email
:param res_id: ID of the record
:param message: Body of the email
:param attachments: List of attachments of the email.
List of tuple: (filename, base 64 encoded content)
"""
if model not in self._mail_content_logging_models_whitelist():
raise Forbidden()
if attachments:
attachments = [
(name, base64.b64decode(content))
for name, content in attachments
]
request.env[model].browse(res_id).message_post(body=message, attachments=attachments)
return True
@http.route('/mail_plugin/get_translations', type="json", auth="outlook", cors="*")
def get_translations(self):
return self._prepare_translations()
def _iap_enrich(self, domain):
"""
Returns enrichment data for a given domain, in case an error happens the response will
contain an enrichment_info key explaining what went wrong
"""
enriched_data = {}
try:
response = request.env['iap.enrich.api']._request_enrich({domain: domain}) # The key doesn't matter
except iap_tools.InsufficientCreditError:
enriched_data['enrichment_info'] = {'type': 'insufficient_credit', 'info': request.env['iap.account'].get_credits_url('reveal')}
except Exception:
enriched_data["enrichment_info"] = {'type': 'other', 'info': 'Unknown reason'}
else:
enriched_data = response.get(domain)
if not enriched_data:
enriched_data = {'enrichment_info': {'type': 'no_data', 'info': 'The enrichment API found no data for the email provided.'}}
return enriched_data
def _find_existing_company(self, email):
"""Find the company corresponding to the given domain and its IAP cache.
:param email: Email of the company we search
:return: The partner corresponding to the company
"""
search = self._get_iap_search_term(email)
partner_iap = request.env["res.partner.iap"].sudo().search([("iap_search_domain", "=", search)], limit=1)
if partner_iap:
return partner_iap.partner_id
return request.env["res.partner"].search([("is_company", "=", True), ("email_normalized", "=ilike", "%" + search)], limit=1)
def _get_company_data(self, company):
if not company:
return {'id': -1}
fields_list = ['id', 'name', 'phone', 'mobile', 'email', 'website']
company_values = dict((fname, company[fname]) for fname in fields_list)
company_values['address'] = {'street': company.street,
'city': company.city,
'zip': company.zip,
'country': company.country_id.name if company.country_id else ''}
company_values['additionalInfo'] = json.loads(company.iap_enrich_info) if company.iap_enrich_info else {}
company_values['image'] = company.image_1920
return company_values
def _create_company_from_iap(self, email):
domain = tools.email_domain_extract(email)
iap_data = self._iap_enrich(domain)
if 'enrichment_info' in iap_data:
return None, iap_data['enrichment_info']
phone_numbers = iap_data.get('phone_numbers')
emails = iap_data.get('email')
new_company_info = {
'is_company': True,
'name': iap_data.get("name") or domain,
'street': iap_data.get("street_name"),
'city': iap_data.get("city"),
'zip': iap_data.get("postal_code"),
'phone': phone_numbers[0] if phone_numbers else None,
'website': iap_data.get("domain"),
'email': emails[0] if emails else None
}
logo_url = iap_data.get('logo')
if logo_url:
try:
response = requests.get(logo_url, timeout=2)
if response.ok:
new_company_info['image_1920'] = base64.b64encode(response.content)
except Exception as e:
_logger.warning('Download of image for new company %s failed, error %s', new_company_info.name, e)
if iap_data.get('country_code'):
country = request.env['res.country'].search([('code', '=', iap_data['country_code'].upper())])
if country:
new_company_info['country_id'] = country.id
if iap_data.get('state_code'):
state = request.env['res.country.state'].search([
('code', '=', iap_data['state_code']),
('country_id', '=', country.id)
])
if state:
new_company_info['state_id'] = state.id
new_company_info.update({
'iap_search_domain': self._get_iap_search_term(email),
'iap_enrich_info': json.dumps(iap_data),
})
new_company = request.env['res.partner'].create(new_company_info)
new_company.message_post_with_view(
'iap_mail.enrich_company',
values=iap_data,
subtype_id=request.env.ref('mail.mt_note').id,
)
return new_company, {'type': 'company_created'}
def _get_partner_data(self, partner):
fields_list = ['id', 'name', 'email', 'phone', 'mobile', 'is_company']
partner_values = dict((fname, partner[fname]) for fname in fields_list)
partner_values['image'] = partner.image_128
partner_values['title'] = partner.function
partner_values['enrichment_info'] = None
try:
partner.check_access_rights('write')
partner.check_access_rule('write')
partner_values['can_write_on_partner'] = True
except AccessError:
partner_values['can_write_on_partner'] = False
if not partner_values['name']:
# Always ensure that the partner has a name
name, email = request.env['res.partner']._parse_partner_name(
partner_values['email'])
partner_values['name'] = name or email
return partner_values
def _get_contact_data(self, partner):
"""
method used to return partner related values, it can be overridden by other modules if extra information have to
be returned with the partner (e.g., leads, ...)
"""
if partner:
partner_response = self._get_partner_data(partner)
if partner.company_type == 'company':
partner_response['company'] = self._get_company_data(partner)
elif partner.parent_id:
partner_response['company'] = self._get_company_data(partner.parent_id)
else:
partner_response['company'] = self._get_company_data(None)
else: # no partner found
partner_response = {}
return {
'partner': partner_response,
'user_companies': request.env.user.company_ids.ids,
'can_create_partner': request.env['res.partner'].check_access_rights(
'create', raise_exception=False),
}
def _mail_content_logging_models_whitelist(self):
"""
Returns all models that emails can be logged to and that can be used by the "log_mail_content" method,
it can be overridden by sub modules in order to whitelist more models
"""
return ['res.partner']
def _get_iap_search_term(self, email):
"""Return the domain or the email depending if the domain is blacklisted or not.
So if the domain is blacklisted, we search based on the entire email address
(e.g. asbl@gmail.com). But if the domain is not blacklisted, we search based on
the domain (e.g. bob@sncb.be -> sncb.be)
"""
domain = tools.email_domain_extract(email)
return ("@" + domain) if domain not in iap_tools._MAIL_DOMAIN_BLACKLIST else email
def _translation_modules_whitelist(self):
"""
Returns the list of modules to be translated
Other mail plugin modules have to override this method to include their module names
"""
return ['mail_plugin']
def _prepare_translations(self):
lang = request.env['res.users'].browse(request.uid).lang
translations_per_module = request.env["ir.http"].get_translations_for_webclient(
self._translation_modules_whitelist(), lang)[0]
translations_dict = {}
for module in self._translation_modules_whitelist():
translations = translations_per_module.get(module, {})
messages = translations.get('messages', {})
for message in messages:
translations_dict.update({message['id']: message['string']})
return translations_dict