pengyuthon/odoo_source
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[IMP] doc: add information on the new populate feature.

Browse Source 
* Cmdline interface: how to trigger database population
* Testing: how to implement database population on a given model.
  * autodocumentation of the population methods

+ improve population methods docstrings.

closes odoo/odoo#50596

Signed-off-by: Victor Feyens (vfe) <vfe@odoo.com>
This commit is contained in:
Victor Feyens
2020-05-13 15:47:04 +00:00
parent 55e7b6bbcd
commit d2652e971a
4 changed files with 206 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/reference/cmdline.rst
+33 -3
View File
@@ -536,6 +536,10 @@ starting requirements are.
Scaffolding is available via the :command:`odoo-bin scaffold` subcommand.
.. code-block:: console
$ odoo_bin scaffold my_module /addons/
.. option:: name (required)
the name of the module to create, may munged in various manners to
@@ -551,8 +555,34 @@ Scaffolding is available via the :command:`odoo-bin scaffold` subcommand.
a template directory, files are passed through jinja2_ then copied to
the ``destination`` directory
.. code-block:: console
$ odoo_bin scaffold my_module /addons/
This will create module *my_module* in directory */addons/*.
.. _reference/cmdline/populate:
Database Population
===================
.. program:: odoo-bin populate
Odoo CLI supports database population features. If the feature is
:ref:`implemented on a given model<reference/testing/populate/methods>`, it allows automatic data
generation of the model's records to test your modules in databases containing non-trivial amounts of records.
.. code-block:: console
$ odoo_bin populate
.. option:: --models
list of models for which the database should be filled
.. option:: --size (small|medium|large)
population size, the actual records number depends on the model's `_populate_sizes` attribute.
The generated records content is specified by the :meth:`~odoo.models._populate_factories` method
of a given model (cf. the :file:`populate` folder of modules for further details).
.. seealso::
:ref:`reference/testing/populate`
doc/reference/testing.rst
+99
View File
@@ -526,6 +526,105 @@ Two new command line arguments were added since Odoo 13.0 to control this behavi
:option:`--screenshots <odoo-bin --screenshots>` and :option:`--screencasts <odoo-bin --screencasts>`
Performance Testing
===================
Query counts
------------
One of the ways to test performance is to measure database queries. Manually, this can be tested with the
`--log-sql` CLI parameter. If you want to establish the maximum number of queries for an operation,
you can use the :meth:`~odoo.tests.common.BaseCase.assertQueryCount` method, integrated in Odoo test classes.
.. code-block:: python
with self.assertQueryCount(11):
do_something()
.. _reference/testing/populate:
Database population
-------------------
Odoo CLI offers a :ref:`database population<reference/cmdline/populate>` feature.
.. code-block:: console
odoo-bin populate
Instead of the tedious manual, or programmatic, specification of test data,
one can use this feature to fill a database on demand with the desired number of test data.
This can be used to detect diverse bugs or performance issues in tested flows.
To specify this feature for a given model, the following methods and attributes can be defined.
.. _reference/testing/populate/methods:
.. currentmodule:: odoo.models
.. autoattribute:: Model._populate_sizes
.. autoattribute:: Model._populate_dependencies
.. automethod:: Model._populate
.. automethod:: Model._populate_factories
.. note::
You have to define at least :meth:`~odoo.models.Model._populate` or :meth:`~odoo.models.Model._populate_factories`
on the model to enable database population.
Example model
^^^^^^^^^^^^^
.. code-block:: python
from odoo.tools import populate
class CustomModel(models.Model)
_inherit = "custom.some_model"
_populate_sizes = {"small": 100, "medium": 2000, "large": 10000}
_populate_dependencies = ["custom.some_other_model"]
def _populate_factories(self):
# Record ids of previously populated models are accessible in the registry
some_other_ids = self.env.registry.populated_models["custom.some_other_model"]
def get_some_field(values=None, random=None, **kwargs):
""" Choose a value for some_field depending on other fields values.
:param dict values:
:param random: seeded :class:`random.Random` object
"""
field_1 = values['field_1']
if field_1 in [value2, value3]:
return random.choice(some_field_values)
return False
return [
("field_1", populate.randomize([value1, value2, value3])),
("field_2", populate.randomize([value_a, value_b], [0.5, 0.5])),
("some_other_id", populate.randomize(some_other_ids)),
("some_field", populate.compute(get_some_field, seed="some_field")),
('active', populate.cartesian([True, False])),
]
def _populate(self, size):
records = super()._populate(size)
# If you want to update the generated records
# E.g setting the parent-child relationships
records.do_something()
return records
Population tools
^^^^^^^^^^^^^^^^
Multiple population tools are available to easily create
the needed data generators.
.. automodule:: odoo.tools.populate
:members: cartesian, compute, constant, iterate, randint, randomize
.. _qunit: https://qunitjs.com/
.. _qunit_config.js: https://github.com/odoo/odoo/blob/51ee0c3cb59810449a60dae0b086b49b1ed6f946/addons/web/static/tests/helpers/qunit_config.js#L49
.. _web.tests_assets: https://github.com/odoo/odoo/blob/51ee0c3cb59810449a60dae0b086b49b1ed6f946/addons/web/views/webclient_templates.xml#L594
odoo/models.py
+34 -11
View File
@@ -6150,22 +6150,41 @@ Record ids: %(records)s
return 'placeholder.png'
def _populate_factories(self):
""" Return a list of pairs ``(field_name, factory)``, ``factory`` is a function that creates
a generator of values (dict of field values). Its API suggested by::
""" Generates a factory for the different fields of the model.
# generator yield values by iterating on base_generator and adding a value
# for the given field in the dicts yielded by base_generator
generator = factory(base_generator, field_name, model_name)
# suggested usage
for values in generator:
model.create(values)
``factory`` is a generator of values (dict of field values).
Factory skeleton::
def generator(iterator, field_name, model_name):
for counter, values in enumerate(iterator):
# values.update(dict())
yield values
See :mod:`odoo.tools.populate` for population tools and applications.
:returns: list of pairs(field_name, factory) where `factory` is a generator function.
:rtype: list(tuple(str, generator))
.. note::
It is the responsibility of the generator to handle the field_name correctly.
The generator could generate values for multiple fields together. In this case,
the field_name should be more a "field_group", covering the different fields
updated by the generator (e.g. "_address" for a generator updating multiple address fields).
"""
return []
@property
def _populate_sizes(self):
""" Return a dict mapping symbolic sizes (``'small'``, ``'medium'``, ``'large'``) to integers,
giving the minimal number of records that method ``_populate`` should create.
giving the minimal number of records that :meth:`_populate` should create.
The default population sizes are:
* ``small`` : 10
* ``medium`` : 100
* ``large`` : 1000
"""
return {
'small': 10, # minimal representative set
@@ -6175,12 +6194,16 @@ Record ids: %(records)s
@property
def _populate_dependencies(self):
""" Return the list of models which have to be populated before the current one.
:rtype: list
"""
return []
def _populate(self, size):
""" Create records to populate this model.
:param size: symbolic size for the number of records: ``'small'``, ``'medium'`` or ``'large'``
:param str size: symbolic size for the number of records: ``'small'``, ``'medium'`` or ``'large'``
"""
batch_size = 1000
min_size = self._populate_sizes[size]
odoo/tools/populate.py
+40 -3
View File
@@ -17,7 +17,7 @@ def format_str(val, counter, values):
return val
def chain_factories(field_factories, model_name):
def chain_factories(field_factories, model_name):
""" Instanciate a generator by calling all the field factories. """
generator = root_factory()
for (fname, field_factory) in field_factories:
@@ -35,6 +35,14 @@ def root_factory():
def randomize(vals, weights=None, seed=False, formatter=format_str, counter_offset=0):
""" Return a factory for an iterator of values dicts with pseudo-randomly
chosen values (among ``vals``) for a field.
:param list vals: list in which a value will be chosen, depending on `weights`
:param list weights: list of probabilistic weights
:param seed: optional initialization of the random number generator
:param function formatter: (val, counter, values) --> formatted_value
:param int counter_offset:
:returns: function of the form (iterator, field_name, model_name) -> values
:rtype: function (iterator, str, str) -> dict
"""
def generate(iterator, field_name, model_name):
r = Random('%s+field+%s' % (model_name, seed or field_name))
@@ -48,6 +56,14 @@ def randomize(vals, weights=None, seed=False, formatter=format_str, counter_offs
def cartesian(vals, weights=None, seed=False, formatter=format_str, then=None):
""" Return a factory for an iterator of values dicts that combines all ``vals`` for
the field with the other field values in input.
:param list vals: list in which a value will be chosen, depending on `weights`
:param list weights: list of probabilistic weights
:param seed: optional initialization of the random number generator
:param function formatter: (val, counter, values) --> formatted_value
:param function then: if defined, factory used when vals has been consumed.
:returns: function of the form (iterator, field_name, model_name) -> values
:rtype: function (iterator, str, str) -> dict
"""
def generate(iterator, field_name, model_name):
counter = 0
@@ -64,8 +80,16 @@ def cartesian(vals, weights=None, seed=False, formatter=format_str, then=None):
def iterate(vals, weights=None, seed=False, formatter=format_str, then=None):
""" Return a factory for an iterator of values dicts that picks a value among ``vals``
for each input. Once all ``vals`` have been used once, resume as a ``randomize``
generator.
for each input. Once all ``vals`` have been used once, resume as ``then`` or as a
``randomize`` generator.
:param list vals: list in which a value will be chosen, depending on `weights`
:param list weights: list of probabilistic weights
:param seed: optional initialization of the random number generator
:param function formatter: (val, counter, values) --> formatted_value
:param function then: if defined, factory used when vals has been consumed.
:returns: function of the form (iterator, field_name, model_name) -> values
:rtype: function (iterator, str, str) -> dict
"""
def generate(iterator, field_name, model_name):
counter = 0
@@ -83,6 +107,9 @@ def iterate(vals, weights=None, seed=False, formatter=format_str, then=None):
def constant(val, formatter=format_str):
""" Return a factory for an iterator of values dicts that sets the field
to the given value in each input dict.
:returns: function of the form (iterator, field_name, model_name) -> values
:rtype: function (iterator, str, str) -> dict
"""
def generate(iterator, field_name, _):
for counter, values in enumerate(iterator):
@@ -95,6 +122,11 @@ def compute(function, seed=None):
""" Return a factory for an iterator of values dicts that computes the field value
as ``function(values, counter, random)``, where ``values`` is the other field values,
``counter`` is an integer, and ``random`` is a pseudo-random number generator.
:param function function: (values, counter, random) --> field_values
:param seed: optional initialization of the random number generator
:returns: function of the form (iterator, field_name, model_name) -> values
:rtype: function (iterator, str, str) -> dict
"""
def generate(iterator, field_name, model_name):
r = Random('%s+field+%s' % (model_name, seed or field_name))
@@ -107,6 +139,11 @@ def compute(function, seed=None):
def randint(a, b, seed=None):
""" Return a factory for an iterator of values dicts that sets the field
to the random integer between a and b included in each input dict.
:param int a: minimal random value
:param int b: maximal random value
:returns: function of the form (iterator, field_name, model_name) -> values
:rtype: function (iterator, str, str) -> dict
"""
def get_rand_int(random=None, **kwargs):
random.randint(a, b)