This page summarizes what has been done recently with ZODB Components and from
September 2010 to February 2011, related to portal type classes, ZODB property
sheets, and accessors generation.
You cannot create instances of documents directly:
This use should be banned, and will break. Instead, you should use:
You can experiment to understand the difference: the first use creates a
Document instance, the second used creates a portal type class. We only want
portal type classes.
This only explains how to use ZODB Components from a developer point of view and
what has changed since the presentation at Europython 2012.
Documents, Extensions and Tests in bt5 are no longer stored in instance home but
directly into ZODB. This provides several benefits, such as:
The general technical implementation of ZODB Components for ERP5 is documented
in the slides written for Europython 2012. You can find the slides and notes
there and the video there. Even though there has been some changes since then,
the general idea still stands so this document is still worth reading.
Note that TYPE, used in this document, refers to ZODB Component type and may
be currently equal to document, extension or test.
Here is what you should know when using ZODB Components:
Unless a ZODB Component is in validated or modified validation_state,
it will never be used (on filesystem, it would mean that the file does not
exist at all).
A Component is in modified state if it has been previously validated, but
some errors were found after it has been saved. Until these errors have been
fixed, reference, version and source code set when the Component was
validated will be used.
Also, as ZODB Component are lazily loaded, no error may be displayed nor reported
until you actually access the object.
Only the reference and version matters to lookup for a Component to
be loaded. Even though ID could be set to anything, it should only follow
naming convention defined in the section below.
Generally speaking, any ZODB Component can be imported like any Python
module (but for most of them you should not do that except in tests, see the
per-Components sections below for further information), for example:
Import Component of the version with the highest priority:
Import specifically Component of version PROJECT:
For security reasons, a Developer Role has been introduced which is not
available through the UI. Only users with Developer Role can modify ZODB
Components. Similarly to Class Tool which requires creating a file in
ERP5Type Product, you must edit zope.conf on the filesystem to add users to
Developer Role. For instance, to add zope users to Developer
Upon export on the filesystem, a Component is split up into 2 files:
metadata (.xml) and source code (.py).
Likewise ZODB Property Sheets and Portal Type as Classes, when a ZODB
Component is modified, this reset Components and Portal Type as Classes as
well as inheritance may have changed or Property Sheets.
By default, textarea is used to edit source code through the web browser, but
Ace Editor (provided in erp5_ace_editor) provides a much better UI along with
maximize and fullscreen modes, jumping to line and column where an error or
warning has been found... After installing erp5_ace_editor bt5, you can enable
it as Source Code Editor in User Interface tab in Site/User
Preference. Later on, Ace Editor will also be useable from ZMI (Jérôme).
You can also edit ZODB Components through WebDAV or FTP (actually supported by
Zope directly with a few code to make it work for ZODB Components). You must modify
zope.conf and add the following section:
For example to mount your ERP5 instance and edit ZODB Components with davfs2 (on
Debian, the package is davfs2, read mount.davfs(8), umount.davfs(8) and
mount -t davfs -o uid=UID,gid=GID,username=USERNAME http://IP:PORT/erp5/portal_components /MOUNT/DIRECTORY
Even though ID could be anything, it should be in the following format:
Migrating bt5 Documents, Extensions and Tests from filesystem actually follows
this naming convention.
For projects, you should add at least one specific version, which can be
achieved by the following steps:
For ERP5 Components, there is already erp5 version defined in erp5_core, so
you don``t need to add anything.
Except for ERP5-specific version, you should create a new version, see previous
section for that.
You can migrate Business Template thanks to Migrate Components from Filesystem
action in Business Template view. In the next screen, you can specify versions
of Components to be migrated.
Note that the migration is all or nothing and ZODB Components will be
automatically validated so that they can be used straightaway.
Also, Products in bt5 are deprecated, instead you must either migrate your
Products to Documents or move them to normal Products, through your SlapOS
Software Release recipe.
When adding an External Method, you can specify Module Name exactly as you
used to do.
For example, a ZODB Extension Component whose version is project, reference is
Bar and ID is extension.project.Bar, you must only specify Bar. Unless you
have an Extension Component with the same reference and whose version has an
higher priority, then it will be used automatically. From an implementation
point of view, this will actually import erp5.component.extension.Bar,
equivalent to erp5.component.extension.HIGHEST_PRIORITY_VERSION_version.Bar.
By default, when specifying Bar as Module Name, ZODB Components will be
lookup and if there is no such Components, then it will fallback on the
Likewise filesystem bt5 Document, ZODB Document Components in bt5 must only
be used as Portal Types Type Class. But if you use these documents in tests for
example, you must use erp5.component.document instead of erp5.document.
Basically, a Test Component behaves like a Document Component. However, as a
Test Component is within a bt5, there is a chicken & egg issue with
runUnitTest command for installation of bt5 dependencies because in current
Unit Test, the list of required bt5s is defined in
getBusinessTemplateList() class method which requires to load the
Component. However, it cannot be loaded until the bt5 (and its dependencies)
have been installed as it may depend on Document or other Test Components.
One solution would have been to fiddle with sys.path and implement workaround
to load the Component without installing any bt5, but that would be hackish
and would not work when trying to import Document Components.
Therefore, the solution implemented is to specify through runUnitTest command
line the bt5 where the test can be found. This bt5 will be installed as well
as its dependencies using bt5list file (so you must make sure that this
file is up-to-date before doing running any test).
This should already be the case but all the bt5 dependencies must be
properly defined (dependency_list Business Template property or
Dependencies field on Business Template view).
For bt5s required specifically to run tests, there is a new property,
test_dependency_list (Test Dependencies on Business Template view)
where they can be added. Please note that in contrary to filesystem test,
the bt5 are not forced installed so you must define all dependencies,
including solving virtual dependencies (for example, for
erp5_full_text_catalog, you can add erp5_full_text_myisam_catalog
to Test Dependencies).
For customer project, make sure that your SlapOS recipe generates bt5list
for your customer bt5s. Also, to your customer tests/__init__.py, add
the following path to your tests path:
Finally, to execute a Test Component as a Live Tests, you can do through Run
Live Tests Component Tool Action. As of runUnitTest command considering that
testFoo is in bt5 called hogehoge:
This section lists major changes, excluding bootstrap issue, minor bug fixes and
UI improvements here and there.
Access to Component Tool has been further restricted (anyone was able to view
Components) and is now set through Component Tool class rather than instance, so
it can be easily changed anytime, rather than only being set at creation or
through an upgrade script.
Before, in order to check that the source code was somewhat valid, the code was
actually executing, but this approach has the following drawbacks:
The source code is now checked statically through Pylint if it can be
imported, otherwise it fallbacks on executing the source code as before (please
note that Pylint has been added to SlapOS recipe specifically for ZODB
Components so you may need to update your environment).
Pylint has been chosen in favor of (faster) other implementation such as
pyflakes because it can also check coding style and naming conventions, which
will be used in the future. Moreover, it seems to report errors that pyflakes
could not find.
As a side note, edition of ZODB Components through Ace Editor has been greatly
improved so you can click directly on the errors or warning and it will go the
corresponding line and column.
Upon any import in Python < 3.3, the Python global import lock is acquired (to
avoid race conditions while checking sys.path, avoid incomplete modules from
being seen by other threads or processes and also to avoid a module from being
executed twice). Therefore, ZODB Component import hooks (following PEP302) are
protected by import lock.
However, there was a deadlock when trying to import Components, as these import
hooks tries to fetch properties from ZODB, which may unpickle objects in another
thread (for an Exception class with ZEO for example) and thus trying to acquire
import lock when importing classes.
From now on, the import lock is released in import hooks until sys.path is
actually modified and there is another lock (aq_method_lock, common to Portal
Type as Classes and ZODB Property Sheets) to prevent entering import hooks in
A solution would be to introduce a per Component package lock (still coarse
grain) or a per Component lock (finest grain we could do) if the performances
end up being too bad but it seems to be working well enough as it is.
The best solution would be to use Python 3.3, as there is no more global lock
but a lock per module. However, the import machinery has completely changed
(implemented in Python and not C anymore), so it would be probably quite
difficult to backport...
A Component was automatically validated on Business Template installation, but
this meant, among other things, that exchanging bt5 containing Components with
errors was not possible.
The last Workflow History of Component Validation Workflow is now exported
without adding anything, thanks to a new Property introduced in Business
Template (before, you could only export the full Workflow History).
As requested by a customer, this could be fairly useful to be able to change
ZODB Components on one specific node before the changes are actually propagated
on other nodes (for example, when fixing a bug on production to allow testing on
only one node).
Being able to know where a given ZODB Component comes from and where it is
currently used (which Portal Type classes, which Property Sheets and so
on). This should be common to ZODB Property Sheets and Portal Type as Classes.
Once bugs found with bt5 ZODB Components and other bugs requiring restart of
ERP5 have been fixed, the next milestone is to migrate filesystem Products to
For now and likewise ZODB Property Sheets and Portal Type as Classes, everytime
a ZODB Component is modified, a reset of all ZODB Components is done, but
after implementing nicely introspection, it should be possible to only reset the
modified Components and its dependencies.
ClassTool is no longer necessary and could probably be removed after the merge.
Currently, an intermediate file on the filesystem is used to perform static
checking, but this should not be necessary. Therefore, pylint should be patched
so that it can take a string instead of a filename.
We're getting rid of Base._aq_dynamic, and accessors are now generated directly
from Property Sheet definitions, and put into Accessor Holder: one Accessor
Holder for each existing Property Sheet item.
Accessor Holder are classes, and you can see them in the method resolution order
of your ERP5 objects. For instance, for a person, person._``_class_``_.mro()
<class Products.ERP5.mixin.encrypted_password.EncryptedPasswordMixin at 0xcce42cc>,
<class Products.ERP5Type.CopySupport.CopyContainer at 0xb340b0c>,
Each accessor_holder class corresponds to accessors that come directly from a
Property Sheet. Note as well accessor_holder.BaseAccessorHolder which contains
common methods such as related category getters and portal type group getters.
_aq_reset is gone as well.
The effective tradeoff of this change is the following: we trade dynamic lazy
generation for static generation plus a few mro()-deep lookups. Check for
example the Base._edit code, where we have to lookup in a class mro() to fetch
the list of restricted methods. This kind of places where we have to walk one's
mro() are costly places. On the other hand, it's EASY to optimize them. With
lazy aq_dynamic, environment was constantly changing, and we had no guarantees
that everything was generated. But with portal type classes/accessor holders,
nothing ever changes once the class has been generated once: at the end of
loadClass() (ERP5.dynamic.lazy_class) nothing will ever happen to the class
anymore, data is "static". So it means that all deep computations we do can be
safely cached on the class object for later. Back to our _edit example, the list
of method ids that are restricted can be, and should probably be computed once
and stored on the portal type class
A new performance test can now be written. On a tiny instance, that only has
for property_sheet in portal.portal_property_sheets.contentValues():
And time this loop.
The impact of accessor generation is now easy to measure and improve, instead of
being a giant octopus with tentacles that unfold at every dynamic call.
Once you've looped over this list, you're mostly done, and the rest of the code
only gathers useful accessor holders and puts them on the right classes. Cherry
picking with workflow twists, as you still need to wrap accessors as
WorkflowMethod on the portal type class. We may start with a relatively higher
cost, but that's easier to improve, easier to profile, easier to optimize.
If then, you want to assess the cost of Workflow method generation, you can do
for portal_type_id in portal.portal_types.objectIds():
And once again, time it.
Generally speaking, we generate things less blindly, and after cleanups the
memory usage should drop to a lower figure than with aq_dynamic.
The globals in Utils are evil, and cache too much. I suppose that removing them
or emptying them WILL save a lot of memory. Similarly, I'm questioning the
validity and use of the workflow_method_registry attributes on portal
type/property holder/accessor holder classes
There was something relatively bad in the way we were using _aq_reset.
This edition triggers two workflow triggers, one for the class change, and one
for the base category change. Each trigger used to cause an _aq_reset call.
Generally speaking, if during one transaction we had N property changes on M
different objects, we would trigger N*M times _aq_reset. That begs the
question: is it absolutely compulsory to reset accessors immediately after one's
If we think about it, 100% of the actions that can trigger accessor regeneration
are user-triggered. Meaning that transactions will be short-lived, and that in
case of a success, a commit() will happen under a short time.
So can we delay the reset at commit time? Yes, it seemed so.
It has a few nice properties:
Why did I care so much about the number of resets? With new accessor generation,
we do a bit more during generation; and especially the generation of basic
properties are very costly. So chaining several resets is costly, much more
than two aq_resets.