Most Powerful Open Source ERP

Catalog Migration

New ERP5 Catalog and migration process.
  • Last Update:2018-12-10
  • Version:001
  • Language:en


I recommend starting with reading the merge request description (here) to get basic explanation on erp5 catalog

  1.  Updation of erp5_core bt:
    • New portal_types:
      • Catalog : It is defined on ERP5Catalog class from ERP5Catalog/Document/
      • Catalog Tool :  It is defined on ERP5CatalogTool class which is erp5 catalog tool(earlier used to be ERP5.ERP5Catalog.CatalogTool )
      • SQL Method : It is defined on SQLMethod class which we were using from zope products document till now. The corresponding class has been added in ERP5.Documents.SQLMethod.
    • New Property Sheet
      • Catalog
      • CatalogTool
      • SQLMethod
      • CatalogFilter : Earlier filter_dict for any method(ZSQLMethod or Script(Python)) used to be a PersistentMapping object. This has been removed and the keys in the filter_dict has been added as properties for the method objects. For compatibilty of running instance, there are some changes been done in CatalogMethodTemplateItem installation for bt5. 
  2. Files changed for new erp5 catalog installation
    • :
      • Now instead of creating the old catalog and migrating to new one afterwards, its better to create a new erp5 catalog_tool for new erp5 site. This change has been mentioned in this commit [Add commit]
    • :
      • Installation of CatalogMethodTemplateItem objects should be according to the new catalog, so we have added constraints wherever necessary. 
  3. Describe why we create new files for ERP5CatalogTool and ERP5Catalog rather than just changing the old objects.

Important Points:

  1. We create newfiles for ERP5CatalogTool and ERP5Catalog instead of just changing the old files because we don't want user to have its instance crashed after it updates Products code without updating bt5.
  2. On migration, we do also have this script in which deletes old SQL Catalog and adds ERP5Catalog object (not good in case of dynamic migration). This script also uses functions to create PythonScript and SQLMethod as erp5 objects. Moreover, these functions to convert SQLMethod and PythonScript to erp5 objects can be called separately also. For example: In bt5 installation, we use these objects to get used to convert the catalog methods generated from xml to erp5 objects.

Dynamic Migration :

As there might be users who would be running their instance, so we handle migration for them. For new instance, we give users with the new erp5 portal_catalog itself. The code is consistent with the users who don't want to migrate to the new catalog and just update the code base. For migration, we do have 2 options:

  1. Dynamic migration: Which is making use of __of__ for erp5site and _bootstrap. 
  2. Migration to new objects: Which calls the migration function in ERP5Site and creates new portal_catalog object and copies all properties, attributes and objects from old portal_catalog.

Migration Steps:

  • Here we are concerned about only the running instance with old catalog. Regarding how the installation for a new erp5 site works, please see the last paragraph.
  • For migration we make use of '__of__' magic function, which is called on first access of erp5. So, someone with a running instance updates/rebase on the new catalog code and restarts his/her instance. This leads to calling of __of__ function for erp5_site which checks if we do have a portal_catalog installed of 'ERPCatlaogTool' class or not. If not, it calls 'synchronizeModule' function from 'portal_type_class'.
  • This function is where we deal with most of the migration. We check for existing catalaog tool class via _isBootstrapRequired and then we run the _bootstrap for it.
  • During _bootstrap, we loop inside objects of catalog_tool and catalog and dynamically change the class of the objects using changeObjectClass function from extensions.
  • Take note, this function has been updated to deal with migrating 'Folder' and 'XMLObject' also.
  • We remove the filter_dict from catalog which was a useless persistent object and update it as property of migrated SQL Method and Python Script objects.
  • This was the last step after which we have our catalog tool migrated. Also, during the _bootstrap we install some portal_types and property_sheet we need to have the compatible running erp5 site even after we don't update bt5.

For new ERP5Site, it comes with ERP5 Catalog and CatalogTool preinstalled. So, no migration needed in that case. 

Changes still to be made:

  1. SOLVED: [workaround]Component unable to update _registry_dict with _p_oid for lazy objects even after the formation of objects.
  2. SOLVED: Prefer immutable object.
  3. SOLVED: Override accessors for multiple selection objects in erp5_catalog objects
  4. SOLVED: Remove the change in PropertyManager regarding 'base_id' and implement it in SQLMethod and PythonScript class.
  5. SOLVED: Add run script and run method(SQL) in object_action
  6. SOLVED: Dynamic migration
  7. SOLVED: [Low Priority]getAcquiredCategoryList might be creating problem for IndexableObjectWrapper which creates Broken objects for some portal_id generator. Look into it.
  8. SOLVED: [Not required] Same with 'getQuantityUnitConversionDefinitionRowList'. SQLCatalog failed to call method
  9. SOLVED: Base_viewDict gives unauthorized error 

Problems persisting[All solved]:

NOTE: If you find some problem, please add it here with suitable tags. Thanks. 

1. [SOLVED] While installing erp5site, the bt5 'erp5_mysql_innodb_catalog' is responsible for creating new erp5_catalog inside portal_catalog. Earlier the catalog created was not indexable as it was not an erp5 object, but now the catalog is also indexable in itself. So, this creates problem while we install other business template items, as every item now created will try to generate an uid as the default catalog is indexable, but at the same time we don't have our catalog table ready to save the index as it is created at the end of installation. 

Simple example of this is when we try to install business template 'erp5_jquery', we delete the empty trashbin in the installation process itself(take note that this trashbin was created during the installation process). Now, this object has an uid, so calling manage_deleteObject on it will try to unindex the object, which is not possible as we don't have any catalog  table till now to unindex, hence we get the error "'<db_name>.catalog' doesn't exist".

For better understanding of problem, let's look at this error traceback:

1.While running erp5site setup process, we are at the point where we try to install erp5_jquery bt5(note that it is after we have installed 'erp5_mysql_innodb_catalog' bt5, that means we have a erp5_catalog, which is indexable) 

File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5/", line 2570, in setupERP5Core bt.install(**kw) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5Type/", line 232, in __call__ result = self.__dict__['_m'](instance, *args, **kw) File

2. Now here in bt5 installation process, we try to delete a trashbin object which was also created in the same process(bt5 insatllation), but now as we have a catalog which isIndexable, we get a uid for every object we create(thanks to the function newUid) . This means this trashbin object comes with an uid.

"/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5/Document/", line 5378, in _install trash_tool.manage_delObjects([trashbin.getId(),]) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5Type/", line 219, in manage_delObjects return ObjectManager.manage_delObjects(self, ids, REQUEST) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/eggs/Zope2-2.13.24-py2.7.egg/OFS/", line 540, in manage_delObjects self._delObject(id) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5Type/Core/", line 1619, in _delObject object.manage_beforeDelete(object, self) File

3. Calling delete function will try to call unindex function, which looks if the object has an UID or not, and if yes, unindex it. But the problem starts here, as we are still in a transaction where we are installing erp5site, our catalog table isn't ready yet. So, when we try to unindex object, it searches for the table to unindex and hence gives us the error. 

"/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5Type/", line 342, in manage_beforeDelete self.unindexObject() File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5Type/", line 379, in unindexObject catalog.beforeUnindexObject(None,path=path,uid=uid) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5Catalog/", line 971, in beforeUnindexObject self.beforeUncatalogObject(path=path,uid=uid, sql_catalog_id=sql_catalog_id) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ZSQLCatalog/", line 898, in beforeUncatalogObject catalog.beforeUncatalogObject(uid=uid,path=path) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ZSQLCatalog/", line 1735, in beforeUncatalogObject method(uid = uid) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ERP5Type/patches/", line 226, in DA__call__ result=DB__.query(query, self.max_rows_) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ZMySQLDA/", line 381, in query c = self._query(qs) File "/srv/slapgrid/slappart16/srv/runner/software/43d62734835a4a6fe6c005cf13c62653/parts/erp5/product/ZMySQLDA/", line 364, in _query raise exception ProgrammingError: (1146, "Table 'erp5_test_0.catalog' doesn't exist")

This leads to error while we try to install new items from a BusinessTemplate before creating all tables(atleast catalog tables). There can be 2 alternate solution for this problem :

  1.  We can run the scripts for creating the table just after we create a new erp5_catalog, as we know that any next object being created using .newContent on an existing portal_type will now get a uid attached to it, which might need unindexing or reindexing if we change anything(copy, delete, paste). This is uncertain as we(I, Ayush specifically) don't exactly know the step when we decide to create catalog table and move on the call the method to create it.
  2. Other option which was done earlier was to not create a new 'erp5_catalog' inside the portal_catalog, i.e, let the old catalog be insatlled during erp5site setup. Now, after we finish installing the bt5 and its sub objects, we migrate the sql_catalog to the erp5_catalog.