This blog post is part of a series introducing 3rd-party open-source technologies used within the free software solutions created by Nexedi. One of these solutions is NEO, our distributed, redundant and transactional ZODB storage designed as alternative to ZEO and FileStorage. With flexibility and reliability in mind, Nexedi chose MariaDB as a core component of the NEO system architecture.

Photo: fotolia.fr - pixel

"You are the one, NEO"

The volume of data handled by Nexedi is growing across the board with IoT, Big Data and Machine Learning becoming more relevant and projects such as Wendelin gaining industry traction. As Nexedi services often range from network infrastructure to business applications great emphasis is put on efficient data handling making accessability, availability, distribution and performance high-priority and research topics.

NEO (short for Nexedi Enterprise Object database - not to be confused with the neo graph database) was the result of such a research project intended to replace all implementations of ZEO (Zope Enterprise Object database) being used at Nexedi and to switch to an architecture that could be scaled automatically while also providing persistence and resiliency. During development, SQL and MariaDB proved to be a good choice because they met our immediate needs while also being flexible enough to reshape tables as the design of NEO grew. This was key, because frequent schema changes in early project phases were expected. MariaDB allowed us to focus on specific features of NEO while not having to call the low-level InnoDB BTree implementation directly which reduced development time considerably.

The initial challenge for the NEO project was to find the minimal set of things which could not be distributed while retaining and being able to closely monitor all the strong transactional consistency guarantees a ZODB implementation had to provide. In doing so we found that the we could reduce the critical section of the generic ZODB commit protocol to a shorter version than what ZODB provided at that time. We determined elements that had to be constant and areas where persistence was needed with MariaDB and SQL letting us redesign on-the-fly and eventually settle on a data structure to implement a working prototype. The resulting NEO cluster features different "personalities": configuration-only master, client and admin nodes acting besides pure storage nodes with configuration and persistence.

Leveraging the Power of MariaDB

NEO uses MariaDB to store low level object, transaction and location data and makes these accessible via the "medium" ZOPE layer which our main software solutions such as ERP5 are running on. Using MariaDB we could overcome the shortcomings of ZODBs traditional way of maintaining indices and are now able to easily scale our application architectures automatically and depending on need. MariaDB also provides well-test durability, which is especially important for the more advanced features of NEO, such as storing data in non-chronological order when lagging nodes are catch up or a node's workload is reduced by moving data to another node. Finally, since most of our implementations are running long-term, MariaDB also provides us the needed stability in terms of source code availability - it's a proven open-source embeddable solution.

As part of NEO, MariaDB was included in all latest ERP5 implementations as well as Nexedi's internal system and test infrastructure (30 users, 70 test nodes) currently handling indices of about 97GB (TokuDB) or 133GB (InnoDB backup) which without NEO compression would equate to a FileStorage equivalent between 250GB and 350GB.

Scaling MariaDB into the Future

Initially no customizations were made in MariaDB but we have since modified the underlying storage engines using TokuDB for its performance and space-efficiency characteristics instead of InnoDB.

For the past couple of years we have increased our efforts to contribute back to MariaDB, because we believe there should be open-source industry-grade software solutions available and in terms of database technologies MariaDB is an essential solution which we want to support into the future.

For us this translates to working with MariaDB in joint research projects, sponsoring some of the MariaDB developer salaries and helping to improve MariaDB code itself by reporting and collaborating on issues related to data availability, consistency and performance.

About Nexedi

Nexedi is one of the largest open-source Software Publishers in Europe providing services around their solutions for business (ERP5), cloud management (SlapOS) and Big Data analytics (Wendelin). As creator and maintainer of ERP5, Nexedi has deployed its no-license-cost and fully customizable ERP and related software solutions in a variety of industries across the globe ranging from aerospace, apparel and banking to healthcare, finance and government.