NEO is a distributed, redundant and transactional storage designed to be an alternative to ZEO and FileStorage. It is developed and maintained by Nexedi and is being used in several applications maintained by Nexedi as well as Nexedi internal production instance including all websites.
NEO implements ZODB's Storage interface, and supports the following standard extensions:
Note: There is no plan to support "version" extension in NEO, because of its pending-deprecation state.
NEO adds the following features:
See Changelog for a complete and detailed list of releases.
With growing data volumes and lessons learned from managing a petabyte in hindsight, below you can find additional chracteristics of NEO:
Finally, it seems that the biggest difference between described systems and NEO/ZODB sits around the meaning of "transaction" and expected application behavior inside a transaction: NEO provides the same level of isolation as ZODB does, which is (supposed to be) PL-2+, as per Atul Adya's thesis denomination (see below), which looks stricter than transaction isolation (shortly) described here.
The reliability of a data storage - such as NEO - is critical. To ensure the quality of NEO design, its protocol is in the process of being formally proven through model-checking. To ensure code quality, NEO project relies on automated testing:
You can get the source code in the following Git repository: https://lab.nexedi.com/nexedi/neoppod.git (Github mirror) or browse it online.
It is also published on PyPI.
The following software is required:
Old (contains obsolete information):
Automated test results are published on stack.nexedi.com.
Q: How does NEO scale compared to ZODB?
A: For "normal" database use (1+TB) NEO is running very stable. Non-scalability topics being worked on include pruning of old data being too slow and reshaping of a cluster (NEO moving data when cluster changes).
We did a number of scalability tests going up to 150TB to find bottlenecks. Issues found and being investigated:
The size of one NEO server currently used and connected to 60 zopes is 83724 GB (stored in ERP5 data stream module on top of NEO, smaller disk consumption due to compression). We want to gradually increase this 80 TB stored in ZODB to 1 PB and ran one test storing an array of 4.9 million rows using wendelin.core persistent numpy array in ZODB without consuming too much memory. Doing something similar in ZODB and FileStorage would be difficult in terms of disk space and performance.
NEO is Free Software, licensed under the terms of the GNU GPL v3 (or later). For rationale, please see Nexedi licensing.
Projects directly related to NEO, but not actually touching its core. They might or might not be strictly dependant on NEO.
Idea: write a FUSE wrapper using NEO as a storage back-end, instead of a hard disk partition, or cd, etc.
Goal: stress-testing with filesystem benchmark suites.
Progress: Started in may 2010, stalled since then. Code to be cleaned up and published when I (Vincent Pelletier) find time. Unstable.
Idea: write a memcached server using NEO as a storage back-end, instead of ram (original memcached) or other back-ends (kumofs, etc).
Goal: benchmark with memcached-oriented tools.
Progress: Not started, assigned.
API: Storage preferred, otherwise ZODB
To the best of our knowledge, there is no other Storage interface implementation offering both scalability and fault tolerance the way NEO does:
Some interesting pages on topics related to NEO, but not written for/about NEO:
NEO project was initiated in 2005 by Nexedi, a French company developing ERP5 - a Free Software ERP for small to large enterprises implemented on top of Zope - since 2001. NEO was then endorsed in 2009 by System@tic competitive cluster, by Paris Region and by FEDER programme of the European Union.