These guidelines provide extended rules and information for Nexedi staff on how to write and run unit tests.
Table of Contents
If you would like to run automated tests on your project, you will have to define what we call a "test suite" document on Nexedi ERP5. Then Nexedi ERP5 will automatically look for you for some free resources to run your tests. Results can be seen in the module "Test Result Module".
ERP5 Repository is structured like this :
tests/__init__.py defines a test suite. A test suite is used to describe the list of test that will be executed, and it can be used to handle project specific way of running tests (like using --save). You can see right now in tests/__init__.py of ERP5 repository, there is class "ERP5" to launch all ERP5 unit test, and "PERF" to launch only performance tests.
In order to launch a new test suite, please go on Nexedi ERP5 in the module "Test Suite Module". You can then copy/paste an existing test suite, do some changes on the copy, and then validate.
Here the setting of ERP5-MASTER test suite which is used to test all generic erp5 tests:
On the document level, you can define several parameters:
The automated test suite will checkout every repository in vcs_repository_list, and then as soon as there is a commit in one of them, the list of revision of all theses repositories will be given to Nexedi ERP5. Then Nexedi ERP5 can check if there is other automated test suites running on other computers and will either create a new tests result or ask the current node to synchronize.
Once the testnode knows that it must run tests, it first run slapos commands in order to update/create software and to create new instance (so we have software and instance inside erp5testnode partition). Once the software is built and the instance is created, the command runTestSuite is launched.
In order to define where email notification will be sent, please setup the field "Stakeholder" on project. If you are using a mailing list, you can create an organisation representing the mailing list (which is a group of person).
We could see later if we can use RSS or ATOM feeds.
Running UI tests happens with help of firefox which is run inside a virtual X screen using Xvfb. As these test simulate user interaction with system it is required to make sure that always node-quantity=1.
If developer wants to see how firefox is working he can use following command on the machine running tests (or use ssh tunneling if appropriate):
x11vnc -display :123
If tests are on remote machine he can use ssh tunnel as follows:
ssh -t -L 5900:localhost:5900 user@machine_ip-474 'x11vnc -localhost -display :123'
Having tests for the work we do is very important. Generally speaking, writing tests allows to save plenty of time. It allows checking some parts of the code quickly. It allows to automate regression tests. Thanks to them, doing an important refactoring of the code is possible. An without them it's impossible to reach good quality of services.
A good test is a test covering well existing code, using small amount of lines of code an which is fast to run.
Whenever possible, it's important to organize the code into distinct parts (documents, classes , methods) that could be tested separately. Like this every part could be easily modified and tested without checking the whole code every time.
It is not necessary to test huge amount of data. Let's say you would like to test the method setTitle on a document, checking a very few values is enough (like simple ascii value and utf8 values with a string in Japanese), you do not need to test various complete dictionaries.
Please make sure to use live tests all the time. Old tests must be converted to be compatible with live tests.
What is a live tests ? A live test is a test that we can launch directly through the ERP5 interface. A live tests create data inside your instance, therefore it must be written in such a way to not destroy everything.
It is really much better when the code is well orgazined and when it is possible to test subpart of the code directly instead of doing only global tests. When an unit test takes only 1 second, it is very fast to improve the related code. When a test takes 50 minutes, it is a nightmare to check any change and this is very inefficient.
What a test MUST NOT DO:
It's not necessary to destroy all data to make good tests. So here some tips in how to deal with existing data:
initial_count = len(module)
initial_count = len(module)
It's really not an issue to cumulate data from the run of various tests. It's even very helpful in many cases. When you do new developments, you directly have data to works with. And you already have examples of data of previous run, it can help solving issues more quickly.
Live tests are really good candidates to check if a production server is running correctly. Some advices above can help, but there is some risk. Like it can be possible to show to users data entered by tests. Some messages could be wrongly sent to external software (like bank money transfer !). Some data entered by users could be altered.
So some rules and good usage practice must be written to do live test on production environment in a safe way.
Here we will publish more informations once we have more experience with it.
While working on a project, everyone doing features should write tests that should be launched automatically. We make the distinction between different types of tests:
Tests are controled by a Testing Master : an ERP5 instance used to control clouds of testing machines. In Nexedi, the Testing Master is Nexedi ERP5
Tests are defined on the Testing Master by developpers on a Test Suite in test suite module. A Test suite contains all informations needed to know which test should be launched (where to find code, how to retrieve the list of tests to execute, what is the software used).
Each test suite needs to be attached to a Distributor. A distributor is a document on the Testing Master allowing to define a cloud of testing machine. We have distinct usage of distributors :
The software used to control tests on a machine is called testnode. Depending on machines, we can create one or more test node per machine. Each testnode is associated to a distributor.
Unit test are run by launching one or several nodes, each launching one or more parallel tests. All nodes are synchronized through Testing Master which is in charge of distributing the work and to make sure every node is testing same revision.
Each node is installed thanks to Slapos tools through Slapos Master (Vifib). Code is handled by git, and nodes are in charge of checking if there is new commit, and if there is new commit they can ask Nexedi ERP5 if there is some work to do.
See (outdated) op-network-devel-functional-testing
Nexedi ERP5 provide an API to start, stop unit test. This API also allows to report failures, results (tests passed or failed), status and allows to know if there still ongoing work. The API is available here.
Any tool test running reporting results on Nexedi ERP5 must use this API. Soon a library will exist to allow using it easily. This API allow to handle from simple cases (like one unit test at a time on a single server) to complex cases (an unit test running on several servers, with different number of cores on each server). For now a working example using this library is available in runTestSuite and ERP5TypeTestSuite.
Test nodes are created on vifib.net using nexed_development_service account. Their configuration is quite trivial, please see existing examples if you wish to create a new test node. A typical configuration is:
<?xml version='1.0' encoding='utf-8'?>
<?xml version='1.0' encoding='utf-8'?>
For a 4 core server (8 with hyperthreading), it is good to create only 2 test nodes with a node-quantity of 3. Like this at most 6 threads are used. Also having only 2 test nodes is important, if we use 6 testnodes of 1 core, then we need to install much more software than if we have only 2 nodes. So to save disk space, it is better to not have too much test nodes.
Test Nodes¶ doesn't know in advance on which project they are going to work on. Therefore every test node is defined with the url of a distributor. The test node will call "startTestSuite" on the distributor and it will get all needed parameters to work on one or many projects.
The first time a test node calls startTestSuite, Nexedi ERP5 is going to look if this test node already exists. If not, then it will be created under test node module.
From time to time, an alarm (task_distributor_alarm_optimize) looks at all defined test suites and available test nodes and distribute the work to do. This alarm avoid moving test suite from a test node to another uselessly. In the same time, this alarm is checking for all test node that seems dead (10 hours without sending message) and invalidate them. Like this test suite allocated to a dead test node will be moved to another test node automatically.
Let's say we have 2 testing nodes daemon (A and B) running on two different computers. Each daemon is doing checkout or update of repositories. Since A and B can run with several minutes or even more of interval, Nexedi ERP5 needs to make sure that both A and B are testing same revision. Therefore testnode A (assuming no test is already started) will do:
[('repository 1',x), ('repository 2',y)]
And then testnode B will do (running a little bit after testnode A):
[('repository 1',X), ('repository 2',Y)]
Like this we are sure that all computers running the same test suite will be synchronized.
Sometimes developer may need to create a testnode, and log in. To see what happened in that testnode when running a test suite. In order to do that. Developer need to create a webrunner and install erp5testnode software release. Before compile the erp5testnode, you need go to software/erp5testnode/instance-default.cfg . Change the working-directory to a custom directory path, shorter is better, e.g: /srv/slapgrid/slappartX/t. Don't forget to change the md5sum in the software.cfg. See the example in here: [HARDCODED] Hardcode erp5testrunner directory
Then when the testnode download the repositories which defined in a test suite. It will store the code in this working-directory. Then developer should follow the Test Nodes section in above, add that three parameters in Webrunner -> Services -> Parameter.