Most Powerful Open Source ERP

OfficeJS Iodide: Migration of Jupyter Notebooks to Iodide Notebooks

OfficeJS Iodide notebooks can be used offline or online on any modern HTML5 web browser such as Firefox or Chrome. With the support of data science libraries such as NumPy, Pandas, Matplotlib, Plotly, etc. and soon scikit-learn, its nice to be server independent and carry on with data science just using browser. Hence, we tried to migrate some already famous lectures on data science on github. This web page mentions the steps and problems faced while migrating and obviously, the final result.
  • Last Update:2018-11-20
  • Version:002
  • Language:en

Notebooks are a simple, fun and efficient way to create business reports or play with data science libraries such as scikit-learn or pandas. They are also a great tool for interactive visualization of data. We have been using Jupyter notebooks at Nexedi as part of Wendelin out-of-core Big Data project. Using Jupyter, we could increase the productivity of engineers in charge of producing reports about the structural health of wind turbines in Germany. Using Jupyter, we could also provide a tool to analyse and visualize sales trends of a trading company in China. We are currently integrating Jupyter-Lab as the default IDE of SlapOS Edge Computing software.

Migrating Jupyter notebooks to Iodide

With the ongoing development on Iodide, we were able to use some of the ported packages for pyodide to migrate some of the scientific jupyter lectures. There are some differences between how the cell execution works for jupyter and for iodide, thus the migration needed to make some changes in the notebooks. Various ways to migrate notebook are being covered as follows.

Paste code from cell to cell

Migrating jupyter notebook to iodide by copy pasting the codes from the cell. This is a long process but it ensures the running of cell after every addition, hence one can find out if the cell is going to run or not and the error directly. This also ensures that we can modify the code to make it work in iodide because there are some differences between how the code execution in jupyter and iodide.

Some of the differences are

  • Eval Support - print 'iodide' would display 'iodide' in the output for jupyter as it supports both exec as well as eval, but wouldn't print anything in iodide as it uses eval to display the output.
  • Magics - Jupyter magics (statements starting with %) don't work in iodide, so extra cell with code complementing the use of magics needs to be added explicitly whenever we want to use magics.
    For example: %matplotlib inline in jupyter would not be useful in iodide, hence, we would be using updating the plot and using every-time we want to display a plot for matplotlib.
  • Data from External Source: This is also not possible in pyodide currently, but we can still move the data source to the base directory for pyodide and use it from there.

Migrating using Iodide Website

On the Iodide official website, they provide a feature to export raw ipynb files and then convert it to jsmd format with some boilerplate cells to tell about the objects the jupyter notebook. The only problem in this is that we don't get the data and there would be changes regarding the libraries being used. Remember, we can only use this method to convert version 4 of Jupyter Notebook.

The link jsmd-and-importing-notebooks clearly mentions:

?ipynb=<url-to-ipynb-file> - Imports a Jupyter notebook as jsmd and adds some boilerplate Python language plugin cells. This effectively makes a large percentage of Jupyter notebooks directly runnable in Iodide using our Empscripten-compiled Python + NumPy + Matplotlib stack without any other servers other than our CDNs. Of course, it doesn't import the data used in the Jupyter notebook - there's no reference to the data in most notebooks. but this should get you 80% of the way there.

Migration steps for an example jupyter notebook using iodide website

  1. There is an interesting collection of Jupyter Notebooks on github. Let's convert a Basic Python Notebook to Iodide. As mentioned earlier, we will need raw format of the Jupyter Notebooks to convert them to Iodide Notebooks, thanks to github, we have the raw content for the the Notebook.
  2. Now, we use iodide website to convert the notebook. As mentioned earlier, adding the URL of the raw file to<url-to-ipynb-file> will give us the converted Iodide Notebook. In case the conversion gives empty file, recheck the version of Jupyter Notebook as it is strictly required for it to be of version 4. In our case, we are able to convert the notebook, hence we get converted Iodide Notebook 

    Now, let us go one by one across the cells and check if the required content is what we expected or need. The first cell is boilerplate cell added by iodide. Next is the plugin to tell iodide that we are going to use python in the notebook, hence activating pyodide. Next is an extra cell which loads matplotlib. This is not needed in our case, so we can remove this cell (Shift + Backspace). After that, its cells for code which are either Markdown code or python code.
  3. Let's run all the cells and see the result now. There are some errors in our case.

    Let's see what those are one after another. First, let's remove the one with the error due to use of python2 (iodide only supports python3). After this, notice that print('Hello Python World !') doesn't display anything in output. As mentioned earlier, this is because iodide run python code using eval, and print function here doesn't returns anything, that's why, there is no output here, but we can still see this output in the browser log.

Final results of migration

  1. Migrated basic python lectures where we use libraries such as numpy, matplotlib, pandas. These Iodide Notebooks are hosted in Nexedi Gitlab Repository. Soon enough, we will be able to use scipy and scikit-learn on pyodide also. We already have a merge request to add them to ERP5 so that we start using them on Office JS Notebook App.
  2. Uploaded these lectures on Nexedi ERP5. The lectures are also published on Nexedi ERP5, hence users can also synchronize and use them on Office JS AppStore.