We use Jupyter Notebook to develop and operate our on-premise private cloud infrastructure. Our operations team consists of 4-5 engineers, each with a different skill set. The members use Jupyter Notebook to keep and share operational activities with the colleagues to keep them sustainable. In this poster, we introduce a set of Notebook Extensions used for recording and sharing.
We, the cloud operation team at National Institute of Informatics which consists of 4-5 engineers, have been running our private cloud using Jupyter Notebook for about 4 years. Over 10,000 notebooks have been created and recorded as the result of our operations. We have developed several extensions for Jupyter Notebook to address the problems associated with sharing know-how.
The binder link to try the extensions is https://mybinder.org/v2/gh/NII-cloud-operation/Jupyter-LC_docker/jupytercon2020-demo
Record cell’s origin: To maintain an infrastructure that includes various operations such as troubleshooting, engineers will create a new notebook for an operation and reuse a per-notebook or per-cell from existing notebooks. To keep track of how they reuse cells, we have developed "nblineage" extension. The nblineage extension is an extension that puts tracking ID, which we call MEME, to every notebook and every cell in it.
https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
Enhanced Logging to preserve execution history: Changes to the infrastructure are persistent, in other words, operations performed by notebooks in the past are more likely to affect the present than in areas such as data analysis. However, Jupyter Notebook allows multiple cell executions and deletion of a cell once executed, which may cause the execution log of the operation that caused the problem to disappear. Therefore, we developed a Kernel that stores logs separately from the Notebook to store all logs. All logs are logged along with MEME by the nblineage and executed code, which helps us understand why the cell was executed.
https://github.com/NII-cloud-operation/Jupyter-LC_wrapper
Add annotations to cells that have a similar origin: If a problem is found in a reused cell, it must be communicated to an engineer using cells that have the same origin. sidestickies extension which we have developed enables us to attach sticky notes to each cell utilizing Scrapbox. It provides the ability to add a sticky note to a cell that has a MEME of the same origin. We would like to have separate channels for meta-, side-, and reflective-communications, which are well-known use cases for sticky notes.
https://github.com/NII-cloud-operation/sidestickies
Search notebooks based on their origins: nbsearch extension provides search capabilities for Jupyter Notebooks which you created. It supports search by MEME in addition to search by keywords and modified times like a search engine. Therefore, users can easily find cells of the same origin in sticky notes added by sidestickies.
https://github.com/NII-cloud-operation/nbsearch
We will demonstrate how we use these extensions and visualize the activities of our team using MEME and how we communicate with each other.