JupyterCon 2023

Reminder - The date and time of this sessions are placeholder due to the limitation of the conference software.
All Poster will be presented during the Poster Session from 6:30 to 8:00 on Thus 11th,

https://cfp.jupytercon.com/2023/talk/AKPRE8/

--

Many Jupyter users want to run their notebooks in batch processing and production environments. Some popular and well-regarded open source projects and products now let users run and schedule notebooks as jobs. Until now, though, Jupyter itself has not offered its own notebook job capability. This talk will introduce you to Jupyter Scheduler, a new open source JupyterLab and Jupyter Server extension that lets all Jupyter users run and schedule notebook jobs via a simple yet powerful UI and API.

Jupyter Scheduler provides several benefits. First, notebook jobs can let users run more notebooks than their system could normally handle, without the overhead of setting up additional infrastructure. Second, notebook jobs improve the user experience of managing and parametrizing multiple long-running notebooks. Third, notebook jobs, unlike other types of production options, can include rich output (charts, data, widgets, etc.) and narrative text (markdown cells) in a periodic fashion, suitable for reporting. Fourth, in many cases, notebook jobs are a better way of moving to production than the traditional process of converting notebooks to Python scripts, containers, and CI/CD infrastructure.

Jupyter Scheduler, which is now part of the official Jupyter Server Subproject, provides an extensible notebook job architecture and user experience for notebook jobs. Not only does this capability enable users to run and schedule notebook jobs out of the box with Jupyter, it also enables third parties to plug different backends into the system to run notebooks as jobs on different platforms. In this talk, we will cover Jupyter Scheduler for end users who would like to run notebook jobs and for developers who would like to implement a new backend.

For end users, we will show how to install the extension, and use it to create, update, and manage both “run now” and scheduled jobs. We will demonstrate how to create and manage jobs with Jupyter notebooks, and walk through the different types of jobs supported by the scheduler. While this is not a full tutorial, you will be able to follow along and create your own jobs by installing the Scheduler extension prior to the talk.

For developers, we will also describe the architecture of this extension and how it allows you to customize both the user interface and the backend. Jupyter users are on a wide range of platforms: cloud providers, Kubernetes clusters, supercomputers, etc. Jupyter Scheduler addresses users’ needs by offering public extension APIs for running jobs on different platforms and for customizing the user interface for job creation. For example, Amazon Sagemaker Studio and Amazon SageMaker Studio Lab have created extensions for Jupyter Scheduler that let users run, schedule, and store notebook jobs on AWS through SageMaker.

Lastly, we will go over the future roadmap for this project, and talk about some of the interesting features expected in future releases. By the end of this session, you will have a good understanding of how to install, use and customize the Jupyter Scheduler extension in your notebook workflows. We are also very interested in learning about your unique use cases, so that we can adapt this extension to serve a wider Jupyter community.