JupyterCon 2023

Generative artificial intelligence (AI) models are trained to generate new, previously unseen data (text, images, etc.). The generated data is both similar to the training data and a response to a user provided natural language prompt that describes a task or question. Recent generative AI models such as Amazon CodeWhisperer, Codex, Stable Diffusion, and ChatGPT have demonstrated solid results in performing a wide range of tasks involving natural language (content generation, summarization, question answering), images (generation, explanation, inpainting, outpainting, transformation), data (synthetic data generation, transformation, querying, etc.), and code (autocompletion, explanation, debugging, code reviews, etc.). In this talk, we describe open source work in Jupyter to enable end users to perform a wide range of development tasks using generative AI models in JupyterLab and the Jupyter Notebook.

Since their creation, Jupyter and IPython have always had an architecture and user interface for code generation through autocompletion. Indeed Jupyter users have been well trained to hit “tab” as they are coding to get contextual autocompletions for functions, variable, methods, properties, classes, etc. This architecture has already been used to integrate AI provided autocompletions into JupyterLab and the Jupyter Notebook (Kite, Tabnine). As generative AI models expand to perform other tasks, a generalization of the Jupyter autocompletion architecture is needed to enable users to perform potentially any task in Jupyter’s applications through a generative AI. In this talk we will describe an architecture and user experience for generative AI tasks in Jupyter.

The Jupyter generative AI architecture is founded on an extensible Jupyter Server API for registering generative AI models and the tasks they perform (represented as task prompts) with Jupyter. This enables third parties to quickly integrate their generative AI models into Jupyter and for end users to enable the models and tasks using the JupyterLab extension manager UI or a simple pip/conda install. Once models have been enabled, users working in JupyterLab or the Jupyter Notebook can 1) select anything (text, notebook cell, image, file, etc), 2) type a prompt to perform a task with the selection, and 3) insert the AI generated response in a chosen location (replace, insert below, new file, etc.). The task system is integrated with Jupyter’s MIME type output system, which enables GAI tasks to work with inputs and outputs of any MIME type. From the end users perspective, this enables them to enable and use models to perform tasks on text, code, images, data, audio, video, etc. This architecture also has APIs in Jupyter Server and JupyterLab that enable third parties to extend and integrate with the models and UI.

In the talk we will also give concrete examples of the AI driven tasks that can now be performed in JupyterLab. Examples include code refactoring, debugging, code transformation (e.g., Python to C/C++), code explanation (“generate and insert a markdown cell that describes this code”), synthetic data generation, data transformation, API documentation generation, technical content generation, and more. The focus of these examples will be less on the capabilities of the underlying AI models, and more on the seamless integration of the models into Jupyter to perform various tasks. Our assumption is that generation AI models will continue to improve and that Jupyter users will want a simple user experience that works with any such model.

Another potential application of this architecture is as a general purpose “playground” for exploring the capabilities of generative AI models. While not the primary goal of this work, the architecture effectively turns JupyterLab into a general-purpose generative AI playground. This enables users to quickly integrate models, and then use the familiar interface of Jupyter, with its support for editing files/notebooks, version control, real-time collaboration, etc., to play with generative AI. We discuss the pros and cons of this user experience and pose questions for future research and development.

This talk will be useful and of interest to 1) researchers who are building generative AI models to perform different tasks, 2) JupyterLab extension authors who would like to leverage generative AI, and 3) end users of Jupyter, who would like explore generative AI models and use them to perform tasks in their work.