Building Notebook-based AI Pipelines with Elyra and Kubeflow

https://www.youtube.com/watch?v=KR_m20pFXtU&ab_channel=Databricks 

1. Interactive Notebooks

 

 

2. Elyra Overview

    a. Elyra is a set of AI Centric extensions to JupyterLab Notebooks.

 

3. Elyra Key Features

    a. Visual Pipeline Editor 

        i. Visual editor for building AI pipelines, enabling the conversion of multiple notebooks into batch jobs or workflows.

    b. Notebooks as batch jobs

    c. Python script execution

    d. Automated Table of Contents

    e. Code Snippets

    f. Git integration

 

4. Power of Elyra

The power of the Elyra is that this one pipeline specification can be run both locally and remotely allowing you to test things out locally.

 

Once the iterative and experimentation phase is completed locally, the code and set of the notebooks can be modularized into a set of notebook modules which are the nodes in this graph as well as poetentially python scripts and these can all be packaged together as a batch job that runs through Kubeflow.

 

So each node is executed in its own isolated container environment and has access to the full cluster resources.

 

So related to this is the ability to execute single notebooks as batch jobs and effectively this is a single node pipeline. and this is also possible for python scripts.

 

So python scripts are also first-class citizens, they can be edited within a elyra in the editor and executed against either local cloud-based resources.