# Intake Note Book: Access Notebooks from a intake catalogue and executing them using Paper Mill

**URL:** <https://discourse.pangeo.io/t/intake-note-book-access-notebooks-from-a-intake-catalogue-and-executing-them-using-paper-mill/571>\
**Category:** Science\
**Created:** [April 17, 2020, 7:36am UTC](https://discourse.pangeo.io/t/intake-note-book-access-notebooks-from-a-intake-catalogue-and-executing-them-using-paper-mill/571 "2020-04-17T07:36:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![NickMortimer](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/nickmortimer/32/438_2.png) [@NickMortimer](https://discourse.pangeo.io/u/NickMortimer)\
**Post date:** [April 17, 2020, 7:36am UTC](https://discourse.pangeo.io/t/intake-note-book-access-notebooks-from-a-intake-catalogue-and-executing-them-using-paper-mill/571/1 "2020-04-17T07:36:51Z")

</div>

In November last year I visited the Met Office with @kaedonkers. We spent some time looking at scientific workflows and parameterizing notebooks. To this end, we came up the concept of a set of parameterised notebooks accessible via an intake catalogue and then their output held in a catalogue.

> **[informatics-lab/intake-notebook](https://github.com/informatics-lab/intake-notebook)**
>
> intake drive that uses paper mill . Contribute to informatics-lab/intake-notebook development by creating an account on GitHub.

The way it works at the moment is I have two drivers,

Notebook : this provides access to the notebooks, so a ymal catalogue can be made like:

sources:  
enso:  
args:  
urlpath:

- “{{ CATALOG\_DIR }}/experiment/calculate\_enso.ipynb”
- “{{ CATALOG\_DIR }}/experiment/calculate\_enso\_clim.ipynb”  
description: ‘’  
driver: intake\_notebook.notebook\_source.NotebookSource  
metadata: {}

calculate\_enso\_clim.ipynb has a cell:

```auto
#parameters 
''' calcaulte the climatology for ENSO area given a start and enddate
    Parameters
    ----------
    catfile : file path to intake catalog that has a variable called sst
    startdate : date of first point to average (must be string e.g. "1974-1-1")
    enddate : dat of last point to average (must be string e.g. "1984-12-31")
'''
catfile='c:/data/providance/sst.yml'
startdate = "1974-1-1"
enddate = "1984-12-31"

```

this is read and monkey patched by the driver to form an execute function

```auto
data_cat =os.path.abspath("C:/data/providance/sst.yml")
books = intake.open_catalog('C:/data/providance/notebook.yml')
nb =books.enso.read()
nb.calculate_enso_clim.execute(startdate='1960-1-1',enddate='2020-1-1',catfile=data_cat)

```

Executing the notebook causes a new directory to be made in the current directory with a uuid number. the resultant notebook output and parameters are sored there.

Next step I can load all the output as a catalogue using my experiment driver

```auto
cat =intake.open_experiment('.')
cat.get_params()

```

that returns all the parameters used in a pandas data frame, it also looks for data outputs and concatenates them together

This was just a an experiment and I’m keen to hear people’s thoughts about it!