Code re-use in cookbooks

Education Project Pythia
1

How should code re-use be handled in cookbooks? In particular, let’s say I have a function that is used in multiple chapters AND the code is something that I want to illustrate to the reader. Ideally I’d like to define and show the functions only once (e.g. in the introduction), but then call it from other notebooks in the collection. Current practices seems to be to re-define the function in every chapter where it is used, which is a maintenance headache and results in a lot of clutter. Putting the function in a local module might be one option, but then how does the reader view it? Also, it puts up another barrier for cookbook creators in that they have to know how to create Python modules.

Any thoughts?

2

We use the literalinclude directive (related discussion and example in the sphinx-book-theme docs) to do something like this on some GeoCAT projects (i.e. show a separate script in a notebook).

If it’s something getting a lot of reuse though, a more robust home than a local module might make sense (e.g. a package somewhere). Not sure if that’s practical or makes sense for the cases you’re talking about.

1 Like
3

If you want to use the local module approach, you can display selected source code for specific objects with the builtin Python inspect module, like so:

import inspect

def myfunc(x):
    """Docstring"""
    # Comment
    return x

print(inspect.getsource(myfunc))

This will literally print the source for the myfunc function as a string. But it won’t be pretty, because what you probably really want is to syntax-highlight that string so it looks nice in a Jupyter Notebook.

For that, you can use the Python pygments package. Combined with the builtin Jupyter core methods for displaying HTML, you can write a fairly simple display_source function like so:

from IPython.display import display, HTML
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

def display_source(obj, color=True):
    obj_src = inspect.getsource(obj)
    if color:
        display(HTML(highlight(obj_src, PythonLexer(), HtmlFormatter())))
    else:
        print(obj_src)

This will use pygments to syntax-highlight the source string returned by inspect.getsource(obj) and return it as HTML, which Jupyter’s display(HTML(...)) function combination can display for you as pretty syntax-highlighted Python source.

You can package the display_source function in another local module in your cookbooks. Then when you want to show people the source for a given function, you simply have to:

from display_source import display_source
from mymodule import myfunc

display_source(myfunc)

It’s a bit more involved, but perhaps not so involved that it creates a barrier to usage.

Just an idea.

4

Here’s an example of the above approach.

I created a module called mymod.py with the contents:

def myfunc(x):
    """
    This is my function
    """

    # Just return 1
    return 1

And another module called display_source.py with the contents:

from IPython.display import display, HTML
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

def display_source(obj, color=True):
    obj_src = inspect.getsource(obj)
    if color:
        display(HTML(highlight(obj_src, PythonLexer(), HtmlFormatter())))
    else:
        print(obj_src)

Then, in my notebook I did:

from mymod import myfunc
from display_source import display_source

display_source(myfunc)

The output is (screenshot):

image

5

@clyne ,

Have a look at the %run jupyter magic. It might do what you want. This is not pythia-specific, but rather a bog-standard notebook capability that fits in jupyter-book projects.

I’ve used this successfully on jupyter-book builds where one notebook in the intro explains some process and builds a particular utility function. Other notebooks %run this intro notebook, and that puts the function into the execution environment. If you want to get really fancy, tag the %run cell with one of the tags which direct jupyter-book’s behavior to show (or not) that cell. I’ve also used this technique to include utility functions defined in an appendix to the book (instead of its intro).

There are also good arguments against using %run in this way. You need to be careful with what you include in the re-used notebook. It will be run in its entirety, so watch out for side effects or long execution times. You can also get into circular or recursive execution if you start running things without a well-defined structure for what runs what. But for the simple use case you describe–where exactly one small notebook contains code to be used by many other notebooks–it might be a good fit for you.

–gt

6

I agree - IMO we should be trying as much as possible to identify commonly-repeated patterns and develop the package ecosystem so that such tasks become automated (or trivially quick).

7

Thanks for all of the suggestions. I think are some good options here for dealing with cookbook utility functions that are re-used within a cookbook, but probably aren’t generalizable for broader use.