IPython: beyond plain Python

When executing code in IPython, all valid Python syntax works as-is, but IPython provides a number of features designed to make the interactive experience more fluid and efficient.

First things first: running code, getting help

In the notebook, to run a cell of code, hit Shift-Enter. This executes the cell and puts the cursor in the next cell below, or makes a new one if you are at the end. Alternately, you can use:

• Alt-Enter to force the creation of a new cell unconditionally (useful when inserting new content in the middle of an existing notebook).
• Control-Enter executes the cell and keeps the cursor in the same cell, useful for quick experimentation of snippets that you don’t need to keep permanently.

print("Hi")

Hi

Getting help:

?

Typing object_name? will print all sorts of details about any object, including docstrings, function definition lines (for call arguments) and constructor details for classes.

import collections
collections.namedtuple?

collections.Counter??

*int*?

An IPython quick reference card:

%quickref

Tab completion

Tab completion, especially for attributes, is a convenient way to explore the structure of any object you’re dealing with. Simply type object_name.<TAB> to view the object’s attributes. Besides Python objects and keywords, tab completion also works on file and directory names.

collections.

The interactive workflow: input, output, history

2+10

12

_+10

22

You can suppress the storage and rendering of output if you append ; to the last cell (this comes in handy when plotting with matplotlib, for example):

10+20;

_

22

The output is stored in _N and Out[N] variables:

_10 == Out[10]

True

Previous inputs are available, too:

In[11]

'_10 == Out[10]'

_i

'In[11]'

%history -n 1-5

   1: print("Hi")
   2: ?
   3:
import collections
collections.namedtuple?
   4: collections.Counter??
   5: *int*?

Accessing the underlying operating system

!pwd

/Users/fperez/teach/berkeley/2017-stat159/stat159/lectures/02-ipython

files = !ls
print("My current directory's files:")
print(files)

My current directory's files:
['Beyond Plain Python.ipynb', 'Index.ipynb', 'Notebook Basics.ipynb', 'Working With Markdown Cells.ipynb', '__pycache__', 'images', 'mod.py', 'test.txt']

!echo $files

[Beyond Plain Python.ipynb, Index.ipynb, Notebook Basics.ipynb, Working With Markdown Cells.ipynb, __pycache__, images, mod.py, test.txt]

!echo {files[0].upper()}

BEYOND PLAIN PYTHON.IPYNB

Note that all this is available even in multiline blocks:

import os
for i,f in enumerate(files):
    if f.endswith('ipynb'):
        !echo {"%02d" % i} - "{os.path.splitext(f)[0]}"
    else:
        print('--')

00 - Beyond Plain Python
01 - Index
02 - Notebook Basics
03 - Working With Markdown Cells
--
--
--
--

Beyond Python: magic functions

The IPyhton ‘magic’ functions are a set of commands, invoked by prepending one or two % signs to their name, that live in a namespace separate from your normal Python variables and provide a more command-like interface. They take flags with -- and arguments without quotes, parentheses or commas. The motivation behind this system is two-fold:

• To provide an orthogonal namespace for controlling IPython itself and exposing other system-oriented functionality.
• To expose a calling mode that requires minimal verbosity and typing while working interactively. Thus the inspiration taken from the classic Unix shell style for commands.

%magic

Line vs cell magics:

%timeit list(range(1000))

12.5 µs ± 102 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%%timeit -n 100000
x = list(range(100))
sum(x)

1.87 µs ± 53.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Line magics can be used even inside code blocks:

for i in range(1, 5):
    size = i*100
    print('size:', size, end=' ')
    %timeit -n 100000 list(range(size))

size: 100 1.02 µs ± 38.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
size: 200 1.4 µs ± 57.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
size: 300 2.35 µs ± 39 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
size: 400 3.82 µs ± 41.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Magics can do anything they want with their input, so it doesn’t have to be valid Python:

%%bash
echo "My shell is:" $SHELL
echo "My disk usage is:"
df -h

My shell is: /usr/local/bin/bash
My disk usage is:
Filesystem     Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1    931Gi  501Gi  429Gi    54% 3005697 4291961582    0%   /
devfs         192Ki  192Ki    0Bi   100%     664          0  100%   /dev
map -hosts      0Bi    0Bi    0Bi   100%       0          0  100%   /net
/dev/disk2s2  1.8Ti  517Gi  1.3Ti    28% 4360270 4290607009    0%   /Volumes/Time Machine - Seagate Silver

Another interesting cell magic: create any file you want locally from the notebook:

%%writefile test.txt
This is a test file!
It can contain anything I want...

And more...

Overwriting test.txt

!cat test.txt

This is a test file!
It can contain anything I want...

And more...

Let’s see what other magics are currently defined in the system:

%lsmagic

Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %cl  %clear  %colors  %config  %connect_info  %cp  %d  %dd  %debug  %dhist  %dirs  %dl  %doctest_mode  %dp  %dus  %dx  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

Running normal Python code: execution and errors

Not only can you input normal Python code, you can even paste straight from a Python or IPython shell session:

>>> # Fibonacci series:
... # the sum of two elements defines the next
... a, b = 0, 1
>>> while b < 10:
...     print(b)
...     a, b = b, a+b

1
1
2
3
5
8

In [1]: for i in range(10):
   ...:     print(i, end=' ')
   ...:

0 1 2 3 4 5 6 7 8 9

And when your code produces errors, you can control how they are displayed with the %xmode magic:

%%writefile mod.py

def f(x):
    return 1.0/(x-1)

def g(y):
    return f(y+1)

Overwriting mod.py

Now let’s call the function g with an argument that would produce an error:

import mod
mod.g(0)

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-38-81c06c6c0e90> in <module>()
      1 import mod
----> 2 mod.g(0)

~/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py in g(y)
      4
      5 def g(y):
----> 6     return f(y+1)

~/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py in f(x)
      1
      2 def f(x):
----> 3     return 1.0/(x-1)
      4
      5 def g(y):

ZeroDivisionError: float division by zero

%xmode plain
mod.g(0)

Exception reporting mode: Plain

Traceback (most recent call last):

  File "<ipython-input-39-46ce8a1dbba1>", line 2, in <module>
    mod.g(0)

  File "/Users/fperez/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py", line 6, in g
    return f(y+1)

  File "/Users/fperez/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py", line 3, in f
    return 1.0/(x-1)

ZeroDivisionError: float division by zero

%xmode verbose
mod.g(0)

Exception reporting mode: Verbose

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-40-3f57d27a0745> in <module>()
      1 get_ipython().run_line_magic('xmode', 'verbose')
----> 2 mod.g(0)
        global mod.g = <function g at 0x10d5958c8>

~/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py in g(y=0)
      4
      5 def g(y):
----> 6     return f(y+1)
        global f = <function f at 0x10d595730>
        y = 0

~/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py in f(x=1)
      1
      2 def f(x):
----> 3     return 1.0/(x-1)
        x = 1
      4
      5 def g(y):

ZeroDivisionError: float division by zero

The default %xmode is “context”, which shows additional context but not all local variables. Let’s restore that one for the rest of our session.

%xmode context

Exception reporting mode: Context

Running code in other languages with special %% magics

%%perl
@months = ("July", "August", "September");
print $months[0];

July

%%ruby
name = "world"
puts "Hello #{name.capitalize}!"

Hello World!

Raw Input in the notebook

Since 1.0 the IPython notebook web application support raw_input which for example allow us to invoke the %debug magic in the notebook:

mod.g(0)

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-44-9fa96bd6b3b6> in <module>()
----> 1 mod.g(0)

~/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py in g(y)
      4
      5 def g(y):
----> 6     return f(y+1)

~/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py in f(x)
      1
      2 def f(x):
----> 3     return 1.0/(x-1)
      4
      5 def g(y):

ZeroDivisionError: float division by zero

%debug

> /Users/fperez/teach/berkeley/2017-stat159/stat159/lectures/02-ipython/mod.py(3)f()
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
      4 
      5 def g(y):

ipdb> q

Don’t foget to exit your debugging session. Raw input can of course be use to ask for user input:

enjoy = input('Are you enjoying this tutorial? ')
print('enjoy is:', enjoy)

Are you enjoying this tutorial? yes!
enjoy is: yes!

Plotting in the notebook

This magic configures matplotlib to render its figures inline:

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 2*np.pi, 300)
y = np.sin(x**2)
plt.plot(x, y)
plt.title("A little chirp")
fig = plt.gcf()  # let's keep the figure object around for later...

The IPython kernel/client model

%connect_info

{
  "shell_port": 61750,
  "iopub_port": 61751,
  "stdin_port": 61752,
  "control_port": 61753,
  "hb_port": 61754,
  "ip": "127.0.0.1",
  "key": "7cf2d8a1-64149f698b1ca4929d44b73a",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

Paste the above JSON into a file, and connect with:
    $> jupyter <app> --existing <file>
or, if you are local, you can connect with just:
    $> jupyter <app> --existing kernel-90b8bcf0-cc04-4407-b44f-22dbcb1d2bca.json
or even just:
    $> jupyter <app> --existing
if this is the most recent Jupyter kernel you have started.

We can connect automatically a Qt Console to the currently running kernel with the %qtconsole magic, or by typing ipython console --existing <kernel-UUID> in any terminal:

%qtconsole