IPython and Using Notebooks

Python is an open source platform for interactive and parallel computing. It started with the realization that the standard Python interpreter was too limited for sustained interactive use, especially in the areas of scientific and parallel computing.

For OS X users it’s usually recommended that using package managers such as MacPorts and Homebrew instead, installing Python therein, and avoiding using the system Python. A better solution for novices is to install an independent Python distribution, including Anaconda and Enthought.

Installing Anaconda

Anaconda is a free distribution of Python packages distributed by Continuum Analytics. Conda can be used for package management as well as environment management.

bash Anaconda3-4.3.0-MacOSX-x86_64.sh

Installing Homebrew

Homebrew is a missing package management tool.

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

To check for any issues with the install run

brew doctor

To search for an application:

brew search

To install an application:

brew install <application-name>

To list all apps installed by Homebrew

brew list

To remove an installed application

brew remove <application-name>

To update Homebrew

brew update

To see what else you can do

man brew

If /usr/local/Library/LinkedKegs seems to contain a list of, well, linked kegs, so this should do the trick:

ls -1 /usr/local/Library/LinkedKegs | while read line; do 

echo $line 

brew unlink $line 

brew link --force $line 

done

Installing Python

conda create -n py36 python=3.6 anaconda
source activate py36

Installing IPython

conda install ipython

IPython comes with a test suite called iptest.

iptest

Updating Python

All-in-one distributions -

When pip and easy-install are not enough, both Anaconda and Canopy have their own built-in package management systems.

Anaconda provides a powerful command-line tool called conda. conda can be used for package management as well as environment management. Every program runs in an environment that includes the version of Python, IPython, and all included packages.

conda update conda

conda update python

Python requires v3.

Install Python Packages

pip list
pip install -upgrade
pip install -r requirements.txt

To activate the environment

cd /Users/tkmaemd/anaconda/envs/py35/bin
source activate py35
ipython

To deactivate environment

cd /Users/tkmaemd/anaconda/envs/py35/bin
source deactivate py35

Shell integration
ipython
In[1]
Out[2]
?map
??map

Magic commands

OS equivalents: %cd, %env, and %pwd
Working with code: %run, %edit, %save, %load, %load_ext, and %%capture
Logging: %logstart, %logstop, %logon, %logoff, and %logstate
Debugging: %debug, %pdb, %run, and %tb
Documentation: %pdef, %pdoc, %pfile, %pprint, %psource, %pycat, and %%writefile
Profiling: %prun, %time, %run, and %timeit
Working with other languages: %%script, %%html, %%javascript, %%latex, %%perl, and %%ruby

Installing R in Jupyter

1 installing via supplied binary packages

install.packages(c('repr', 'IRdisplay', 'evaluate', 'crayon', 'pbdZMQ', 'devtools', 'uuid', 'digest')) devtools::install_github('IRkernel/IRkernel')

2 Making the kernel available to Jupyter

IRkernel::installspec()

3 install basic R packages by conda

conda install -c r r-essentials

Extra for magic commands

With magic commands, IPython becomes a more full-featured development environment. A development session might include the following steps:

1. Set up the OS-level environment with the %cd, %env, and ! commands.
2. Set up the Python environment with %load and %load_ext.
3. Create a program using %edit.
4. Run the program using %run.
5. Log the input/output with %logstart, %logstop, %logon, and %logoff.
6. Debug with %pdb.
7. Create documentation with %pdoc and %pdef.

This is not a tenable workflow for a large project, but for exploratory coding of smaller modules, magic commands provide a lightweight support structure.

Some observations are in order:

• Note that the function is, for the most part, standard Python. Also note the use of the !systeminfoshell command. You can freely mix both standard Python and IPython in IPython.
• The name of the function will be the name of the line magic.
• The line parameter contains the rest of the line (in case any parameters are passed).
• A parameter is required, although it need not be used.
• The Out associated with calling this line magic is the return value of the magic.
• Any print statements executed as part of the magic are displayed on the terminal but are not part of Out (or _).

Debug example

x=0

1/x

%debug
h
(help)
w
(where am i)
p x
(print)
q
(drop debugger)
%pdb

A full complement of commands is available for navigation:

u/d for moving up/down in the call stack.
s to step into the next statement. This will step into any functions.
n to continue execution until the next line in the current function is reached or it returns. This will execute any functions along the way, without stopping to debug them.
r continues execution until the current function returns.
c continues execution until the next breakpoint (or exception).
j <line> jumps to line number <line> and executes it. Any lines between the current line and <line> are skipped over. The j works both forward and reverse.

And handling breakpoints:

b for setting a breakpoint. The b <line> will set a breakpoint at line number <line>. Each breakpoint is assigned a unique reference number that other breakpoint commands use.
tbreak. This is like break, but the breakpoint is temporary and is cleared after the first time it is encountered.
cl <bpNumber> clears a breakpoint, by reference number.
ignore <bpNumber> <count> is for ignoring a particular breakpoint for a certain number (<count>) of times.
disable <bpNumber> for disabling a breakpoint. Unlike clearing, the breakpoint remains and can be re-enabled.
enable <bpNumber> re-enables a breakpoint.

Examining values:

a to view the arguments to the current function
whatis <arg> prints the type of <arg>
p <expression> prints the value of <expression>
Matering IPython 4.0

Chapter 1 Using Python for HPC

High Performance Computing

API allowed people to store data on those machines (the Amazon Simple Storage Service, or S3) and an API allowed people to run programs on the same machines (the Amazon Elastic Compute Cloud, or EC2). Together, these made up the start of the Amazon Cloud.

Fortran provided answers to problems of readability, portability, and efficiency within the computing environments that existed in early machines. How Python/IPython, while not originally designed for runtime efficiency, takes these new considerations into account.

Chapter 2 Advanced Shell Topics

IPython beyond Python

There are too many magic commands to go over in detail, but there are some related families to be aware of:

OS equivalents: !ls, %cd, %env, and %pwd
Working with code: %run, %edit, %save, %load, %load_ext, and %%capture
Logging: %logstart, %logstop, %logon, %logoff, and %logstate
Debugging: %debug, %pdb, %run, and %tb
Documentation: %pdef, %pdoc, %pfile, %pprint, %psource, %pycat, and %%writefile
Profiling: %prun, %time, %run, and %timeit
Working with other languages: %%script, %%html, %%javascript, %%latex, %%perl, and %%ruby

Terminal Python
stdin&stdout
Python execution
JSON
IPython Kernel 

Chapter 3  Stepping Up to IPython for Parallel Computing

Serial Processes

Program counters and address spaces
Batch systems
Multitasking (Cooperative multitasking / Preemptive multitasking) and preemption

Threading

Threading in Python

Limitations of threading

Global Interpreter Lock

Using multiple processors

The IPython parallel architecture

Getting started with ipyparallel

Parallel magic commands

Types of parallelism

Data Parallelism

Application steering