Tutorial on The Carpentries

Counting to Five (or, A Plan for Online Tutorials and What's Wrong With It)

Fri, 26 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

My daughter is five and a half years old, and some time in the last couple of weeks, she made a small cognitive breakthrough. It used to be that if I asked her, "What's five plus three?", she would carefully count five fingers, then count three, then go back and count them all together to get eight. She doesn't do that any more; instead, she holds up five fingers, then counts, "Six... seven... eight!" to get the answer. It may sound like a small thing, but it's not: the idea that five is always five, even if you didn't count it this time around, is such a big one that we forget it ever had to be learned.

Handling Variant Configuration Files

Tue, 26 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of our learners came to us with a problem last week. The program she uses depends on some complex configuration files, which she'd like to store in version control. However, a couple of parameters change depending on the machine the program is running on. She doesn't want to check those changes into version control over and over again; what should she do?

This Week's Tutorials

Fri, 15 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We ran five and a half online tutorials this week: one each for the workshop participants from UCL, Newcastle, UBC, and Edmonton, one for students combined from five previous workshops, and a make-up mini-tutorial for students from Plymouth who'd had A/V problems. The topics included:

creating simple HTML pages (both by hand and programmatically)
Python dictionaries (a.k.a. "hashes" or "maps" in other languages)
the Pandas statistical package

People who would like to review should look at the first half of the Version 3 notes on HTML (slides 1-17), the Version 4 videos on sets and dictionaries, and—well, we don't have anything on Pandas yet, but its creator has a book that you can preview.

Tutorial: NumPy, SciPy, and matplotlib

Thu, 07 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Today I did a toy data analysis of some annual temperature data in Australia and Canada over the last ~100 years. The goal of the exercise was to demonstrate loading data, inspecting it, and fitting trends. My last tutorial didn't involve any real data so this week we wanted to change that.

Like my previous tutorial I used the IPython HTML notebook to present. I hadn't been planning to use pylab mode or inline plots but some issues with my matplotlib forced me in that direction. There's an awkward situation here because I actually recommend people not use the pylab interface to matplotlib because the behind-the-scenes magic can cause problems (difficult to debug problems), but for doing demos the inline plots are really the way to go. The obvious upside is that the plots I made as part of the tutorial are embedded in the notebook for you to see now.

Git tutorial links

Sun, 03 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I promised to post some links related to the Git tutorial I taught on May 24, 2012. Most of my material was based on this excellent online book (free): http://git-scm.com/book/ch1-3.html. Here's another resource with help on specific and common Git commands: http://gitref.org/.

Someone asked the differences between Git and Mercurial, but because I haven't used Mercurial I wasn't able to answer. I found these links that I hope are useful: http://www.wikivs.com/wiki/Git_vs_Mercurial and http://www.rockstarprogrammer.org/post/2008/apr/06/differences-between-mercurial-and-git/. Also, here's a video of Linus Torvalds himself (creator of the Linux kernel and Git) promoting Git (and putting down everything that is not Git):

Introduction to NumPy Tutorial

Fri, 01 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Today I did a tutorial moving quickly through the basic usage of NumPy, the essential library for doing numeric computing with Python. We covered building arrays; indexing; array math; NumPy's element-wise functions; array attributes and methods; random numbers; masked arrays; and array comparison.

I presented using the IPython HTML Notebook. I enjoyed it because I never had to switch between a terminal and text editor, but I wonder what the people viewing thought. A nice feature of the notebook is that I can export it, both as a PDF and in the .ipynb format importable by IPython. The PDF is here, and the .ipynb file here.

Dictionaries are a Scientist's Friend

Thu, 31 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

[Code]

Today I would like to share with other scientists the power of dictionaries. I recently learned about this data structure during a Software Carpentry bootcamp that Greg Wilson and Ethan White organized. Greg and Jason Pell (from Michigan State) visited Utah State University for a two day bootcamp where I am a postdoc studying patterns of biodiversity. During that time we covered a whole slew of topics, but today I wanted to demonstrate how dictionaries helped to significantly speed up my Python code.

An Exercise With Matplotlib and Numpy

Mon, 21 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

For this tutorial, we'll be plotting some weather data from a site call Weather Underground. You can download temperature readings and weather events for your local area in a comma-separated file.

I've put weather data for Bloomington, IN in a file called weather.csv. Each row is one day, and there are columns for min/mean/max temperature, dew point, wind speed, etc. We'll be plotting temperature and weather event data (e.g., rain, snow).

Solution to Indented List Problem

Mon, 14 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Last week's homework was to convert a two-level bullet-point list like this:

* A
* B
 * 1
 * 2
* C
 * 3

into an HTML list like this:

<ul>
 <li>A</li>
 <li>B
 <ul>
 <li>1</li>
 <li>2</li>
 </ul>
 </li>
 <li>C
 <ul>
 <li>3</li>
 </ul>
 </li>
</ul>

so it would display like this:

A
B
- 1
- 2
C
- 3

My solution is shown in the video below; the code follows.

Solution to Sets and Dictionaries Exercise

Thu, 26 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Last week, I posted an exercise on working with sets and dictionaries that also included a fair bit of file I/O and string manipulation. My solution is below, in four parts, along with the code produced in each. If someone would like to re-do the file parsing using regular expressions, I'd be happy to post that as well.

import sys

#--------------------

def parse_pair(pair):
 '''
 Parse an atom-count pair. If the count is missing, assume
 that the count value is 1.
 '''
 if '*' not in pair:
 return pair, 1
 atom, count = pair.split('*')
 count = int(count)
 return atom, count

#--------------------

def parse_molecule(text):
 '''
 Get a single molecule description from a text string.
 '''
 name, formula_text = text.split(':')
 name = name.strip()
 pairs = formula_text.strip().split('.')
 formula = {}
 for p in pairs:
 atom, count = parse_pair(p)
 assert atom not in formula, \
 'Already seen atom %s in text %s' % (atom, text)
 formula[atom] = count
 return name, formula

#--------------------

def read_molecules(reader):
 '''
 Read molecules from a molecule file, returning a dictionary
 of {name : formula} pairs.
 '''
 result = {}
 for line in reader:
 line = line.strip()
 if (not line) or line.startswith('#'):
 continue
 name, formula = parse_molecule(line)
 assert name not in result, \
 'Already seen %s!' % name
 result[name] = formula
 return result

#--------------------

print read_molecules(sys.stdin)

Part 2

An Exercise With Sets and Dictionaries

Fri, 20 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

You are working for a nanotechnology company that prides itself on manufacturing some of the finest molecules in the world. Your job is to rewrite parts of their ordering system, which keeps track of what molecules they can actually make. Before trying this exercise, please review:

Introduction
Storage
Dictionaries
Examples
Nanotech Example

Submit your work by mailing Greg:

your final program,
the input file(s) you used to test it, and
a shell script that runs all of your tests.

1. Reading

Data Munging with Regular Expressions

Sun, 15 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Indiana U's Mike Hansen has written a blog post explaining how he used regular expressions to rename and fix some MATLAB files. Even if you don't use MATLAB, you should find lots of useful stuff in it. Thanks, Mike.

Solution to Data Merging with Dictionaries

Thu, 12 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This week's tutorial problem was to merge the data from a set of input files to show how often different species were observed on different dates. The shell pipeline, Python code, and two sample input files follow the video.

{% assign video_title="Data Merging with Dictionaries" %} {% assign video_slug="S-dqlYWs4S0" %} {% assign video_time="00:09:18" %} {% include youtube %}

shell command

grep -h -v '#' *.txt | sort | uniq -c

merge.py

Solution to Data Checking Problem

Wed, 04 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I finally had a chance this morning to record my solution to the final exercise I set the learners from the Space Telescope Science Institute. It demonstrates how to build a little program in Python that checks the consistency of some experimental data; along the way, it uses file I/O, functions, assert statements, lists of lists, and on-the-fly unpacking. As always, feedback on the content and format would be greatly appreciated. (And if anyone would like to post their own solution, either as plain code or as a video, please let me know so that I can include it here.)

Solution to the First Image Processing Homework

Tue, 03 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We had technical issues during yesterday's online tutorial (again), so I have recorded the solution to the image processing exercise we gave the students in Indiana and Toronto. I'd be very grateful for feedback (not just from students): is this a useful way to present ideas? Or should we put our effort into debugging [name of web conferencing software goes here]?

{% assign video_title="Solution to First Image Processing Homework" %} {% assign video_slug="zwl1W4PjaMI" %} {% assign video_time="00:16:25" %} {% include youtube %}

My final code is: