Content on The Carpentries

Changing Our Core Curriculum

Thu, 27 Mar 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We currently say that people must teach task automation, structured programming, version control, and unit testing in order to use the Software Carpentry name. However, many bootcamps either don't teach unit testing at all, or only cover it briefly, while many of us teach data management using SQL, which also seems like a core lab skill for scientific computing. We asked instructors this week whether we should formalize the change, and they responded as follows:

Release 2013.11

Tue, 10 Dec 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We are pleased to announce the release of v2013.11 of our lesson materials. This release represents nearly 2 years of collaborative lesson development from 54 contributors, and features lessons for using the Unix shell to automate repetitive tasks, modular programming in Python and R, personal and collaborative version control with Git, unit testing, and working with databases. Contributors include:

Reorganizing

Sat, 02 Nov 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

TL;DR: We're going to:

split our lesson materials for absolute beginners and people with some previous experience;
build some tools to manage those materials;
update and clarify the guidelines for instructor training, Software Carpentry's scope, making contributions, and use of our name and logo; and
use GitHub issues and blog posts to manage discussion of details rather than our mailing list.

Software Carpentry's Scope

Sat, 02 Nov 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

To help people figure out what's in and what out as we're reorganizing our material, I've drafted an explicit description of our scope inspired in part by AstroPy's idea of an "affiliated" package, which is, "...an astronomy-related Python package that is not part of the astropy core package, but has requested to be included as part of the Astropy project's community."

Domain-specific tools for machine learning and computational fluid dynamics aren't built into LAPACK or NumPy: they're stored in separate repositories and released in their own packages. Similarly, domain-specific lessons don't belong in Software Carpentry. Instead, they should build on our lessons, just as computer vision libraries build on matrix and image libraries. Our examples should be drawn from biology, physics, economics, and similar disciplines in order to appeal to our audience, but everything we include must be accessible to anyone who's done freshman science.

Curriculum Design

Mon, 14 Oct 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I spoke with three different people about curriculum design last week, so this seems like a good time to summarize what I know about doing that properly, how we've actually been doing it for Software Carpentry, why the two are different, and what we hope to do in future. To set the stage, I need to talk about the medical program at McMaster University in Ontario.

How Much Testing Is Enough?

Tue, 24 Sep 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

When we teach testing, we are sometimes asked, "How much is enough?" The short answer is, "It depends." To help construct a longer answer, we'd like to find out how much testing scientists do in other contexts. For example, the Cosmic Origins Spectrograph (COS) and the Space Telescope Imaging Spectrograph (STIS) are two of the instruments on the Hubble Space Telescope. During Cycle 21 (which starts this month, and runs for about a year), they will spend roughly 25% and 67% of their time respectively on calibration:

Connecting Bootcamp Content to Motivation and Best Practices

Wed, 03 Apr 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As our next Software Carpentry bootcamp quickly approaches, we have been struggling with many of the usual questions of what to teach and what not to teach. We expect a broad array of skills and experience (pre-assessment is yet to be completed), and want to make sure that all the learners leave with a clear motivation to build on what they learn.

Testing Image Processing

Sun, 17 Mar 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Testing has always been part of Software Carpentry, but it's also always been one of our weak spots. We explain that testing can't possibly uncover all the mistakes in a piece of software, but is useful anyway, then talk about unit testing and test-driven development. Separately, in the extended program design example, we demonstrate how to refactor code to make it more testable.

What we don't do is show people how to test the science-y bits of scientific software. More specifically, our current material doesn't contain a single example showing how to check the correctness of something that does floating-point. You won't find much mention of this in books and articles aimed at mainstream programmers either: most just say, "Oh, round-off," then tell you to use an almostEquals assertion with a tolerance, without telling you how to decide what the tolerance should be, or what to do when your result is a vector or matrix rather than a single scalar value.

Moving Up and Moving Down

Wed, 05 Dec 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our existing workshops/material are aimed at people who know enough to write a hundred-line script, but don't yet use version control or do any systematic testing. Many scientists either haven't gotten this far, or are well past it, and we'd like to start helping them. We think that means we need:

"Software Carpentry for Complete Beginners", which would only cover basic programming and version control, and
"Advanced Software Carpentry", which would cover parallel computing, integration with legacy code written in C/C++ and Fortran, and so on.

The challenge, as always, is resources: doing this requires people to put together the content and teach half a dozen times to tune it and help show others how to teach it as well. It's a big commitment, but the potential rewards are pretty big too. If you're willing to commit a hundred hours or so to help make the world a better place and give your career a boost, please get in touch.

Updating Our Reading List

Sat, 17 Nov 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We're planning to launch an update to this web site in the next few days, and as part of that, we're revisiting some of our content. For example, we'd like to shorten and update our recommended reading list—the current short list is below, and we'd welcome suggestions for additions. However, if you'd like to put something in, please suggest something to take out (so that the list doesn't become a Sunday stew).

Pelican Guts: on content management for Software Carpentry

Thu, 01 Nov 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This morning I spent some time talking over the content management problem for the Software Carpentry website with Greg. To recap: we'd like to make it easier for anyone to contribute and manage the website content (lecture pages, new bootcamps, events, etc). For example, there are plenty of bits of site building that could be automated so that contributing a new lecture or topic doesn'thave to involve wiring in links to a table of contents, forward and backward links to other lectures/topics, and so on.

Usability Testing and Instructional Design

Sun, 28 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This is a story in several parts.

1. From Guido van Rossum's intermittently-updated blog about the history and design of Python:

Python's first and foremost influence was ABC, a language designed in the early 1980s [that] was meant to be a teaching language, a replacement for BASIC, and a language and environment for personal computing. It was designed by first doing a task analysis of...programming...and then doing several iterations that included serious user testing... ABC's authors did invent the use of the colon that separates the lead-in clause from the indented block. After early user testing without the colon, it was discovered that the meaning of the indentation was unclear to beginners being taught the first steps of programming. The addition of the colon clarified it significantly: the colon somehow draws attention to what follows and ties the phrases before and after it together in just the right way.

This story and others like it were something of a revelation to me when I first encountered Python in the late 1990s. Usability testing of programming languages? Huh—why isn't everyone doing that?

Mozilla Web Literacies White Paper

Fri, 26 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The Mozilla Foundation (which has been our home for the last eight months) have posted a white paper on web literacies that outlines their vision for the web:

...the world's largest public resource, the operating system of the future, and (we believe) one of the greatest drivers of happiness and human flourishing the world has ever seen.

The skills/competencies/literacies grid gives a condensed overview of how beginners and intermediates could explore, create, connect, and protect with and on the web. Over the coming months, we'll be working to find ways of connecting this with scientists. If you'd like to help, please get in touch.

Prime Numbers, Biologists, and Data Visualization

Thu, 25 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A couple of days ago, Titus Brown posted a question from Randy Olson: are there relevant, non-abstract problems (for biologists) that you can solve with minimal programming skills? I.e., what's the biology equivalent to traditional intro CS problems like "calculate the primes".

The answer has three parts:

Calculating primes, finding the longest line in a file, etc., are lousy introductory problems because they're completely valueless to most of the intended audience. You have to already believe you really, really want to learn how to program, and be good at delayed gratification, to get through them to the good stuff. (This isn't just true in computing: see the discussion in How Learning Works of the impact of motivation on learning.)
Guzdial, Ericson, et al's work at Georgia Tech showed pretty conclusively that a media-first approach to computing has better outcomes, i.e., if people are manipulating images (and audio and video) right from day 1, they'll retain more of what they learn, and more students will stay in the program. (But note: you probably can't use an industrial-strength image manipulation library like PIL for intro teaching, at least not directly: you need something that assumes less background knowledge both in domain and skills.)
Robbins, Senseman, and Pate have been piloting a "visualization first" programming course for biologists (see this paper if you can—apologies for the paywall). While it's early days, it seems that this should have the benefits of "media first" for scientists. For example, a programming exercise could be "draw the curves in red if the end point is less than the starting point, and in blue otherwise".

We haven't reorganized our intro material around this idea yet, partly because of inertia, but partly because of the installation headaches of getting visualization working on N platforms in a two-day workshop. I'm very keen to try it out, though, particularly if the IPython Notebook really does make simple visualization simple to do on all major platforms.

25 Questions

Tue, 23 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've tried several times to define Software Carpentry's aims and content in terms of the questions that researchers ask (see for example our competence matrix and this post mapping eight questions to our current content). Our latest attempt is crowd-sourced with a small 'c': the people who are taking part in the first round of our online study group for instructors and would-be instructors put together their own lists of questions, along with the answers they'd expect from novice, intermediate, and expert scientific programmers. I've consolidated those lists into 25 questions, which I'd like to boil back down to no more than 10. If you'd like to take a crack at doing that, please post your suggestion as a comment, and I'll send you a Software Carpentry t-shirt if we use it.

Key Points

Tue, 23 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

On the flight back from Vancouver yesterday, I finally did what I should have done eight months ago and compiled the key points from our core lesson content. The results are presented below, broken down by lesson and topic; going forward, we're going to use something like this as a basis for defining what Software Carpentry is, and what workshop attendees can expect to learn.

Rebuilding Redux

Fri, 12 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The time has come to replace our creaking combination of web tools with something that will let us do more for more people while spending less time switching between browser tabs. To recap an earlier post, we currently use:

WordPress for our blog, web site pages, and comments;
YouTube to host our videos (links to which are embedded in the WordPress pages);
Subversion (administered via the Dreamhost control panel) to manage our slides, code examples, and other files;
Mailman (administered the same way) to manage mailing lists;
a Google Map to show the locations of past and future bootcamps;
a Google Calendar to show their dates;
Google Analytics to track web site traffic;
EventBrite to handle event signup; and
OpenBadges to keep track of who's accomplished what (currently in beta).

We also use Vidyo for video conferencing (primarily tutorials), and run Etherpad beside it so that people have a text channel for asking questions, posting links and code fragments, taking minutes, and so on. As I pointed out in that aforementioned post, this means admins need at least six different logins to get things done, and need to copy a lot of information from place to place. For example, whenever we commit to a new bootcamp, someone needs to:

Transitioning to the IPython Notebook

Thu, 04 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Here at Software Carpentry we've been teaching Python for a little while. We teach Python in our bootcamps and we have a library of instructional videos on Python topics including basics, dictionaries, regular expressions, and object oriented programming.

As usual, though, we're not satisfied with the status quo. For both our bootcamps and our online material we're in the process of switching to the IPython Notebook. The Notebook lets the user write and execute Python code and see the results, all inside their favorite web browser.

Best Practices for Scientific Computing

Wed, 03 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The following pre-print is now available on arXiv:

Best Practices for Scientific Computing

D.A. Aruliah^a, C. Titus Brown^b, Neil P. Chue Hong^c, Matt Davis^d, Richard T. Guy^e, Steven H.D. Haddock^f, Katy Huff^g, Ian Mitchell^h, Mark Plumbleyⁱ, Ben Waugh^j, Ethan P. White^k, Greg Wilson^l, and Paul Wilson^g.

^aUniversity of Ontario Institute of Technology, ^bMichigan State University, ^cSoftware Sustainability Institute, ^dSpace Telescope Science Institute, ^eUniversity of Toronto, ^fMonterey Bay Aquarium Research Institute, ^gUniversity of Wisconsin, ^hUniversity of British Columbia, ⁱQueen Mary University London, ^jUniversity College London, ^kUtah State University, and ^lSoftware Carpentry.

What Would You Like in an Instructor's Guide?

Mon, 01 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of our goals for the next three months is to create an instructor's guide for Software Carpentry to help people teach this material more effectively: something akin to the SSI's "How to run a Software Carpentry bootcamp", but larger, and focused on the content. For example, when we teach the Unix shell (which is normally the morning of the first day), we actually don't introduce many commands—last time I actually checked, I only showed learners pwd, mkdir, cd, ls, rm, mv, cat, head, tail, wc, cut, and history. Instead, our real aims are to give them what they need to get through subsequent programming exercises in Python (i.e., creating data files, moving program files around, etc.), and to:

Why This Stuff Is Hard To Teach

Thu, 20 Sep 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

If we get funding to continue our work (we hope to find out in a month), one of the first things we want to do is put together an introduction to web programming for scientists. As I've remarked many times before, we won't try to teach people how to build web applications: all we can do in the time we have, starting from what they know, is teach them how to create security holes. What we would like to show them is how to pull data off the web and post data of their own for others to consume, but even that turns out to be a lot harder than it should be.

What's In Your Stack?

Tue, 18 Sep 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As a long-delayed follow-up to a conversation with Travis Oliphant: based on our experience, researchers who are computational novices want more than numerical computing and visualization tools. They want a complete stack—an end-to-end, nachos-to-cheesecake [1] solution to their basic computing needs, including an editor, a debugger, a version control system, blogging tools, data management tools, some way to create presentations and papers, and so on. My personal stack includes:

Systematic Curriculum Design

Sun, 16 Sep 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Executive summary: we'd appreciate your help organizing and motivating our material better.

One of the good things about traveling is that it gives me time to think. One of the bad things about thinking is that every time I do, I wind up with more work than I had when I started. For example, to organize and motivate our content, I'm using eight questions that scientists frequently ask:

Patterns Wanted

Wed, 12 Sep 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

At some point or other, most programmers have encountered the idea of design patterns in software, and many (including myself) have been zealous about them, at least for a while. They haven't actually revolutionized either the practice of software development or the way we teach it, but becoming familiar with them is to programmers what learning the Beatles' greatest hits is to musicians.

Free As In Pretty Much Whatever You Want

Tue, 04 Sep 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A couple of different people have asked us recently whether they can use our materials in their courses. The answer is an emphatic "yes": all of our slides, essays, posts, and what-not are covered by the Creative Commons — Attribution license, and all of our code is covered by the open source MIT License, so basically, you can do whatever you want, whenever you want, without seeking special permission. (And yes, this means you can charge for your courses, even though we don't charge for ours.) The only requirement (imposed by the CC-BY license) is that you cite us as the source, and of course we'd be grateful if you'd point us at your material and/or send us any changes you make.

Lifted by the Audience

Sun, 02 Sep 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I spent Thursday and Friday recording most of the material we've been using in workshops for the past six months [1]. One thing that kept bugging me was how flat and uninspiring it was to talk to a camera: I had forgotten just how much a live audience energizes me. I also realized how much I rely on people's questions, and the expressions on their faces, to keep me on track: it took me almost 15 minutes to remember that I needed to explain that Python uses indentation to show nesting, whereas I always do that at the right time in a live class because someone either asks or looks puzzled.

A Problem With Badges

Wed, 29 Aug 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We issued the first learning badges for Software Carpentry a couple of weeks ago, and in doing so, uncovered a significant flaw in the design of existing badging systems. My intent was to give a bunch of people "Instructor" badges; by mistake, I set the pulldown to "Organizer", then clicked "Issue". As a result, two dozen people got the wrong kind of badge—and there doesn't appear to be any way for me to undo that. According to Dave Lester, who created the WPBadger plugin for WordPress that I'm using:

What We Talk About When We Talk About Software Carpentry

Mon, 20 Aug 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A Question and Answer Matrix for Software Carpentry

Tue, 14 Aug 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Following up on yesterday's post about applying educational principles to this course, here's a not-yet-completed Q&A matrix for this course. The section headings are questions people ask (or equivalently, tasks they want to perform). The headings underneath are the major topics Software Carpentry covers, and below each of those is my attempt to relate those topics to the questions. "TBD" means "I haven't written it yet", while "N/A" means "I can't think of any relationship." This matrix is going to be the basis of our next big reorganization of material (which should start this fall), so we would be very grateful for your input:

Record and Playback

Mon, 30 Jul 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The biggest bottleneck Software Carpentry faces right now is a shortage of experienced instructors. To help fix that, we are going to record a complete presentation of our core two-day material so that people who want to teach it themselves can see how we say things, as well as what we say [1, 2].

As soon as we say "record", though, we have to ask, what exactly are we recording? Audio and video of a presenter in front of a whiteboard? Sure—that helps humanize the presentation. But what about the presenter's desktop? Viewers definitely need to see it, but should they see an MP4 in which the text on the presenter's screen appears as colored pixels arranged in the shapes of characters, or should we record the characters directly? I think the latter is by far the best option, since:

Handling Variant Configuration Files

Tue, 26 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of our learners came to us with a problem last week. The program she uses depends on some complex configuration files, which she'd like to store in version control. However, a couple of parameters change depending on the machine the program is running on. She doesn't want to check those changes into version control over and over again; what should she do?

Ten Simple Rules

Thu, 07 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The "Ten Simple Rules" series being run in PLoS Computational Biology has a lot of useful gems. Written by editor-in-chief Philip Bourne and others, the entire collection to date is available as a single PDF, but for those who prefer bite-sized reading, here are ten simple rules for:

...Starting a Company (PDF)
...Getting Involved in Your Scientific Community (PDF)
...Teaching Bioinformatics at the High School Level (PDF)
...Developing a Short Bioinformatics Training Course (PDF)
...Getting Help from Online Scientific Communities (PDF)
...Building and Maintaining a Scientific Reputation (PDF)
...Providing a Scientific Web Resource (PDF)
...Getting Ahead as a Computational Biologist in Academia (PDF)
...Editing Wikipedia (PDF)
...Organizing a Virtual Conference—Anywhere (PDF)
...Chairing a Scientific Session (PDF)
...Choosing between Industry and Academia (PDF)
...Combining Teaching and Research (PDF)
...Organizing a Scientific Meeting (PDF)
...Aspiring Scientists in a Low-Income Country (PDF)
...Graduate Students (PDF)
...Doing Your Best Research, According to Hamming (PDF)
...a Good Poster Presentation (PDF)
...Making Good Oral Presentations (PDF)
...a Successful Collaboration (PDF)
...Selecting a Postdoctoral Position (PDF)
...Reviewers (PDF)
...Getting Grants (PDF)
...Getting Published (PDF)

So, what would your ten simple rules for good computational science be?

What Skills Are Required to Implement Open Access?

Mon, 04 Jun 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Late last night, local time, the Access2Research petition at whitehouse.gov got 25,000 signatures, which means that the White House is committed to responding. As someone who no longer reviews for journals that aren't OA, I obviously think that's great news, but it once again raises the question: what concepts and skills do researchers need in order to implement open access? Jon Udell has written at length about some things (most recently here), but to sharpen up the discussion:

SoundSoftware 2012: Workshop on Software and Data for Audio and Music Research

Wed, 30 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The SoundSoftware project — a UK based project aiming towards software sustainability in the Audio and Music Research community — is organising a one day workshop on "Software and Data for Audio and Music Research". The aim of this workshop is to discuss issues such as robust software development for audio and music research, reproducible research, management of research data, and open access.

An Exercise With Functions and Plotting

Sun, 06 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Let's say you have a text file called workout.csv that contains information about your workouts for the month of March:

# date, kind of workout, distance (miles), time (min)
"2012, Mar-01", run, 2, 25
"2012, Mar-03", bike, 10, 55
"2012, Mar-06", bike, 5, 20
"2012, Mar-09", run, 3, 42
"2012, Mar-10", skateboarding, 2, 10

# Broke my leg :(

"2012, Mar-11", Wii, 0, 60
"2012, Mar-12", Wii, 0, 60
"2012, Mar-13", Wii, 0, 60
"2012, Mar-14", Wii, 0, 60

It's a common-separated value (CSV) file, but contains comments and blank lines. The first line (a comment) describes the fields in this file, which are (from left to right) the date of your workout, the kind of workout, how many miles you traveled, and how many minutes you spent.

Video Update

Thu, 12 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Back in February, we asked people to make short screencasts of themselves solving a simple programming problem. The submissions convinced us that it's a good idea, but so many people ran into so many problems that we're taking a step back and trying to write better instructions, select a simpler (more stable, more reliable) set of tools, etc. Some people's screen recording software stopped recording at 10 minutes without giving any signal; others produced files that play properly on some platforms, but have no audio on others; and screen resolution problems have made text unreadable in several cases. I didn't think this would be so hard in the early 21st Century, but then, I say that a lot...We'll pick this up again as soon as we can; in the meantime, our thanks (again) to everyone who has contributed so far.

Straw Man for Web Programming

Tue, 10 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Last week, I asked what we should teach researchers about the web. I think that I have an answer, and that the easiest way to describe it is by describing what we want learners to be able to build when we're done. So, imagine you are studying changes in rainfall due to climate change in North America. As part of that work, you're comparing results from your simulation with historical data from Environment Canada. Since your calculations may be useful to other scientists, you want to share them on the web. You are therefore going to build a command-line tool so that:

What to Teach Researchers About the Web

Sun, 01 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One reason I'm reflecting on what I've learned in the last two years is a question that is back on the top of my work pile: what should we teach researchers about the web? Partly, it's a priority because I'm currently embedded in Mozilla; their mandate is to defend and extend the open web, and their educational efforts are all aimed at that, so I ought to be doing something too. The real reason, though, is that a lot of things have brought this into sharper focus recently:

What We Teach in Two Days

Fri, 30 Mar 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This week's workshops at MBARI and NERSC both had more lecturing and less hands-on practical work than either I or the students would have liked, but when we're trying to squeeze so many things into two days, that's probably unavoidable. We hope that the online tutorials we're going to run over the next few weeks will make up for that by giving learners a chance to practice their skills at leisure.

Programs as Experimental Apparatus

Mon, 05 Mar 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Suppose you have two photographs of a patch of the night sky:

and you want to know how different they are. The simplest way would be to see if any of the pixels' values differ, but that's pretty much guaranteed to return a "yes". A better measure is to see how many pixels differ by more than some threshold, but that raises two questions:

Badges (Finalized)

Mon, 27 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have finalized our first set of Software Carpentry badges—with luck, we'll list the first set of recipients later this week.

	for people who complete the core curriculum
	for helping to teach (either workshops or online)
	for organizing workshops
	for creating (or fixing) content

Frustration (continued)

Mon, 27 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It's been a frustrating couple of days. To recap, I want to convert our material from PowerPoint to HTML5 to make it easier for people to fork and merge, to make things easier to re-style, because it's an open format, and so on. David Seifried has welded an HTML5 audio player to Caleb Troughton's deck.js to create a display tool, which I'm very pleased with, but the content is killing me. Seriously. Here are three slides taken from our first episode on the Unix shell:

Fourth (or Sixth) Online Tutorial

Fri, 24 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

For the past four weeks, I've been meeting online with learners from the Space Telescope Science Institute to work through some Python topics and exercises. (We split the group in two for a couple of those sessions to accommodate different interests and levels.) I think it's been going pretty well: about half the people who took part in the two-day workshop in January are still with us, and we've just covered basic image processing with PIL. I'm looking forward to finding out whether the model scales to more groups with more people in the coming four months.

Should We Relocate Our Repository?

Thu, 23 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Software Carpentry has been an open project since 2004: MIT License for the code, Creative Commons for everything else. All our stuff has lived in a public Subversion repository since then as well—it's at http://svn.software-carpentry.org/swc if you want to check it out. Today's question is, should we move that repo off our server and onto GitHub, BitBucket, or some other repo-hosting service? We want to make it easy for people to remix our content; as I said back in December, that means making it easy for them to fork, merge, and share. Would you be more likely to do this if our slides, learning plans, diagrams, and code samples were on some well-known hosting site rather than in Subversion on our own machine?

Watch Me: Trial Run

Wed, 22 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A dozen people have come forward since I asked last week for volunteers to make short screencasts showing how they program. I just sent them a sample problem to work on to test things out (see below the fold); the videos they create won't be made public, but I hope it gives readers an idea of the scale of problems we're going to be looking at. If you have suggestions for interesting problems of a similar size, please add them as comments on this post.

Badges (Mark 1)

Tue, 21 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of our key deliverables for the Sloan Foundation-funded work is a badging program built on top of Mozilla's Open Badges Initiative. Riffing on our new logo, Carri Han has designed three badges for us:

	for people who have mastered our core content
	for people who have organized and run workshops
	for people who have created content

Please let us know what you think.

A Flash (well, MP4) from the Past

Sun, 19 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

In July 2009, we held a one-day symposium on open science at the University of Toronto. I recently uploaded video from those talks to YouTube; the audio is a bit shaky, but I hope they're useful despite that. The talks are linked below.

Titus Brown: Choosing Infrastructure and Testing Tools for Scientific Software Projects

Cameron Neylon: A Web Native Research Record: Applying the Best of the Web to the Lab Notebook

Slide Drive

Wed, 15 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Speaking of new kinds of content (which I've been doing a lot), David Seifreid has built a working prototype of a new slideshow tool that combines deck.js with an HTML5 audio player. You can check out a demo or grab the source from https://github.com/dseif/slide-drive. Slides are pure HTML like this:

<section popcorn-slideshow="24">
 <h2>Solution</h2>
 <p>Short intensive workshops</p>
 <div>
 Our solution combines short, intensive workshops...
 </div>
 <div popcorn-slideshow="27">
 <p>Plus self-paced online instruction</p>
 <div>
 ...with self-paced online instruction.
 </div>
 </div>
</section>

which combines slide pages and transcripts in a single file suitable for diffing and merging. (Images are still in external files, but I can live with that.) You can pause the slideshow at any point to select and copy the content (something you definitely can't do with a video), and we'll add support for translations into other languages and so on.

Watch Me: Volunteers Wanted

Wed, 15 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Back in 2007, Jon Udell observed that screencasts facilitate accidental knowledge transfer in a way that more traditional media don't. As I said yesterday, we'd therefore like to start recording short screencasts of programmers thinking aloud as they solve small problems using their preferred tools. The aim is to show learners how to program—what order to write things in, how to debug, when and how much to test, and so on. Everything will be covered by the same Creative Commons license as our other material, and made freely available for remixing and other use.

New Kinds of Content

Tue, 14 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Mark Guzdial, whose blog on CS education is always interesting, recently posted about using worksheets to help people learn to write programs. As he says, research going back 30 years shows that reading and writing skills develop independently; there's also a ton of research showing that partially-worked examples are a very effective (possibly the most effective) way to teach people new skills.

Which immediately suggests two questions:

Advertising Flyer

Mon, 13 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've put together a one-page flyer to advertise the course (each site would customize the "Local info" space at the bottom). Please let us know what you think...

Formatting Revisited

Mon, 13 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

David Seifreid has been working for the past week to combine deck.js, a Javascript-plus-CSS slideshow package, with the AudioJS audio player, so that we can re-do slides as pure HTML5 (instead of using PNGs exported from PowerPoint). At the same time, I'm trying to turn the course notes into long-form prose (what used to be called a "book") for people who prefer reading at leisure. How should all this content be managed? My previous post on 21st Century teaching formats described what I'd eventually like, but the tool I want doesn't exist yet, so what can be done now?

Our New Look

Mon, 13 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have a new logo:

which is also available in stacked form for coffee mugs (thanks, Steve):

We also have some merchandise, and the first batch of laptop stickers is on its way :-)

Pre-Workshop Questionnaire

Sun, 12 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've been asking people to fill in the short questionnaire below before our workshops in order to give us a better idea of what they want and what they already know. How do you score, and how could we improve the questions?

Roughly what fraction of your working time do you spend creating, modifying, or testing software?
What operating system will you be using in this class?
- Linux
- Mac OS X
- Windows
Do you understand the following Unix shell commands well enough to explain them to someone else?
- ls data/*.txt
- find ~ -name '*.py' -print
Do you understand the following Subversion commands well enough to explain them to someone else?
- svn update
- svn diff -r 1723
Do you understand the following Python statements well enough to explain them to someone else?
- {'east' : 5, 'west' : 11}
- __init__(self, *args)
Do you understand the following testing concepts well enough to explain them to someone else?
- fixture
- mock object
Do you understand the following Make terms and commands well enough to explain them to someone else?
- dependency
- cp $< $@
Do you understand the following SQL terms and commands well enough to explain them to someone else?
- select * from data where data.left < data.right;
- inner join
What do you hope to gain from this workshop? What could we do to make it most useful for you?

Comparing Software Carpentry to CS Principles

Thu, 09 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A lot of new educational initiatives in computing have sprung up in the past couple of years, ranging from Mozilla's Hackasaurus to the UK rethinking its grade-school curriculum. One of the biggest is the "Computer Science: Principles" project in the United States, which, with backing from the National Science Foundation, is trying to re-define the high school Advanced Placement course. They have some well-grounded ideas about what should be taught and why; how does Software Carpentry compare?

Multiple Pitches

Thu, 09 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Based on the feedback we've had on my first attempt [no longer online] at "Software Carpentry in 90 Seconds", it seems as though we should actually create three videos:

one aimed at potential participants, i.e., students, workshop organizers, and content creators (since we're hoping these will largely be the same people);
one for potential sponsors, who are both funding bodies and the senior scientists who set strategic direction for departments and disciplines; and
some "talking head" testimonials from former participants.

Does this sound like the right breakdown?

Software Carpentry in a Minute and a Half

Thu, 02 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've recorded a first draft of the quick introduction I mentioned yesterday. Feedback would be very welcome.

Re-doing the Three-Minute Pitch

Wed, 01 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It's time to revise Software Carpentry's three-minute pitch. Here's what I think I need to say; as always, comments would be welcome.

Opening slide: large logo, the title "Computing Skills for Scientists and Engineers", and a small block at the bottom with the date and license.

Our mission is

to help scientists and engineers be more productive...
...by teaching them basic computing skills

The problem is

Learners and Their Needs

Sun, 29 Jan 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I originally wrote these descriptions as part of a post on formats for learning material. I'm finding them useful in other contexts as well, so I'm re-posting them separately. Our description of our audience describes four scientific users in more detail.

Zuzel likes textbooks. More specifically, she likes prose that she can read and re-read at leisure. She'll use a tablet, but would rather listen to music while she learns than to someone lecturing.
Yeleina prefers interactive learning. She wants to see things evolve on the whiteboard or on the screen; recordings of live coding sessions with voiceover are OK, but slide after slide of bullet points puts her to sleep. She also wants to be able to share ideas about learning content with her peers.
Xanthe is a surfer, not a diver—she wants to skim the pages that Giggle searches turn up and piece things together herself. Bullet points and brief sentences work well for her, particularly in areas she's already familiar with.
Wafiya teaches programming at a city library. She needs to remix content created by other educators to meet her learners' needs, but doesn't have a lot of time to do so. She also needs to be able to find content that fits into her (individualized and group) learning plans.
Veronique is a programmer who is passionate about teaching. She spends several hours a week writing short tutorials, answering questions in Stuck Underflow and other online forums, and occasionally recording screencasts. She'd like to do more with less effort (she finds today's tools frustrating), to make the content she's creating more useful, and to get more feedback from its users.
Ursula's preferences are more constrained than Zuzel's, Yeleina's, or Xanthe's because she is visually impaired. Her main assistive aid, a screen reader, can only "see" text (captions in PNG or JPEG images don't count), and becomes confused when pages are modified in place by Javascript.
Tahura is an assistant professor at a medium-sized university. She has to teach three undergraduate courses each year, one of which is always "Introduction to Scientific Programming". While she'd like to experiment with new teaching methods, her priority right now is to get tenure, so she (reluctantly) sticks to the university's traditional format: three hour-long lectures a week, one homework exercise every three weeks, a midterm, and a final exam.

Terminology

Sun, 29 Jan 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Before going further with the redesign of the Software Carpentry curriculum, I need to define a few terms and their relationships. These definitions refer to another post on learners and their needs, which you may want to read first.

A concept is an atom of learning. It can be a fact (e.g., what a call stack is), a technique (e.g., how to pass parameters to a function), or a rule (e.g., when to copy data).
An association is any connection between two concepts. Obviously, all concepts in a topic should be strongly associated.
A dependency is a prerequisite association between two concepts. Circular dependencies may exist (you can't understand A unless you understand B, which depends on C, which in turn depends on A), but we'll work hard to avoid such cycles, since every particular learner encounters concepts in some sequential order.
A topic is a set of related concepts; for example, the topics "Parameters" includes all of the concepts described above. A concept may be part of several topics, but for learning purposes, we would like its first appearance for an individual learner to be gentler than subsequent appearances for that same learner.
A lesson is an atom of teaching. It will typically comprise a small number (1-4) of closed-related concepts, and be a few minutes long. Where possible, it should be followed immediately by some kind of reinforcement (discussed below). A concept may appear in several lessons (same idea, different contexts).
A tutorial is a specific sequence of lessons on related topics; for example, the tutorial "Functions" would include lessons on parameters, returning values, and so on. Tutorials aren't strictly necessary for free-range learners—they can go through lessons in any order that respects dependencies—but we must define them:
- to give learners like Zuzel the larger narrative arcs they need (learners like Xanthe only need lessons); and
- to give instructors like Tahura, who are working in traditional classroom settings, guidance on how to fill an hour-long lecture.
Reinforcement is something done to help people absorb a concept, such as further examples, class discussion, quizzes, or exercises. All four are problematic:
- Further examples are based on guesses of what learners might not have understood correctly; such guesses are often wrong.
- Class discussion is hard to implement for asynchronous, self-directed learners.
- Multiple choice quizzes can mislead or frustrate learners (most questions have more than one right answer), or fail to clear up learners' misconceptions ("You're wrong, but I won't tell you why").
- Long-answer quizzes and exercises require a human assessor's attention (which is also hard to implement for asynchronous learners).
Assessmentis anything done to determine how well something else has been done. "How well" is important: assessment will not always (or even usually) be black-and-white. Specific kinds are:
- formative assessment, which provides short-term diagnostic feedback to learners and teachers;
- summative assessment, which demonstrates attainment of some level; and
- evaluative assessment, which tell us how well the teaching is working.
Formative assessment is a kind of reinforcement; summative and evaluative are (usually) not.

The diagram below shows these relationships. I've drawn most relationships as many-to-many, but in practice, we hope that most will actually be one-to-many, i.e., that concepts will fit neatly into topics, and topics into lessons. The one-to-one relationship between summative assessments and topics is also a bit misleading: what it's meant to imply is that any particular learner will eventually demonstrate mastery of some topic (e.g., Yeleina will show that she understands functions). This is the smallest "chunk" for which we'd contemplate awarding a badge, though as discussed earlier, we're going to start with something coarser-grained.

Never Mind the Content, What About the Format?

Thu, 26 Jan 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm still gnawing on the problem of how to construct content for 21st Century learning—or, more prosaically, what I should use to build the next version of Software Carpentry. My starting point is the need to serve several different kinds of users [1], whose descriptions I have moved to a separate post on learners and their needs.

Textbook: big blocks of prose in some narrative order, with pictures, either printed or electronic, read at the learner's pace, alone.
- Zuzel likes this.
- Yeleina doesn't.
- Xanthe uses content out of order via the index or search bar.
- Wafiya remixes content from several textbooks to create lessons (by photocopying, merging PDFs, or whatever). Like Zuzel, she has read content in order, but like Xanthe, she mainly uses the index now.
- Veronique has thought about writing one, but (a) doesn't think she has that much to say about any single topic, and (b) is put off by the effort that would be required.
- Note: the comments below about the difficulty of copying, pasting, and altering also apply to electronic textbooks, as do the proposed remedies.
Static slideshow: a page-by-page dump of a PowerPoint deck, possibly accompanied by a transcript of what the lecturer would say when delivering it.
- Zuzel uses this as if it were a badly-written textbook, with the transcript as the prose and the slides as diagrams.
- Yeleina finds it distracting to switch attention back and forth from slides to transcript.
- Xanthe searches the transcript to find what she wants, then curses because her search engine can't "see" the text in the slides. She also hates the fact that she can't copy and paste the code in the slides (since they're PNGs embedded in a web page).
- Wafiya remixes this content like any other. She's too polite to curse, but she finds it tedious to re-type the code that's shown in the slides (but isn't duplicated as text in the accompanying transcript). She's also finds it wearying to have to re-do diagrams: since the slides are PNGs, it's difficult for her to copy part of a slide, move its elements around, and add a few of her own.
- Veronique doesn't create material in this format because she thinks it's old-fashioned and not useful.
- Note: source code can be made available as copy-and-pasteable text directly in the page, or for download; diagrams can similarly be made available as SVGs to facilitate remixing. Doing either currently requires considerable extra work on the part of content creators.
Voice-over slideshow screencast: a video recording of the slides (as they would appear on screen in a lecture) with someone speaking over them, and subtitles.
- Zuzel ignores the video and reads the transcript as if it were a static slideshow. If a transcript isn't available, she (reluctantly) watches the video.
- Yeleina prefers this to a static slideshow, but prefers the doodling screencast described below even more.
- Xanthe hits the "back" button as soon as she realizes it's a video (unless there's a transcript, in which case she curses because she can't copy and paste code out of a video).
- Wafiya directs students like Yeleina to these, but finds them harder to remix than other formats.
- Veronique thinks this format is also old-fashioned and not useful.
- Note: I'm assuming the subtitles are duplicated as a transcript, or available in some other searchable form. I'm also assuming that code is available for donwload or duplicated in the page for coying and pasting, though all of this requires extra work.
Voice-over doodling screencast: a Khan Academy-style recording of someone doodling on a tablet or coding live.
- Zuzel treats this like a slideshow screencast.
- Yeleina likes this format a lot, particularly if she can add comments at specific points and see her peers' comments.
- Xanthe has mixed feelings: she dislikes explanations delivered this way, but frequently watches "how to" videos, since they're more likely to be accurate and complete than written descriptions.
- Wafiya treats these like slideshow screencasts.
- Veronique creates these fairly regularly: they're easy to do, and easy to re-do when systems change or she discovers a mistake.
- Note: I'm making the same assumptions about transcripts, code, and diagrams as above.
Recorded whiteboard lecture: someone with a camera has recorded someone giving a lecture in a lecture hall, and spliced that with whatever was on the lecturer's screen.
- Zuzel treats this like any other screencast.
- Yeleina prefers this to doodling screencasts because she can see the speaker's body language.
- Xanthe treats these like any other screencast, i.e., she'll use it if there's a searchable transcript and things to copy and paste, or if it's a recording of a live "how to" coding session.
- Wafiya treats these like slideshow screencasts.
- Veronique doesn't create these, partly because of the setup required, but also because she doesn't think seeing her adds value—the lesson's supposed to be about the stuff.
- Note: I'm assuming an electronic whiteboard, since video of someone writing on an actual whiteboard is usually illegible.
Radio drama: a voice-only podcast-style presentation.
- Zuzel ignores the audio and reads the text transcript.
- Ditto for Yeleina.
- Ditto for Xanthe.
- Ditto for Wafiya.
- Veronique doesn't create these.
- Note: but for Ursula, who is blind, this is the only format—all the others fold into it. She doesn't need code samples as text for copying and pasting: she needs them so that her screen reader can tell her what's that code contains.
Star Wars: high-quality video with custom animations, cut scenes, and other special effects.
- Zuzel watches these sometimes, but doesn't learn any more from them than she would from a slideshow.
- Yeleina enjoys these, which means she pays more attention to them, which means she learns more (but no more than she'd learn from an engaging lecturer).
- Xanthe doesn't see the point. Unless something blows up.
- Wafiya likes their high production values, and remixes the special effects segments frequently.
- Veronique can't afford to produce this kind of material.
- Note: again, I'm making assumptions about transcripts, copy-and-pasting, etc.
Write your own adventureexploration: typically a set of connected ideas or challenges with explicit dependency information (i.e., you should/must learn A and B before tackling C).
- Zuzel finds the lack of narrative difficult.
- Yeleina enjoys these if each node in the graph is in one of her preferred formats. She enjoys them even more if she is exploring with peers.
- Xanthe ignores the ordering and searches for what (she thinks) she needs. If content is locked down—i.e., if the system won't let her see or search C until she's "completed" A and B—she writes an angry tweet and moves on.
- Wafiya likes this format for several reasons, but only if everything is always visible. First, it tells her how other teachers think ideas connect (something that is missing or out-of-band for other delivery formats). Second, it's easy to remix: again, providing it's open, she can reorder things as she thinks best for particular learners.
- Veronique would like to do this, but has discovered that creating the metadata about dependencies and recommended paths is as hard as writing a textbook.
Wander aroundexploration: lots of little snippets, but no explicit dependency information.
- Zuzel finds this even more difficult.
- Yeleina likes this less than the "write your own adventure" format: she thinks it's no different than just using Giggle to find things.
- Xanthe likes this because it's just like using Giggle searches. In fact, she uses every other format as if it were this one.
- Wafiya feels the same way as Yeleina: she likes having stuff to remix, but she has to do that remixing before this material is useful to those of her students who aren't as independent as Xanthe.
- Veronique creates content like this almost without realizing it by answering questions at Stuck Underflow.
Jam session: a bunch of learners in a room working through material simultaneously.
- Zuzel doesn't like it: it's too noisy for her to concentrate, and she can't go back at her own pace to review.
- Yeleina thinks this is the best... thing... ever.
- Xanthe is Giggling for information as soon as the presenter tells people what the topic is, but will stop and watch carefully when the presenter is typing live on screen.
- Wafiya can only book space to do this occasionally, and even then, she doesn't enjoy improv teaching.
- Veronique enjoys doing this—she's volunteers with a local free-range learning group—but can only find time once every couple of months.
- Note: in theory this can be combined with any of the formats above. In practice, it's almost always short, live lectures interspersed with hands-on practical work.
Personal tutoring: one-to-one instruction, a.k.a. "pair learning".
- Zuzel doesn't mind this, but she really does prefer books...
- Yeleina actually prefers jam sessions, since they tend to be more lively.
- Xanthe likes having someone available to answer questions on demand, but is happy Giggling on her own for most of what she needs.
- Wafiya wishes she could do this with every one of her learners, but there simply aren't enough hours in the day. The personalized lesson plans she draws up are the closest approximation she can manage.
- Veronique does this a lot, but is frustrated that it doesn't scale—she really wants to help more than one person at a time.

I'm sure some of the above is inconsistent or just plain wrong, but here are my takeaways:

Take Out Agile, and Add...What?

Tue, 24 Jan 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Based on the feedback we've received so far (both as comments and by email), it looks like we should take development methodologies (i.e., agile development) out of the core curriculum and replace it with two hours on:

Nothing: there's already too much in the core.
Spreadsheets: because many scientists use them badly.
NumPy and/or Pandas: because many of them are crunching matrices/doing stats.
Visualization: which in practice would mean the basics of matplotlib.
Image manipulation: because it's fun as well as useful, and lets us talk about binary vs. text data.

I am quite arbitrarily limiting options to those five. Please cast your vote (one vote, not three out of five) in comments. We'd be grateful if you could include a brief explanation as well.

Revising the Curriculum

Mon, 23 Jan 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've been thinking some more about what the foundation and core of Software Carpentry actually are (and not just because Jon Pipitone keeps pestering me to do so). My last attempt had a foundation of seven principles and dozen topics in the core. I think I can slim that down even further; in fact, I think three big principles form the foundation of computational thinking:

The What, Why, and How of Bootcamps

Fri, 13 Jan 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've just added a single-page description of the two-day bootcamps we're planning to run in the next six months. In brief, their aim is to ensure that people have a few core skills, so that they can tackle our online material productively, and to help them get past startup hurdles such as software installation. If you have questions, comments, or suggestions, please add them to that page; if you'd us to help you organize and run a bootcamp, please get in touch.

Updating to HTML 5

Fri, 14 Oct 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We are looking for a volunteer to update our episodes on HTML to the new HTML 5 standard. The changes are (probably) fairly minor, and as always, we'll help with post-production (editing of audio, etc.). If you're interested, please get in touch.

Four New Episodes on Databases Using Microsoft Access

Fri, 07 Oct 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks to the indefatigable Ethan White, we have just added four more episodes on using databases with Microsoft Access:

aggregating data
combining data with 'join'
nested queries
handling missing data

We hope you enjoy them.

I'm Not Normally Lost for Words

Tue, 20 Sep 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I mentioned last week that I'm trying to put together a lecture on packaging and installation. It's proving harder than I expected: I'm not normally lost for words, but I'm struggling to get these ones to come together. My goal is to help people understand what happens when they install software on a computer, so that they can:

diagnose and fix problems when things go wrong;
understand why the things that are in packaging tools are there, and how to use them.

After a few false starts, I think the best way to do this will be challenge/response, i.e., trace the evolution of a typical scientific software package from a single Python file shared by email to a mix of Python and C that plays nicely with distutils2. Steps I've identified are:

Progress Of A Sort

Tue, 13 Sep 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As I mentioned a few months ago, I'm going to turn Software Carpentry into a book. Here's the present status:

Chapter	Word Count
Introduction	768
Spreadsheets	3924
Subversion	6405
Python	6345
Interlude: What Is Text?	1071
Functions and Libraries	9409
Interlude: Boolean Logic	921
Case Study: Invasion Percolation	8929
Interlude: How Are Numbers Stored?	1828
Testing	4549
Error Handling	1529
The Shell	12919
Make	5668
Interlude: Provenance	1810
Sets and Dictionaries	7627
Case Study: Phylogenetic Trees	1689
Systems Programming	60
Interlude: Configuring Programs	2146
Numerical Programming	6193
Multimedia Programming	4125
Steganography	1657
Installation	135
HTML and XML	6404
Databases	10676
Regular Expressions	8037
Object-Oriented Programming	4002
Building Desktop GUIs	137
Interlude: Persistence	5984
Web Programming	9639
Security	1285
Performance	8043
Parallel Programming	931
Software Engineering	6391
Epilog	873
Acknowledgments	202
Glossary	12074
Bibliography	3378
Total	167,763

167,673 words might seem like a lot, but based on past experience, I think I'm about halfway to a readable book—editing is always as much work as writing stuff in the first place.

What Happens When You Install Something?

Thu, 08 Sep 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The most frustrating part of this course is always getting things set up on students' machines. (Yes, we could give them all a Linux VM with everything installed properly, but if they're going back to Windows or Mac OS when the course is over, that's what they want to learn on.) Setting up a development environment is often just as frustrating for professionals (it took many of us half a day to get Lernanta running on our machines a few weeks ago), so I think we'd be doing Software Carpentry students a favor if we taught them what actually happens when you install software. Knowing that would help them debug things when they go wrong (and they always go wrong), and also help them understand why packaging their own work to share with others is as complicated as it is. My first cut at a syllabus for such a lecture is:

Teaching Security to Scientists

Fri, 02 Sep 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks to everyone for their suggestions regarding what we should teach about computer security if we only have one hour (the usual constraint for topics in this course). The outline below is based in part on the lecture on security from Version 3 of this course, in part on Rick Wash's excellent study of folks models of computer security, and in part on mistakes I've seen (or made) myself in the past five years. Feedback would be very welcome, but remember: we're teaching scientists and engineers who are programming as a way to do science, not as an end in itself.

Introducing Programming a Different Way

Mon, 08 Aug 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our quick introduction to Python is the module I'm least happy with, so I've been thinking about how to re-design it. I've included a new outline below; comments would be very welcome.

Programming is what you do when you can't find an off-the-shelf tool to do what you want

Scientists' needs are so specialized that there often isn't such a tool
Even if it exists, the cost of finding it may be too high (can't Google by semantics)

Why is programming hard to teach/learn?

New Episode: MATLAB Structs and Cell Arrays

Wed, 15 Jun 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The title says it all: thanks to the tireless Tommy Guy, we have a new episode on MATLAB structs and cell arrays.

Audio Processing in Python

Fri, 10 Jun 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks to Becky Stewart, we now have a 12-minute episode on audio processing in Python. We hope you find it useful—as always, feedback is very welcome.

Five on Systems Programming

Thu, 02 Jun 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks the Software Sustainability Institute's Mike Jackson, we now have five episodes on how to inspect and manipulate files and directories from inside a Python program—many thanks. If you would like to contribute to this project as well, please get in touch.

Damn the Torpedoes (but I could use some help navigating)

Fri, 13 May 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Despite my own calculations (which were optimistic to begin with, and are more so now that I have a new job), I'm going to try to turn Software Carpentry into a book. Here's what's I'm planning to include, and its presesnt status; I would be very grateful for feedback, especially from alumni, about what doesn't need to be here, and what's missing. I think I'm still overshooting the knowledge and needs of far too many people; if you think so too, please let me know.

Managing Data

Mon, 02 May 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have just posted a screencast on managing data written by Orion Buske, a graduate student in bioinformatics at the University of Toronto (and former Software Carpentry TA). We hope you enjoy it.

Chapters

Sat, 23 Apr 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've converted three more Software Carpentry lectures to chapter format, bringing the total to four:

the shell
number crunching with NumPy
regular expressions
software engineering

It's all CC-A licensed, and yeah, the images could be a lot better: they're just screen captures from the original PowerPoint slides, because re-doing them as SVG felt even more like yak shaving than converting the text. Please let me know if this is useful.

Prototyping

Mon, 11 Apr 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The numbers might not make sense, but I'm still curious about whether Software Carpentry would make sense as a book. To find out, I spent five hours converting the existing lecture on regular expressions to chapter format. I'd welcome feedback about how useful this is. In particular, without the highlighting and animation of PowerPoint slides and video, how much harder is it to understand what's going on? What could be done to make it easier? More pictures? (I really want to draw arrows to point at the tabs and spaces in the sample data files...) Fully hyperlinked audio? Side-by-side display of code and text (which would be difficult to squeeze in, but not impossible)? Please let me know...

Using Bein

Thu, 31 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Many thanks to Frederick Ross for putting together a short screencast on Bein, a workflow manager and miniature laboratory information management system (LIMS) built in Python that fills the gap for the working scientist between the classical shell and big workflow managers like Galaxy and major LIMS systems like OpenBIS. Please have a look and see if it could help make your computational life easier.

And I'm on a Horse

Sat, 26 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Patrick Mackenzie (whom I've never met) gave a good lightning talk at the Business of Software that sums up a lot of what we haven't done for Software Carpentry. Any reworking of the material really (really) has to be built around what you (the researchers) need, rather than what we (the programmers) know.

A Better Way to Teach Programming to Scientists

Thu, 24 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This year's SIGCSE conference on computer science education featured a very cool paper by Robbins, Senseman, and Pate, of the University of Texas at San Antonio, called "Teaching Biologists to Compute using Data Visualization". In it, they describe CS 1173: Data Analysis and Visualization using MATLAB, which introduces students in the life sciences to programming using a problem-first approach. Like the media-first approach that Mark Guzdial and his colleagues introduced at Georgia Tech, this rewards students right away—they don't have to wade through a "CS first" morass of data types and arcane rules about Boolean expressions for three or four weeks in order to get to the useful bits. Given another year, I'd have rewritten our intro to Python this way...

Our First Episode on Microsoft Access

Wed, 23 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have just posted our first episode on using a database with Microsoft Access — many thanks to Utah State's Ethan White for creating it.

You'll Need a Large Screen

Tue, 22 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've been working on a graph showing

the connections between the questions this course tries to address,
our answers to them,
the knowledge and skills needed to understand and apply those answers, and
the big concepts behind that knowledge and those skills.

The latest version is linked from the thumbnail below. I know it's tangled, but I hope it's the first step toward a roadmap for redesigning this course to serve your needs better. Feedback and assistance would both be very welcome.

Using a Debugger

Mon, 21 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've just uploaded a new video showing how to use a debugger to track down a problem. Using a debugger instead of 'print' statements will save you a lot of time, and the skill transfers directly to pretty much any language. Please let us know what you think.

Videos of Autumn School Lectures

Mon, 21 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm pleased to announce that video recordings of the Software Carpentry lectures I gave in London last fall are now online at http://soundsoftware.ac.uk/autumnschool2010video. Please excuse the mustache—I don't know what I was thinking... Many thanks to Chris Cannam, Ivan Damnjanovic, Luis Figueira, and of course Mark Plumbley.

Questions and Answers

Thu, 17 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A first cut of a question-and-answer matrix for this course is now up for viewing. It doesn't include everything, and the answers aren't particularly helpful without a second matrix showing how they connect to specific lecture topics and big principles, but I hope it's a start. Please give feedback as comments on this post; we'll add more Q's and A's in the coming days.

Graph Layout, Models vs. Views, and Computational Thinking

Wed, 16 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Money for me to keep working full-time on Software Carpentry hasn't materialized, so as I've mentioned in a couple of recent posts, I'm trying to find a way to organize the course material more coherently in the hope that other people will (finally) start to contribute content as well. As part of that, I have spent the last two days trying to draw a graph showing the questions Software Carpentry seeks to answer, and the concepts that underpin the answers. It's been harder than I expected, and I think the reasons might give some insight into how a computational thinker thinks.

Twenty Questions (Minus Two)

Tue, 15 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Following up on last week's musings about reorganizing the course, we've drawn up eighteen questions that we think cover the reasons people come to this course. That leaves us two short of the traditional twenty: what would you add to this list?

How can I automate this task?
How can I read/control hardware?
How can I control my program?
How do I count things?
How can I clean up this data?
How can I get insight into my data?
How do I track down bugs in my program?
How can I parse this legacy data file?
How can I save data so that I can read it later?
How can I tell if my program is working correctly?
How can I make this program easier to use?
How can I use remote machines?
How can I reuse this legacy program?
How can I share my work with others?
How can I make my program faster?
How can I make my code easier to reuse?
How can I keep track of my work?
How can I plug my code into a framework like Galaxy or Taverna?

Musing About Reorganization

Fri, 11 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm increasingly unhappy with the organization of this course. On the off chance that funding materializes and we're able to undertake a major redesign, I'd like to explain why and ask for your input.

Right now, our lectures are broken into topics along lines a computer scientist would instantly recognize: basic programming, regular expressions, databases, and so on. That is not how members of our intended audience see things when they first come to us—if it was, they probably wouldn't need this course. They start with problems like:

An Easy Place to Start: Systems Programming

Thu, 24 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As a follow-up to recent posts on how to contribute, what better looks like, and Elango Cheran's secure shell episode, here's a specific request: we would like three episodes on systems programming, by which we mean writing programs to:

manipulate files and directories (chdir, stat, etc.)
working with archives (tar, zip, etc.)
run other programs (the subprocess module in all its fearsome glory)

A typical use case is writing a Python script to run a compiled legacy application a few hundred times for slightly different parameter values, putting each run's output in a separate directory that's created on the fly, and so on. If you're interested, please get in touch — it's easier than you think.

Ask, And Ye Shall Receive

Wed, 23 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks to Elango Cheran, we now have an episode on the secure shell—yay!

Three More Episodes on MATLAB

Sat, 19 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Three more episodes on MATLAB are now available, two on input and output, and one on plotting. Thanks to Tommy Guy for putting them together; as always, feedback is appreciated.

Mirroring Software Carpentry

Fri, 18 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As another followup to the recent post on how to contribute, I'd welcome pointers to places that might host or mirror Software Carpentry material. Efforts like MIT's OpenCourseware and CMU's Open Learning Initiative seem to host only the content they produce themselves; the Khan Academy seems open to contributions, though I don't see any, and then there's Academic Earth. Where else could we put our content to get it in front of more people?

How to Contribute

Wed, 16 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Following up on yesterday's post, I have written a short guide explaining how you can help Software Carpentry improve and grow. Reporting bugs, suggesting changes, contributing content, or running the course yourself: they're all a lot easier than you think, and a great way to gain some experience with new kinds of teaching and learning that are becoming more important with every passing year.

First Four MATLAB Episodes

Tue, 15 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have just posted the first four episodes of our introduction to MATLAB; more should be up this week and next. As always, feedback is very welcome.

Audio for Three Software Engineering Episodes

Mon, 14 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've added audio (and a few images) for three of the episodes on software engineering (the introduction, agile development, and the principles of computational thinking). It's a slightly different format than most of our previous episodes, and we need to choose a better way of embedding MP3 audio in web pages, but we hope it's useful as is.

Two More Episodes on Spreadsheets

Mon, 14 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have added two more short episodes on spreadsheets, covering pivot tables and named cell ranges. Please let us know if you find them useful, and what else we should cover to help researchers who use spreadsheets to work with data.

Updates to Spreadsheet Lecture

Fri, 11 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have just posted updates to the first seven episodes of the lecture on spreadsheets: the new version uses Excel 2010, and the screencasts are larger to make things easier to see. Please let us know what you think.

What Computational Science Means to Me

Tue, 08 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

My latest attempt to define what "computational thinking" actually means is now on the web in draft form—comments would be very welcome. In brief, the eight principles are:

It's all just data.
Data doesn't mean anything on its own—it has to be interpreted.
Programming is about creating and composing abstractions.
Models are for computers, and views are for people.
Paranoia makes us productive.
Better algorithms always trump better hardware.
Automation is the key to acceleration.
The tool shapes the hand.

Scripts for Two More Software Engineering Episodes

Thu, 03 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've posted the scripts for two more episodes on software engineering (but not the slides or videos, since they don't exist yet). I'm not happy with the discussion of agile: it feels like too much has been left out, but I'm not sure what to add without breaking our self-imposed 10-minute limit. I also don't know how much our audience needs to know: these episodes are meant to be end-of-course pointers to other material they could look at next, so we're after a big-picture guide to "should I dig deeper" rather than the details. As always, feedback is welcome...

First Episode on Software Engineering

Tue, 01 Feb 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I know we promised a lecture on high-performance computing, but today wound up going in a different direction: we have just posted an episode on empirical results in software engineering, which we hope will set the stage for discussion of how to manage larger teams and projects. We hope you enjoy hearing about some of the science behind the things this course teaches; as always, feedback is welcome.

A Competence Matrix for Software Carpentry

Mon, 31 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Last summer, we tried to create a concept map showing how the big ideas in Software Carpentry related to each other. We weren't satisfied with it, so we set it aside. With our first anniversary coming up in three months, though, it's time to revisit the idea of a high-level overview of the course. To that end, I have posted a first attempt at a competence matrix that organizes skills into levels. Please leave comments on that page telling me what's wrong or missing.

Notes Toward a Lecture on High-Performance Computing

Thu, 27 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As I've said several times now, we're going to add a lecture on high-performance computing to the course. We have rough outlines of the content in the version control repository, and feedback would be very welcome.

Software Carpentry Sprint in July

Wed, 26 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Well this was a nice birthday present: the Python Software Foundation announced today that they will provide some support for a Software Carpentry sprint in Toronto in July. Thanks, folks—we'll post details about what, when, and where as soon as we have them.

How to Cite Software Carpentry

Thu, 20 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

If you are citing Software Carpentry in papers or technical reports, the web site and the 2006 article in Computing in Science & Engineering are probably best; the other articles below provide other background that might also be useful. I hope to update the 2006 article some time this year, but a lot of other things are ahead of it in queue...

Version Control and Newline Conventions

Wed, 19 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

There are two widely used conventions for representing the end of a line and the start of the next. Unix-like systems have traditionally used one character, \n (ASCII 10, "line feed"), while Microsoft DOS and Windows used a sequence \r\n (ASCII 13, "carriage return", \n). Being invisible, the difference of conventions usually goes unnoticed, specially because most modern software can correctly display text written with either one. However, there can be interoperability issues when using editors that follow different conventions.

Slower Than Expected

Mon, 10 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've started work on the high-performance computing lecture, and as part of that, we'd like to set a puzzle for our more advanced readers. We've written several versions of invasion percolation; the two of interest here use a list-based representation for the underlying grid, and a NumPy array. Run either program as:

python filename.py -g -n 31 -v 100

-g turns off the end-of-run graphical display; -n 31 tells the program to use a 31×31 grid; and -v 100 tells it to initialize cells to random values between 0 and 100. Try varying the size, compare the speeds of the two programs, see if you can figure out why one is faster than the other, and then tell us:

First Half of Lecture on Object-Oriented Programming

Thu, 06 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The first four episodes of our lecture on object-oriented programming in Python are now available:

Introduction
Basics
Interfaces
Inheritance

There's (lots) more to come, but early feedback on these would be very welcome.

More Detailed Outline for HPC Lecture

Thu, 30 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've added a more detailed outline for a lecture on high-performance computing to the site; feedback on the content and order would be very welcome, as would pointers to tools that are easy to teach with (as opposed to being powerful in the hands of experienced users).

Elimination

Mon, 27 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm working up another essay on software design, and would like to ask readers of this blog how they handle something that comes up when simulating interacting agents. If your program models the behavior of a flock of birds, it probably looks something like this:

# create birds
birds = []
for i in range(num_birds):
 new_bird = Bird(...parameters...)
 birds.append(new_bird)

# simulate movement
for t in range(num_timesteps):
 for b in birds:
 b.move(birds) # need access to other birds to calculate forces

There's a flaw in this—or at least, something questionable. By the time you are moving the last bird for time t, every other bird is effectively at time t+1. There are many solutions, the simplest of which is to calculate each bird's new position in one loop, then update the bird in another:

Local Subversion Repositories

Sun, 26 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A colleague in the UK who is going to teach Software Carpentry asked about setting up repositories. In particular, he doesn't have a server where he can create accounts and repos, so he was thinking of using Git or Mercurial, and having students host their repos on their own machines. That's not actually necessary: if you're going the locally-hosted route, and giving each student a separate repository, you can still use Subversion: just use the "file:" protocol for connecting instead of "http:" or "svn+ssh:". For example:

Extended Examples

Thu, 23 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We'd like to add more extended examples to this course, both because they're fun and because they're a good way to show how our topics relate to one another. Right now, we have:

an entire lecture on invasion percolation (which we'll shorten),
phylogenetic tree reconstruction, and
how to recommend papers to people.

We plan to add a simple N-body simulation (easy to do badly, motivates testing, visualization, and a bit of numerical analysis), but after that, we'd like to find some examples from psychology, linguistics, and other areas that are starting to do more computational work. Parsing text files full of patient records, putting the information in a database, doing some analysis using SQL, and visualizing the results is one possibility; what else would you recommend or like to see?

Slides for First Five OO Episodes Online

Mon, 13 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Slides for the first five episodes of our lecture on object-oriented programming are now online; we'd welcome feedback on the approach we're taking. If you like what you see, we'll try to have recordings up before the holiday break.

Performance and Parallelism

Fri, 10 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Some topics for a lecture on parallel programming:

how to measure/compare performance (raw speed, weak scaling, strong scaling, Amdahl's Law, response time vs. throughput)
the register/cache/RAM/virtual memory/local disk/remote storage hierarchy and the relative performance of each (order of magnitude)
in-processor pipelining (or, why branches reduce performance, and why vectorized operations are a good thing)
how that data-parallel model extends to distributed-memory systems, and what the limits of that model are
the shared-memory (threads and locks) model, its performance
limitations, deadlock, and race conditions
the pure task farm model, its map/reduce cousin, and their limitations
the actors model (processes with their own state communicating only through messages, as in MPI)

It's too much (each point should be an hour-long lecture in its own right, rather than 10-12 minutes of a larger lecture); what do we cut, and what's in there that doesn't need to be?

Where Are My Keys?

Thu, 09 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I was looking through some Python code a few days ago, and noticed that its author was using this:

if something in dict.keys():
 dict[something] += 1

instead of:

if something in dict:
 dict[something] += 1

It seems like a small difference, but it's actually a very important one. The second form checks to see whether something is a key in the dictionary dict. The first form, on the other hand, creates a list of all the keys in the dictionary, then searches that list from start to finish to see if something is there. They both produce the right answer, but the fast form does it in constant time, while the slow form (the one with the call to dict.keys()) takes time proportional to the size of the dictionary. The really bad news is, the "error" is silent.

How Do You Manage a Terabyte?

Wed, 08 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This question has come up a couple of times, and I'd welcome feedback from readers. Suppose you have a large, but not enormous, amount of scientific data to manage: too much to easily keep a copy on every researcher's laptop, but not enough to justify buying special-purpose storage hardware or hiring a full-time sys admin. What do you do? Break it into pieces, compress them with gzip or its moral equivalent, put the chunks on a central server, and create an index so that people can download and uncompress what they need, when they need it? Or...or what? What do you do, and why?

Approaching Objects from a New Direction

Tue, 07 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have just posted early drafts of slides for the first two episodes of our lecture on object-oriented programming. These take a "nuts and bolts" approach, showing more about how objects work than is usual in an introductory course in the hopes of dispelling some of the mystery and confusion that often surrounds them. Feedback would be very welcome...

Pins, Balls, and Arbitrary Decisions

Mon, 06 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We'd like to include more extended examples like invasion percolation in this course, but they're surprisingly hard to write. One that seems simple at first is a Galton box simulator. As the diagram below shows, this is simply a a vertical board with interleaved rows of pins. Small balls are dropped in at the top, and bounce left and right as they hit pins until they land in the boxes at the bottom. Since each bounce randomly goes left or right, the distribution of balls is binomial, which, in the limit, approximates a normal distribution.

Cast Your Votes

Thu, 02 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The solstitial holiday is approaching fast, but with a bit of luck, we should be able to get one more lecture up on the web before it arrives. Topics we could tackle are:

a survey of empirical results in software engineering,
building desktop GUIs,
handling XML, or
object-oriented programming.

(Web programming isn't on the list because it will take more time than we have.) If you have a preference, please cast your vote in a comment on this post...

Prerequisites (or, When to Say No)

Thu, 02 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

How much should Software Carpentry assume students know before they start? Or to put a sharper point on it, how much should this course require students to know? To date, we have said that students should have at least one prior programming course, and have to understand loops, conditionals, functions, arrays, and simple file I/O: the kinds of things that are usually (but not always) covered in a CS-1 course, and hopefully remembered thereafter. To make this more concrete, we've said that students should be able to solve this problem in the language of their choice:

First Four Episodes on Multimedia

Tue, 30 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our first four episodes on multimedia programming, which cover basic image processing, are now online—comments are welcome.

Next Part of Persistence Essay Online

Fri, 26 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The second part of the discussion of persistence is now online. We're almost up to how Python's 'pickle' module works; should get there over the weekend or on Monday.

Hours So Far

Thu, 25 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The chart below shows cumulative hours on Software Carpentry since we started working on Version 4 in May. We're not as far along as I'd like, but progress has been pretty steady...

Four Episodes on Matrix Programming

Tue, 23 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Tommy Guy has recorded four introductory episodes on matrix programming with NumPy covering:

Introduction
Basic operations
Indexing
Linear algebra

Please have a look and let us know what you think.

Phylogenetic Trees

Tue, 23 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Elango Cheran has recorded an episodes for the sets & dictionaries lecture on reconstructing phylogenetic trees. It's a neat application of dictionaries to a problem that's usually explained as a matrix. Please check it out and let us know what you think.

Now Annotated

Fri, 19 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have added brief notes on the books in the course's bibliography—we hope you find them useful.

New Section for Essays

Wed, 17 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

By popular request, we have moved the longer essay-style posts to a new section of this web site. We have also added a new essay on saving and loading data that also talks about metadata, registries, and the open/closed principle.

Ratios and Rework

Tue, 16 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It's been six months and a bit since we started working on Version 4 of this course, so I'd like to share two things we've learned about creating online tutorial videos:

It takes a long time.
A lot of that time feels like it should be unnecessary.

Let's start with "a long time". It takes me 3-5 hours to prepare a good slide deck for an hour-long lecture that I'm going to deliver in person. A slide deck for a screencast takes 8-10, because I need to prepare more slides: since there isn't a lecturer making eye contact and pointing at things to keep viewers engaged, a slide deck for online use has to have many more small transitions.

Counting Things (Part 2)

Sun, 07 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This post has been moved.

Counting Things (Part 1)

Sun, 31 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This post has been moved.

Dubois on Maintaining Correctness

Sat, 30 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Something else I didn't get to at the Met Office last week:

Paul F. Dubois: "Maintaining Correctness in Scientific Programs". Computing in Science & Engineering, May-June 2005, http://doi.ieeecomputersociety.org/10.1109/MCSE.2005.54.

Please, please, please, if you're building scientific software of any kind, find a copy and read it, because it's the best explanation I've ever found of why the right way to do it is, well, the right way to do it. Paul ties together a banker's dozen good ideas under the banner defense in depth: no matter how scrupulous we are, any of these might fail, so we have others in place to catch the mistakes that creep through. To quote the list that opens the paper, they are:

Need Something to Debug

Sat, 30 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Following up on the previous post about the way Paul Dubois organizes good practices as "defense in depth", we would really like to include a lecture on debugging that does more than present a few general rules like "make sure you know what the right output is supposed to be" or "divide and conquer". More specifically, we need an example that contains a handful of qualitatively different bugs that can be found in qualitatively different ways: one by a code analyzer, another using coverage ("Wait, why isn't the 'else' ever being executed?"), a third by unit testing, a fourth by differential comparison of pre-change and post-change output, a fifth using breakpoints and single-stepping, and so on. If you have such an example, or are willing to help us construct one, please let us know.

Would You Prefer...

Sat, 30 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Here are three instructional videos that I enjoyed watching:

What they have in common:

They're informative.
They're fun to watch.
It must have taken a ton of work to create them.

So my questions are:

Do you like them more than what we're doing?
Are they "better enough" than what we're doing to justify the extra effort? We'd produce half as much material if we did this—would that be a good tradeoff?

Provenance (Or, What We Didn't Quite Get to at the Met Office)

Fri, 29 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This post has been moved.

Configuration Files

Sun, 24 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This post has been moved.

Slides Available as PDF and PPT

Thu, 21 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The home pages of the following lectures now have links to PDF and PowerPoint versions of the slides used in each episode:

Python
Sets and Dictionaries
Testing
The Unix Shell
Regular Expressions
Version Control
Make
Program Design

We haven't provided slides for the spreadsheets and databases lectures because they aren't slide-based. Please let us know if you find these useful, or if you notice any typos that we should fix.

Final Four Episodes of Python Lecture

Mon, 18 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The last four episodes of our lecture on Python are now available, covering:

libraries,
tuples,
slicing, and
what is text, anyway?

The last isn't really about Python, but it's something people trip over pretty quickly when they start using it. Hope you find them useful...

Ratings Revised

Sun, 17 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We asked, you answered: here are the latest results from our survey of what topics you'd most like us to cover, with links to the ones that have been posted. A few notes:

N = 188 responses.
This is the first time data visualization has dropped out of the top 5.
It's also the first time that nerdish subjects like computational complexity and functional languages have placed anywhere near the top.
There's a noticeable mismatch between things we think people should know (like version control) and things people think they want (building desktop GUIs).

Conclusion: more data ≠ more insight.

Six More Python Episodes

Fri, 15 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've added six more Python screencasts to the site:

basics
control flow
I/O
aliasing
functions
first-class functions

We hope you like them—please let us know.

Dexy

Thu, 14 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

From Ana Nelson, a neat new reproducible research tool called Dexy that takes the pipe-and-filter model to new heights (or extremes, depending on your point of view :-) ). There's a quick intro here, and a video of her recent talk about Dexy here. I'm already thinking about ways of combining this with consistency-checking tools so that if the descriptions of results get out of step with the results themselves, alarms will go off. Check it out, and let us know what you think—is it useful and approachable enough to include in this course?

Three Python Screencasts Up

Thu, 14 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Three Python screencasts narrated by Dominique Vuvan, an alumnus of the Summer 2009 run of the course, are now online: introduction (episode 1), lists (4), and strings (6). We'll be recording and filling in the other episodes just as soon as construction in the office slows down...

Three More Episodes on Spreadsheets

Wed, 13 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have uploaded episodes 5, 6, and 7 of Jason Montojo's lecture on spreadsheets, which cover conditionals, pivot tables, and the basics of charting. Comments would be very welcome—and if you'd like to contribute material as well, please get in touch (we're particularly looking for help with data visualization).

Python Lecture Coming Online

Tue, 12 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Slides for the first few episodes of our lecture on Python are now online; recordings should follow tomorrow and Thursday, and slides for the rest of the lecture should be up by the weekend. I think the content is OK, but it feels like there's too much shoveling: we're throwing a lot of features at people without showing them how they'll make life better. I think there's inevitably some of that in an intro like this—people need to know a few things before any of them can be put to work—but I'd welcome suggestions on reorganizing the material.

Using Subversion from the Command Line

Tue, 05 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Ainsley Lawson, one of the TAs for the Fall 2010 online course, has recorded an 8-minute episode showing how to use Subversion from the command line. We hope you find it useful.

What Questions Do You (Frequently) Ask?

Sun, 03 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As part of redesigning this site, we're finally putting together an FAQ. It's pretty brief right now: what questions have you had about Software Carpentry that we could add (and answer)?

Ten Short Papers Every Computational Scientist Should Read

Thu, 30 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

No, we don't have a list—not yet—but we'd like to. What short, readable papers or articles do you think every scientist doing computational work should read at some point in their career (preferably early in their career)? Paul Dubois' 2005 article in Computing in Science and Engineering on maintaining correctness in scientific programs is a favorite of mine; so is Evan Robinson's summary of research on the effects of overwork on productivity. (Actually, I think everyone should read Robinson's article, not just computational scientists...) What else should be on the list? To qualify, entries must be short (up to a few pages long), well written, of broad general interest, and have something important to say that's relevant to our target audience. Suggestions in the comments, please...

I'm No Graphic Artist...

Tue, 21 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm not much of a graphic artist (in both my startups, I was told after a few weeks that I was never to work on the user interface again), so I'd appreciate your help designing a flyer/noticeboard poster for Software Carpentry. This PDF is what I have so far; anything more constructive than, "My eyes! My eyes! Aargh!" would be very welcome.

Your Favorite Running Examples?

Mon, 20 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've been fond of invasion percolation since I first encountered it: the problem is simple to state, but its implementation brings up quite a variety of useful ideas, from testing in the face of randomness, to associative data structures, to the limits of parallelism. It's a good size, too: I can develop a first version in front of students in about 30 minutes, then branch off in various directions for almost as long as they'll listen to me.

Five Episodes on Make

Wed, 15 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The first five episodes of the lecture on Make have been recorded, edited, and posted. I still need to do one on using Make to track data provenance, but as I discovered when I started to write it, there are a few other topics I need to fill in first. I will do an episode on managing C/C++ compilation with Make first (probably after I return from teaching in Colorado next week). As always, comments are welcome.

And For My Next Trick...

Thu, 09 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the things people have voted right up to the top of our poll is reproducible research. The phrase means different things to different people, but part of it must be accurately tracking the provenance of every bit of data a scientist touches: where it came from, what was done to it, what was done to the results, and so on. Once I've done the next episode of the Make lecture (on macros), I'd like to spend 10 minutes showing people how to do some of this by:

Slides for the First Four Episodes on Make

Wed, 08 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I don't like Make, but I'm resigned to teaching it: other tools have different, rather than lesser, flaws. I've posted slides for the first four episodes of the lecture about it, and hope to have scripts and screencasts up soon. I'd welcome feedback—in particular, if anyone knows a clean solution to the problem at the end of the fourth episode (getting summary data to depend on stats.py), please let me know.

Eight Episodes on the Unix Shell

Fri, 03 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I had planned to leave it out, but your votes count: the first eight screencasts of our lecture on the Unix shell are now online. Topics covered are:

Introduction (4:09)
Files and Directories (9:55)
Creating and Deleting (6:23)
Pipes and Filters (9:11)
Permissions (10:54)
Finding Things (9:22)
Job Control (5:37)
Variables (6:49)

I've already got a list of changes to make (thanks to feedback from IsaacG2—my apologies, but I didn't see your comments until after I'd recorded), but I hope this will be a good start. And for those who are interested, the eight episodes together took 36.5 hours to write, revise, record, and edit, for 62:20 of screen time. That's better than the 50:1 ratio I was clocking in June, but I need to squeeze it further if I'm going to finish everything on time.

Three More Sets of Slides

Wed, 01 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As promised in yesterday's post, the slides for three more episodes on the Unix shell are now online, covering how to find things, job control, and shell variables. Next up: public/private keys and SSH.

Five Episodes on the Shell (and Three to Come)

Tue, 31 Aug 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've just posted the slides for the first five episodes of the lecture on the Unix shell, covering:

Introduction (or, why would anyone do this to themselves, really?)
Files and directories
Creating and deleting things
Pipes and filters
Permissions

Minor things I have yet to cover:

Drive letters on Windows, and how Cygwin hacks around them
The 'echo' command
(Briefly) Windows access control lists
Wildcards in filenames ('*' has been covered)

Major things (the remaining three episodes):

Four More Screencasts on Testing

Fri, 27 Aug 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

After far too many delays, the next four screencasts on testing are finally available, covering systematic unit testing with Nose, why you should test interfaces rather than implementations, why testing floating point is hard, and how to create and manage fixtures systematically. I hope you enjoy them—have a good weekend.

Another Update on What You Want

Thu, 26 Aug 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Responses have slowed, so here are the final scores for the topics we're including (or thinking about including) in this course. It looks like the Unix shell just might go back in...

2.51	Automating Repetitive Tasks
2.50	Reproducible Research
2.50	Data Visualization
2.47	Version Control
2.44	Performance Optimization
2.41	Data Structures
2.39	Testing and Quality Assurance
2.39	Coding Style
2.38	Basic Programming
2.35	Using the Unix Shell
2.35	Parallel Programming
2.35	Debugging with a Debugger
2.29	Computational Complexity
2.22	Object-Oriented Programming
2.20	Working in Teams/on Large Projects
2.19	Designing a Data Model
2.15	Refactoring
2.10	Matrix Algebra
2.09	Static and Dynamic Code Analysis Tools
2.07	Systems Programming
2.04	Integrating with C and Fortran
2.03	Design Patterns
2.01	Packaging Code for Release
1.95	Functional Languages
1.94	Handling Binary Data
1.82	Image Processing
1.74	Build a Desktop User Interface
1.73	XML
1.65	Create a Web Service
1.39	Geographic Information Systems

What Don't You Understand That You'd Like To?

Mon, 23 Aug 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

What don't you understand about computers and computing that you'd like to? I'm not talking about big stuff, like "How does it all work, anyway?" or, "How do I write a game that's as addictive as BubbleSpinner?" What don't you understand about smaller-scale stuff, like handwriting recognition or how Excel knows which cells to update when you change a number or how your machine decides that it's time to update Java and what it does next. Please let me know, or if you're feeling shy, point your friends at this page and have them ask for you—we have to use something as motivating examples for this course, and if we can explain something interesting while teaching you fundamental computing concepts, so much the better.

Slides and Scripts for the Next Two Episodes

Thu, 19 Aug 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I have posted slides and transcripts for the next two lectures on testing, which cover unit testing with Nose, and testing interfaces instead of implementations. I was hoping to record the screencasts today, but setting up a new 64-bit desktop machine took longer than expected—hopefully they'll be up early next week, along with episodes on floating point and setting up fixtures. Comments, as always, would be greatly appreciated.

A Question About Nose

Tue, 03 Aug 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm putting together an episode of the testing lecture to introduce unit testing frameworks. In the past, I've used unittest, but colleagues have had good experiences with Nose, which doesn't require students to understand classes and methods in order to write tests. As an example, they're going to test a function that finds the overlap between fields in aerial photographs of Saskatchewan (where it's safe to assume that everything is a rectangle :-) ).

Two More Episodes on Version Control

Thu, 29 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The third and fourth episodes of our lecture on version control are now online. These explain how to handle conflicts from concurrent edits, and how to roll back changes. As with the second episode on basic workflow, they use a mix of slides, screen recording, and sound effects.

The next episode is supposed to explain how to create a repository, but I'm still trying to figure out what to show people. A repo on the same machine that's being used for development is better than nothing, but that doesn't help people share work with colleagues. On the other hand, creating a repo on a server somewhere requires at least basic knowledge of the shell: even if someone is willing to type in a password for each interaction (so that they don't need to know about public/private keypairs), they'll need to know enough to SSH in to the server and run "svnadmin create reponame". There are web-based control panels for creating and managing repositories, and we could just require them to ask their friendly neighborhood sys admin to set that up, but it's just enough of a stumbling block to, well, be a stumbling block. Suggestions would be welcome...

Second Lecture on Version Control

Mon, 26 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our second lecture on version control is now on the web. It combines screen recording with static slides; please let us know whether the format works for you; in particular, can you follow what's happening on the desktop?

Introduction to Version Control

Sat, 24 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It took a lot longer to put together than I expected, but I'm pleased with the result—the newest screencast explains what version control is, and why you'd want to use it, in four minutes and four seconds, including a wolf howl and a maniacal laugh. I hope you like it...

And in case you haven't been reading comments, I'd be very happy to include a parallel version control lecture based on a distributed version control system like Mercurial or Git if someone would like to create it. Software Carpentry is an open source project—if that's the itch you want to scratch, then please email me and we'll figure out how to make it happen.

Strictly Speaking, This Isn't Part of Testing

Fri, 23 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The second episode of the lecture on testing is now up. It covers exceptions, which strictly speaking aren't part of testing, but it seemed like a natural place to introduce them. As always, feedback is welcome...

First Episode of Testing Lecture

Thu, 22 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The first episode of the lecture on testing is now online. We're going to be showing people how to use Nose, so I've tried to motivate the idea of a unit testing framework—I'd appreciate feedback on whether it works or not.

And yes, I am working on the version control lecture, but I've been stumbling over minor technical glitches, and I'm afraid that if I try recording today, I'll start using uncivilized language. Hopefully I'll get to it tomorrow...

Popular, Fast, or Usable: Pick One

Wed, 21 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the core skills in any software developer's toolbox is automating repetitive tasks. We've been teaching this in Software Carpentry with Make for 13 years, and for 13 years, we've been looking for something better: something that has a less user-hostile syntax, requires less arcane knowledge of the shell, and at least pretends to do the same thing on different platforms. We briefly considered Ant (where "briefly" means "as long as it took to find the post by Ant's creator saying he would do it all differently next time"), but it's clear that tools like Rake and SCons, that embed build commands in a full-blown scripting language, are the future of build systems. Rake is based on Ruby, but SCons is based on Python (and actually has its roots in the original Software Carpentry project), so it should be a no-brainer, right?

Five... Five... Five Scripts in One!

Tue, 20 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've updated the script for the introductory episode on version control that I posted yesterday, and added four more covering basic operations, handling conflicts, rolling back changes (with a very brief mention of branching and merging), and setting up repositories for personal use (creating a shared repo requires more skills than students are likely to have, and should be left in the hands of sys admins). The five scripts are down below the break; comments before I start making up slides and recording would be very welcome.

Script for Introduction to Version Control

Mon, 19 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

While I'm waiting for artwork, I'd be grateful for feedback on the script for the introductory episode on version control.

Hello, and welcome to the first episode of the Software Carpentry lecture on version control. In this episode, we will explain what version control is, how it works, and why you should use it.
Suppose you and a friend are working together on a paper.
You both want to edit the file at the same time—what should you do?
You could take turns, but then each of you would spend have the time waiting for the other.
Another option would be to go ahead and work simultaneously, then patch things up afterward.
But somehow, stuff always winds up getting lost, overwitten, or duplicated.
The right solution is to use a version control system.
This keeps the master copy of the file in a central repository, usually located on a server—a computer that is never used directly by people, but only by the applications serving them.
No-one ever edits the master copy directly.
Instead, you and your friend each have a working copy on your own computers.
You work independently, making whatever changes you want to your local copies.
As soon you are ready to share your changes, you commit your changes to the repository.
Your friend can then update her working copy to get those changes.
And of course, if your friend finishes her part first, she can commit, and then you can update.
But what if you and your friend want to make changes to the same part of the paper?
Old-fashioned version control systems prevented this from happening by locking the master copy. Everyone's working copy would normally be read-only. When someone wanted to start work on a file, the version control system would make her copy of that file writeable. When she was finished work, the version control system would copy her changes to the repository, then mark her copy as read-only once again.
Only one person at a time could have a writeable copy. This guaranteed that two or more people could never accidentally make changes to the same file at the same time...
...but it also meant that only person at a time could work on any given file.
This is essentially the "one at a time" strategy from the start of this episode, but with the version control system acting as the referee to prevent accidents.
In practice, locking like this isn't as restrictive as it sounds. If you and your friend repeatedly find that you're trying to edit the same file, the solution is to break your paper (or your program) into several smaller files, so that you can work simultaneously.
However, most of today's version control systems use a different strategy, one based on the old saying that it's easier to get forgiveness than permission. In these systems, nothing is ever locked—everyone is always allowed to edit the files in their working copy.
Sometimes, of course, you and your friend will make changes to the same part of the paper.
If your friend updates first first, her changes go into the repository as usual.
If you try to commit something that would overwrite her changes, the version control system will stop you...
...and highlight the conflict by marking the overlapping regions in your working copy.
It's up to you to edit the file to resolve the conflict. You can keep your changes, accept your friend's, or write something new that combines the two—it's up to you.
Once you have fixed things, you can go ahead and commit.
Experience shows that version control is better than mailing files back and forth for at least three reasons. First, it's hard (but not impossible) to accidentally overlook or overwrite someone's changes—the version control system highlights them for you automatically.
Second, there are no arguments about whose copy is the most up to date—the master copy is.
These features mean that version control is worth using even when you're the only person working on a particular set of files.
Because it's a more reliable way to move files around between the computers you use than copying things onto a USB or sending email to yourself.
More importantly, whether you're working on your own or in a group, version control allows you to look at or undo changes what you did weeks, months, or years ago.
This works because the version control system never actually overwrites the master copy in the repository.
Instead, everyone time someone commits a new version, the system saves it on top of the previous master copy, along with some information about when the change was made and who made it.
This means that you can always see what the file looked like last week before someone rewrote the analysis section while you were on holiday.
It also means that you can always fetch old versions of things, like the exact version of the program you used to produce the graph on page 5 of that paper that someone is now challenging.
Version control systems do have one important shortcoming, though. If you are working with plain text files, it's easy for the version control system to find and display differences, and to help you merge them.
Unfortunately, today's version control systems won't do this for images, MP3s, PDFs, and Microsoft Word or Excel files. These aren't stored as text—they use specialized binary data formats, and there usually aren't tools for finding, displaying, or merging the differences between them. In most cases, all the version system can do is say, "These files are different." That's better than nothing, but not by much.
Even with this limitation, version control is probably the most important concept in this entire course. It's not just because it facilitates sharing; version control also allows you to look at or undo changes what you did weeks, months, or years ago.
We'll talk more about using version control to make your research reproducible in a later lecture. In the next episode of this one, we'll look at the most popular open source version control system in use today, called Subversion.

The four episodes after this introduction will cover:

A Gentle Introduction

Sun, 18 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via several routes, I've been pointed at MIT Open CourseWare's "A Gentle Introduction to Programming Using Python", as taught by Sarina Canelake in January 2010. An hour of lecture a day for 10 days (two weeks), plus two hours of lab per day, it is... "a gentle introduction to programming using Python for highly motivated students with little or no prior experience in programming computers". The pace is brisk:

Survey Results

Thu, 15 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Here are the results of the survey that we announced a couple of days ago. I'm a bit surprised that so many computer scientists responded, and equally surprised by the popularity of "biomedical engineering" — who knew? The scores for various topics hold a few surprises as well: I would have predicted that something with the word "web" in it would have scored near the top of the list, rather than at the bottom.

Two New Episodes on Dictionaries

Wed, 14 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have just posted the last two episodes of the lecture on sets and dictionaries, which introduce dictionaries and give examples of their use. Please let us know what you think about the pace and level of detail.

A Shorter Version of the Sets and Tuples Episode

Tue, 13 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the many advantages of online delivery of course material is that it allows us to present content in several ways, tailored to different audiences. For example, some people want to understand how sets are stored in hashtables; others just want to know that you cannot put a list in a set, and should use a tuple instead. We now have episodes catering to both: the longer and more detailed one posted last week, and a shorter and less detailed one posted today. We'd welcome feedback: is this confusing, helpful, or (as is so often the case) a bit of both?

Traffic

Tue, 13 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

In case you're interested, we seem to average slightly around 280 views per blog post right now. I'll post stats on page views (i.e., how many people are looking at the lectures themselves) at the end of this month.

Which Topics Are Most Important to You?

Sat, 10 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have created a survey to find out which of the topics we're planning to cover are most important to you. Please take three and a half minutes to fill it in; we'd be grateful as well if you could re-post the link, forward it to relevant mailing lists, etc.

http://www.surveymonkey.com/s/FM9YV9C

Thanks for your help!

Two Episodes on Sets

Thu, 08 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've just posted an introduction to sets in Python, and a brief look at how they're implemented (which you need to know in order to understand why you can't put a list in a set). The second of these has more theory in it than most of our episodes; I'd be grateful for feedback from non-computer scientists about how much sense it makes.

That's, Uh, Pretty Ambitious

Tue, 06 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The title of this post is taken from the reaction of the first person I showed this course schedule to. If we're really going to teach Software Carpentry in one university term (12-13 weeks), this is the pace we'll have to move at. What do you think? Is it crazy, or just nuts? And what, if anything, can be done about it?

The Violas of Programming

Tue, 29 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Orchestral musicians make jokes about violas and viola players. "What's a string quartet? A great violin player, a mediocre violin player, a bad violin player, and a cellist." Or, "What's the difference between a viola and a trampoline? You take off your shoes to jump on a trampoline." But violas are as essential as they are unglamorous: hardly anyone plays them as a lead or solo instrument, but string quartets just don't sound right without that third voice.

Last Episode on Sets and Dictionaries Posted First

Mon, 28 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

For our lecture on sets & dictionaries, I decided to start at the end with the wrap-up example on nanotechnology. I'll double back tomorrow and start filling in the pieces that this example relies on.

Four Down-What Next?

Sun, 27 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The regular expressions lecture has been promoted from the development area to the main site, where it joins the lectures on program design, databases, and spreadsheets. We have a lot of other topics still to cover: please let us know which you'd like to see first.

Final Episode of Regular Expressions Lecture Now Online

Fri, 25 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The fifth and final episode of our lecture on regular expressions is now online. In this episode, we work through a new problem—extracting bibliographic citations from papers—and introduce a few new tools along the way. Please let us know what you think.

Another Example of small-p Patterns

Thu, 24 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A couple of weeks ago, I asked whether people would find an exploration of ways to count things useful. The consensus was "yes", so I've started drawing up notes. While working on them, it occurred to me that "ways to persist things" might be just as interesting. Some of the approaches that could be discussed are:

Save a list of numbers (one per line).
Save a matrix using a header line to record dimensions (introducing the idea of metadata) and M values per line.
Save a matrix without metadata (which requires the length of the second and subsequent lines to be checked).
Save mixed atomic values (which gets hard when strings are allowed).
Saving records whose structure is known in advance (the hardest of the bunch, since this is the point where aliasing appears).
Saving in binary rather than ASCII.
Creating self-describing data (save the format at the top, then the data).

What other persistence patterns do you think would be worth explaining, and why?

Software Carpentry in Three and a Half Minutes

Wed, 23 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I had a chance to give a quick pitch for Software Carpentry yesterday, which I've since recorded and posted to our web site. I hope it's a useful overview of what we're trying to do and why; I'd be very grateful for ideas about improving it.

Note: a newer version of the pitch, titled "Software Carpentry in Ninety Seconds", is now available.

Episode 4 on Regular Expressions

Tue, 22 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The fourth episode of our lecture on regular expressions is now online. This one explores shortcuts and escape sequences; we'd be grateful for feedback on whether it works a single episode, or ought to be split in two.

A Little Bit of Theory

Mon, 21 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The third episode of our lecture on regular expressions is now on the web. This one describes how regular expressions are implemented using finite state machines: it's more theoretical (with a very small 't') than previous episodes, and we'd welcome feedback.

Second Lecture on Regular Expressions

Sat, 19 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The second lecture on regular expressions has been posted to the development area. This one covers operators like '*' and '+', and introduces character sets. Please let us know what you think.

First Half of Spreadsheets Lecture Now Online

Fri, 18 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The first four episodes of Jason Montojo's lecture on spreadsheets are now online: topics covered include basic calculations, aggregation, controlling the display, sorting, and debugging. Please have a look and let us know what you think.

Our First Few Exercises

Fri, 18 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Jon Pipitone has posted a few self-assessment exercises for the lecture episode on selecting data from a database. Please have a look and let us know what you think: would having little self-tests like this at the end of each episode help solidify your understanding of the material?

Let's Try That Again

Thu, 17 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks to everyone for their feedback on our first attempt at a lecture on regular expressions. Our second attempt [no longer online] is now up for your viewing pleasure: it's about ten minutes long, and uses data scrubbing to motivate the use of REs. As always, we'd be grateful for suggestions.

A Voice from the Back of the Room

Wed, 16 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Mark Guzdial recently posted another thought-provoking piece, this one about how teachers are biased toward assessing a class's progress by their interactions with its brightest students, and how important it is to assess understanding systematically to find out what's actually going on. We're still wrestling with this in Software Carpentry: what can we put on the web to help students determine whether they have understood a particular topic and are ready to move on?

Is Live Coding Worth It?

Wed, 16 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I put together an introductory lecture on regular expressions yesterday, but I'm not happy with the results, and I'd like your feedback to help me make it better.

One of the reasons we're moving from static HTML to videos is to show people how to program, rather than just the results of programming. As part of that, I want (or wanted—I may be changing my mind) to do all the programming examples live instead of showing snippets of code and snippets of output slide-style. Jon has already done that with the database lectures, but his "programs" were single-line SQL statements; this episode is the first time we've had longer examples. It was a pain to do: each time I made a typo inside a function definition, for example, I'd wind up with a dozen or more junk lines on screen (since the error wouldn't show up until the first time I called the function). I don't want to waste viewers' time on trivial errors, but there's no way in IDLE (or other IDEs) to erase the last few lines of interaction with the interpreter. I "solved" the problem by hitting return until the offending text had scrolled off the top of the screen, but it leaves a couple of ugly breaks in the video. If IDLE had a "clear screen" button, I might use that periodically, but (a) it doesn't, and (b) one of the reasons to code live is to keep recent context in view, which "clear screen" would erase.

Glossary and License Online

Tue, 15 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A glossary of terms used in this course is now online; corrections and suggestions for both additions and deletions would be very welcome. The licenses used for course content and code have also been posted—both are as open as we can make them.

Counting Things

Fri, 11 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I re-read Robins et al's "Learning and Teaching Programming: A Review and Discussion" [1] and Eccles and Stacey's "Understanding the Parallel Programming" [2] this morning. One of the key takeaways in both is that novice programmers tend to classify problems according to problem domain or language features, while experienced programmers tend to classify them according to solution strategies. Where a beginner sees a problem in molecular biology or statistics, for example, an expert sees an opportunity to apply divide-and-conquer or accumulate-and-adjust.

Our Lecture on Databases is Now Online

Thu, 10 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our introduction to the basics of SQL is now online: the episodes in this series include:

Introduction
Selecting Data
Filtering
Sorting
Aggregation
Joining Tables
Missing Data
Nested Queries

Please let us know what you think. And please also let us know what topic from the course outline you'd like us to tackle next.

The Big Picture (version 3)

Thu, 10 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Version 3 of the concept map describing what we mean by "computational thinking" incorporates feedback from several people—thanks for sending it. Please let us know what you think...

Reorganizing Content

Wed, 09 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

In anticipation of posting more lecture content, we have reorganized the pages on this site. If you look at the side menu, you'll see that "About/" and "Program Design/" now have sub-pages; we'll organize each lecture the same way to make it easier to follow along.

And if you're having trouble posting or viewing comments, please send us email— we think we've sorted it out, but hey, it's software...

Episode 11: Making It Fast

Tue, 08 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Today's episode is the last in the series on program design using invasion percolation as an example. In it, we look at how to make the program we've developed so far run zillions of times faster. It's the longest episode so far, and probably the most difficult, but it makes two very important points:

The biggest performance gains come from improvements to algorithms and data structures, not from tweaking loops and rearranging conditionals.
If you want to write a fast program, you should start by writing a simple one, test it, and then rewrite it piece by piece, re-testing as you go along.

Here's the complete list of episodes:

Testing Invasion Percolation

Mon, 07 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Finally, after all that refactoring, we get to test our invasion percolation program—although in fact, we spend more time discussing how to structure testing code than we do actually testing.

Introduction
The Grid
Aliasing
Randomness
Finding Neighbors
Resolving Ties
Assembling the Program
Bugs
Refactoring
Testing

The Big Picture (version 2)

Mon, 07 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks to everyone who gave us feedback on the first version of a concept map for Software Carpentry. The second version is shown below, along with a list of terms that ought to be on it, but aren't. Where would you place them, and how would they connect to each other and to the concepts that are already linked?

Concept Map

Fri, 04 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

While re-designing the Software Carpentry course, I have realized that we rely over and over again on some underyling concepts that are hard to capture as lecture topics. I think these concepts are the heart of any useful definition of "computational thinking". The diagram below is a first attempt to capture what these concepts are and how they're related. (The list below the diagram summarizes the relationships in textual form for easier reading.) Suggestions for improvements would be very welcome...

Refactoring Invasion Percolation

Fri, 04 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Today's episode (the ninth in the lecture about designing an invasion percolation program) was supposed to be about testing, but wound up being about refactoring instead—which probably won't surprise anyone who's done any significant amount of programming. We'd be grateful for feedback on how easy or hard it is to follow along as the program is rearranged.

Introduction
The Grid
Aliasing
Randomness
Finding Neighbors
Resolving Ties
Assembling the Program
Bugs
Refactoring

Assembling a Program

Thu, 03 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The seventh episode of our lecture on program design is now online. In this one, we actually assemble a complete version of the program step by step using the pieces designed earlier. The final program still has one nasty bug, though, and I'll award a Software Carpentry mug to the first person who can find it (source code attached to this post). For reference, the first six episodes are listed below, along with another link to this one.

If You Want to Look Ahead...

Thu, 03 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our previous post said that there was a bug in the first complete version of our invasion percolation program, and offered a Software Carpentry mug to the first person to find it. If you're the sort who prefers to look up the answers in the back, we've just posted the solution as episode 8 in the series on program design. For reference, here's the lecture so far:

Program Design: the Second Instalment

Tue, 01 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Having recorded and posted four episodes of a lecture on program design yesterday, I've managed another two today (highlighted below):

Introduction
The Grid
Aliasing
Randomness
Finding Neighbors
Resolving Ties

The remaining episodes are much longer, and will show how to assemble the program from the pieces seen so far, how to refactor the result to make it more testable, how to test it, how to speed it up, and how to build a GUI for it. Slides for the first two of these episodes are done; I hope to have everything up by the end of this week or early next. As always, feedback is greatly appreciated.

Program Design: the First Third

Mon, 31 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I spent last week making and re-making slides for a lecture on program design, and managed to record the first third of them today—the rest should follow tomorrow and Wednesday. We've taken on board your comments about pace and layout, and would be grateful for more feedback—please go ahead and post comments on this post to tell us what you think.

Introduction
The Grid
Aliasing
Randomness

Teaching databases by example

Fri, 28 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Over the last two weeks I've been spending most of my Software Carpentry time working on the database unit. They began as a fairly straight-forward translation of the Software Carpentry 3.0 lecture notes with only a few changes to the sequencing of the topics. The plan for the unit is fairly simple: each screencast introducing only one or two new topics, and builds on the previous screencasts with as few as possible forward references to later topics. The topics themselves are really just the different language features (SELECT, WHERE, JOIN, etc..) presented as tasks you might want to perform, and illustrated by working with toy data.

Day 11: Slides

Mon, 17 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Today's screencast experiment [link no longer active] is narration over PowerPoint slides: it isn't animation per se, but we'd like your feedback on whether something like this is a good way to explain "big picture" concepts.

Day 10: Closed Captioning

Fri, 14 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our latest screencast, on NULL values in SQL, is now online. [Original link no longer active; more recent screencast on NULL values.] Unlike its predecessors, this one has closed captions (as well as a transcript in the enclosing page). Please let us know what you think: are the captions helpful, or do you find them distracting?

Day 8: Exercises (with a screencast)

Thu, 13 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Today I experimented with creating a few simple exercises for the material covered in the first screencast on databases. The very last exercise includes a screencast that walks through the solution, and a potential pitfall along the way. My apologies for the crappy audio and choppiness in a few places — I was using my laptop microphone, and I tried to edit out a few clicks and pops.

Day 9: Programming

Thu, 13 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our first screencast showing ~~a bit of Python programming~~ is now up for comments. [Link no longer active, more recent link to screencast on dictionaries.] It's deliberately low-tech: we used Emacs as an editor, and simply showed the program's textual output instead of stepping through it with a debugger. (Our next programming lectures will be IDE-based for comparison.) Please tell us what you think about the pace, watchability, etc.

Day 7: Mini-screencasts

Wed, 12 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Yesterday Jason and Greg experimented with breaking apart a 10 minute screencast on database grouping and aggregation into even more bite-sized pieces. We have two flavours today: ~~point form notes with links to the screencasts~~, or ~~point form notes with screencasts embedded~~. [Links no longer active, latest screencast on database aggregation as of January 2012.]

Feedback on this format would be really helpful. Are these chunks too small? Do you prefer embedding over linking, or vice versa? Also, we'd welcome any comments on the material itself, of course.

Day 6: Screencast With Point-Form Notes

Mon, 10 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Jason Montojo and I put together an introductory screencast about spreadsheets today. Unlike our previous screencasts, this one is accompanied by point-form slide-style notes instead of prose paragraphs. We'd be grateful for your feedback: are notes of this kind useful? Are they comprehensible without watching the screencast itself? If the notes were available in slide format (Keynote, PowerPoint, Impress, or plain old PDF) would they be enough for you to lecture with?

Day 5: A Different Kind of Screencast

Fri, 07 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I spent most of today creating another screencast, this one explaining what "and", "or", and "not" mean to programmers and how they work. There are no live examples this time: instead, I illustrated the talk with some simple images. Setting aside the question of whether Venn diagrams are the best way to explain Boolean logic, do you prefer this to live desktop of the previous screencast? Do the jumps back and forth between sets and tables make sense, or are they confusing? Do the colors add value, or should we stick to black & white? And how about the pace—is it too fast, too slow, too repetitive, et cetera?

Day 4: First Preliminary Alpha Test Etc. Screencast

Thu, 06 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Jon Pipitone and I have created a screencast showing the basics (and I really mean "the basics") of pulling data out of a database using SQL. It took about 4.5 hours in total to produce 7 minutes of video:

Writing script:	1.5 hours
Doing 3 takes:	1.0 hours
Editing:	1.0 hours
Transcribing:	0.5 hours
Screenshots:	0.5 hours

This was the first time either of us had used Camtasia; I expect that we'll be able to cut the recording, editing, and screenshotting time in half with practice. Transcribing time is dictated by typing and editing speed, so it's unlikely to come down, and scripting time will probably be the same or greater for future topics (since some of them will require diagrams and other time-consuming prep). We're therefore looking at 3 hours of production for 5-7 minutes of screen time; if you figure a lecture is 50 minutes, and there are 25 of them in the course, that works out to... um... carry the six... 750 hours to get all of the existing material online. Pad by half to account for real-world effects, and that's 35-40 weeks — tight, but doable.

Day 1: Shuffling Sticky Notes Around

Tue, 04 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Day 1 of Software Carpentry Version 4: Jason, Jon, and I spent the afternoon shuffling sticky notes around on a whiteboard, trying to map out key concepts for our databases lecture(s) and their dependencies. The result isn't all that different from what's in Version 3, but there was one important insight. J&J commented that many of the students in the winter run of the course had understood the material, but been unable to apply it to their own problems. Our tentative explanation is that in order to apply what they have learned about SQL to their own data, students have to (a) pre-process that data (e.g., read a CSV file, then parse the cells in certain columns to extract timing values embedded in other data) and (b) figure out how to store those values in a database (i.e., do some data modeling and schema design). Our lesson plan for Version 4 already has a slot for a lecture on modeling: maybe this should be given more emphasis?

Day 2: More Sticky Notes

Tue, 04 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Day 2. I feel I should say something like, "We traversed the lower ridge, and are camped in a small valley overlooked by the main peak—I can hear wallabies hooting in the distance," but (a) it hasn't been quite that dramatic and (b) I don't think wallabies hoot. In this reality, Jason, Jon, and I spent another afternoon mapping content: in particular, disentangling the mouse-clicking and data-analyzing aspects of the spreadsheet lecture. Tomorrow we hope to do our first pair of five-minute trial screencasts, just to see how well (or how poorly) our chosen tools work.

File Sharing for Scientists

Fri, 16 Apr 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A scientist I recently met in Toronto had a problem: how to share large files with colleagues. Each file is a couple of hundred megabytes; dozens are produced each week, but each is only interesting for a couple of months; and there are confidentiality issues, so some kind of password protection is needed. Conventional file-sharing services like Dropbox aren't designed for data that size, so in the end she bought a domain and set up secure FTP.

What's Not on the Reading List

Mon, 29 Mar 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I mentioned yesterday that I maintain a list of books that haven't been written yet. Partly it's an exercise in sympathetic magic—if the reviews exist, maybe the books will follow—but it's also useful for organizing my thoughts about what a programmer's education should look like. Looking at the books I've matched to various topics in the Software Carpentry course outline, there are some distressing gaps:

Given that programmers spend upwards of 40% of their time debugging, there are very few books about it, and only one collection of exercises (Barr's Find the Bug).
There's a lot on higher-level programming techniques, but it's scattered across dozens of books as disparate as The Seasoned Schemer, Effective C++, and The Practice of Programming. I haven't read Perrotta's Metaprogramming Ruby yet, but it looks like it will be another rich source of ideas.
Material on systems programming—manipulating files and directories, running sub-processes, etc.—is equally scattered. The Art of Unix Programming includes all the right topics, but covers too much, in too much detail, at too low a level. Gift & Jones' Python for Unix and Linux System Administration has the same two faults (from Software Carpentry's point of view—I think both are excellent books in general), but uses a scripting language for examples, so it made the list.
Mark Guzdial and others have done excellent research showing the benefits of teaching programming using multimedia, i.e., showing students how to manipulate images, sound, and video as a way of explaining loops and conditionals. That's half of why the revised course outline includes image processing early on (the other halves being "it's fun" and "it's useful"). Once again, most of what I'm familiar with is either documentation for specific libraries, or textbooks on the theory of computer vision, but there are some promising titles in the MATLAB world that I need to explore further.
Performance. It's been 15 years since I first grumbled about this, and the situation hasn't improved. Most books on computer systems performance are really textbooks on queueing theory; of that family, Jain's Art of Computer Systems Performance Analysis is still head and shoulders above the crowd. Souders' High Performance Web Sites is the closest modern equivalent I've found to Bentley's classic Writing Efficient Programs, but neither is really appropriate for scientists, who need to think about disk I/O (biologists and their databases), pipelining and caching (climatologists with their differential equations), and garbage collection (everybody using a VM-based language). I had hoped that High Performance Python would fill this gap, but it seems to have been delayed indefinitely. (And yes, I've looked at Writing Efficient Ruby Code; it has some of what our students want, but not nearly enough.)
There are lots of books about data modeling, but all the ones I know focus exclusively on either the relational approach or object-oriented design, with a smattering that talk about XML, RDF, and so on. I haven't yet found something that compares and contrasts the three approaches; pointers would be welcome.
Web programming. There are (literally) thousands of books on the subject, but that's the problem: almost all treatments are book-length, and this course only has room for one or two lectures. It is possible to build a simple web service in that time, but only by (a) using a cookbook approach, rather than teaching students how things actually work, and (b) ignoring security issues completely. I'm not comfortable with the first, and flat-out refuse to do the second: if this course shows people how to write a simple CGI script that's vulnerable to SQL injection and cross-site scripting, then it's our fault when the students' machines are hacked. This gap is as much in the available libraries as in the books, but that doesn't make it any less pressing.

Given these gaps, I may drop one or two topics (such as performance and web programming) and either swap in one of the discarded topics or spend more time on some of the core material. I'm hoping neither will be necessary; as I said above, pointers to books in any language that are at the right level, and cover the right areas, would be very welcome.

How Much Of This Should Scientists Understand?

Thu, 11 Mar 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Let's start with the problem description:

All of the Software Carpentry course material (including lecture notes, code samples, data files, and images) is stored in a Subversion repository. That's currently hosted at the University of Toronto, but I'd like to move it to the software-carpentry.org domain (along with this blog). However, software-carpentry.org is hosted with site5.com, who only provide one shell account per domain for cheap accounts like the one I bought.

Dudley and Butte on Software Skills

Sun, 27 Dec 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via Titus Brown, a new PLoS paper titled "A Quick Guide for Developing Effective Bioinformatics Programming Skills" by Joel Dudley and Atul Butte. Their recommendations are:

Programming languages
Embracing open source
Unix command-line skills
Keeping projects documented and manageable
Preserving source code with version control
Embracing parallel computing paradigms
Structuring data for speed and scalability
Understanding the capabilities of hardware
Embracing standards and interoperability
Put a high value on your time

I think all these things matter, but:

Cloud Computing for Beginners

Sun, 15 Nov 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Ana Nelson has posted step-by-step instructions showing how to use Amazon's EC2 cloud computing platform to run simulations. There are still a lot of fiddly details, but the barriers to entry are getting lower all the time...

Packaging

Fri, 13 Nov 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Martijn Faassen has posted a nice history of packaging and distribution tools for Python. Yes, it's a topic only a geek could love, but anyone who wants to distribute software to other people needs to grapple with these issues. The question is, should these tools, the problems that motivate them, and the technology underlying them be part of this course? Or should something equivalent (and if so, what)?

By Popular Request...

Fri, 30 Oct 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

...I have added a lecture on high performance computing to the revised outline for the course. Several people suggested it, and what's the point of asking for feedback if I don't listen?

Cryptography Isn't Security

Fri, 23 Oct 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One topic that I've tried to include in this course a couple of times, without success, is security. I feel irresponsible not saying something about how to share safely, but I've never found something that (a) would fit into one hour, (b) wasn't platitudes, and (c) gave listeners something they could act on.

One reviewer suggested talking about public/private key pairs (to help people set up SSH), signing things digitally, and the like. I'm leery of going down that road, though, since it could easily leave people with a misplaced faith in technical solutions to security problems. As always, suggestions would be welcome...

Should Modeling Be Part of This Course?

Wed, 21 Oct 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Jon Pipitone has a good description on his blog of work the grad students in our department are doing to translate work in climate change into software engineering terms. Their first step is to represent the ideas in MacKay's excellent Sustainable Energy Without the Hot Air in two of the graphical notations that computer scientists use for system design. I was initially very skeptical, but looking at their work so far, I'm quite impressed. My question is, would it be useful for scientists to know how to do this themselves? More specifically, is the lecture on data modeling that I've planned to include in Version 4.0 worthwhile or not?

Revised Plan

Fri, 16 Oct 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've posted updates to the revised course outline. In particular, I have:

Moved testing earlier.
Clarified intent in a couple of places.
Made an list of things we're leaving out.

As always, feedback would be welcome. I'd also be grateful for pointers to places that might fund this work: as I've found in the past, many people think the course is a good idea, but it doesn't quite fit into their funding mandates :-(

Videos from Symposium Are Now Online

Thu, 08 Oct 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I have put video recordings of the guest talks given at our July 29 symposium on Science 2.0 — please click the titles of the talks on the index page. Thanks again to all of our speakers, to the MaRS Centre technicians for the raw recordings, and to Tanya Murdoch for editing.

Comments on Course Reorganization

Tue, 06 Oct 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm grateful to Lorin Hochstein for sending detailed feedback on my proposal to reorganize the course. His comments are below, with my replies and his counter-replies interspersed; more comments would be very welcome.

Content I think you could drop if you wanted to save time:

Read Data Directly From Hardware. I suspect that this would be relevant to only a small minority of your audience. Especially if you're teaching the course mostly in Python, because this is the sort of thing you should really do in C.

Updated Outline for Revised Course

Fri, 18 Sep 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I have updated the description [no longer online] of how I plan/hope to reorganize the course. My thanks to everyone who commented on the earlier draft; I'd be very grateful for feedback on this one as well—I realize that some of the lectures are still hopelessly ambitious, but I hope it's at least a target to shoot at.

Partial Outline of New Version of Course

Tue, 15 Sep 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I have put a mostly-complete draft of my ideas for a rewrite of the course online. This version would start with specific problems, then backfill the tools and skills needed to solve them, but there are some big outstanding issues:

There still isn't anything on the computational thinking skills that Jon Udell identified. I don't know how to tackle these without relying more than I want to on closed-source offerings like Yahoo! Pipes that could disappear at a moment's notice.
I just can't see how to fit some topics—particularly object-oriented programming and web programming—into the available space (a maximum of two hours of lecture and four hours of lab for each). What are the minimum useful problem/lecture for each?

There are a few others as well (just search for 'TODO'), but those are the big ones. Any help you can give would be very welcome.

R for Programmers?

Sat, 05 Sep 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

What's the best introduction to the statistical language R for experienced programmers who are more interested in its support for objects, systems programming, and the like than they are in statistics?

Is The Future Waving At You?

Sun, 30 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Cameron Neylon has been playing with Google Wave, and he likes it. His presentation at Science Online in London in August explains why (you can also watch video, though sadly there's no soundtrack). He's even writing robots to automate some scientifically interesting tasks. Nature News liked Wave too, which (a) reminds me yet again of how prescient Jon Udell's "Internet Groupware for Scientific Collaboration" was ten years ago, and (b) makes me wonder (also again) how much this course should be re-thought.

How Important is Geospatial Data to You?

Wed, 26 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Software Carpentry currently teaches students how to manipulate text (using regular expressions), XML (using DOM), relational data (with SQL), and binary data. A decade ago, when we first put the course together, that covered everything I'd ever seen more than one or two scientists use. Today, though, an increasing number are using geospatial (map) data as well. How important is this to your work? If the answer is "very", what data do you work with, what do you do with it, and what would you like to be able to do?

The Delight Is In The Details, Too

Sun, 23 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

They say the devil is in the details, but so's the delight, because it's the details that determine whether something works or doesn't. So let's take a look at how to translate the last post's "big picture" into actual course content.

Every competent developer uses some kind of tool to automate tasks that involve dependencies. The best known is still Make, which compares the last-modified-times on files to see which ones are stale, and runs shell commands to bring them up to date. Ant, Rake, and whatever's built into your IDE all work basically the same way, and can all be used (with greater or less ease) to recompile software, re-run tests, prepare distributions, update web sites, and what have you.

The Big Picture

Fri, 21 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the lessons we learned at Los Alamos National Laboratory in the 1990s and early 2000s is that most scientists don't actually want to learn how to program—they want solve scientific problems.To many,programming is a tax they have to pay in order to do their research.To the rest,it's something they really would find interesting,but they have a grant deadline coming up and a paper to finish.

Guest Speakers' Slides Now Available

Mon, 03 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our guest speakers' slides are now on the web; video will follow soon.

Next Steps

Sun, 02 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It's clear from Friday's end-of-course review that the course needs shaking up. Before that starts, though, there's a higher-level question to answer: should the course notes be converted to a wiki to encourage contributions from others? It was always my hope that other people would contribute material, but in four years, only five ever have; perhaps wikification would change that.

Right now, the notes are stored as HTML pages in a Subversion repository and compiled by a little Python script to resolve cross-references, insert code samples, and so on. The advantages of this approach are:

Post-Mortem

Sat, 01 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Today was the last day of the course, so we spent the morning talking about what had gone well and what had not. The high and low points were:

Good

The course was fun.
The TAs were fantastic.
The format (one hour of lecture plus two hours of lab, twice a day) worked well.
Enjoyed the parts where the instructors programmed live.
Liked the emphasis on working practices that complement coding.
Liked the spread of topics, and the variability of things that are useful in all the different fields.
Liked the pair programming.
Welcomed exposure to standard libraries that weren't necessarily covered in the course.
Liked the pre-arrival questions about what people knew, were doing, and wanted from the course.
The examples were good.
So were the donuts.

Bad

Three weeks is too long.
Some of the later topics were not as useful.
Would have preferred to use standard libraries for the image processing lecture and exercises instead of simplified libraries.
Too little coverage of too many subjects.
The formatting of the slides leaves much to be desired.
Too many lectures ran over time (which was particularly hard in afternoon sessions).
Divided attention in FriendFeed is a problem.
The less applied stuff (e.g., computational complexity) wasn't as useful or as interesting.
Students weren't given enough time to work on their own projects.
Didn't feel encouraged to make suggestions or provide feedback.
Not enough on shell programming.
Too much shell programming.
A/V between Toronto and Edmonton was crude by modern standards.

Changes

More on object-oriented programming.
More feedback on the students' solutions to the exercises—they didn't get the equivalent of grading.
Put the exercises up before the class, so that students know what the lecture's going to be leading them to.

It's been a good three weeks—I enjoyed getting to know the students, and look forward to seeing what they do with what they've learned.

A Good Afternoon

Fri, 31 Jul 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Yesterday afternoon, the students and ninety other guests were treated to six engaging talks about Science 2.0 from Titus Brown, Cameron Neylon, Victoria Stodden, David Rich, Michael Nielsen, and Jon Udell. We'll post slides and video here as soon as we get them; until then, you can catch up on what happened in the FriendFeed room or by reading Steve Easterbrook's real-time blog of the event.

Our thanks once again to everyone who made the day possible:

Where This Course Came From

Mon, 27 Jul 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I have created an Amazon.com Listmania list [no longer online] for the books that most directly influenced this course. Some topics aren't directly represented—there's nothing devoted to handling XML or binary data, for example, or on big-Oh complexity or web programming—but I hope what's there will be of use.

Quality Control and Traceability

Mon, 29 Jun 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Nature News recently reported work by Gurt Vriend at other to clean up records in the widely-used Protein Data Bank (PDB). This is great news, but what's missing is a way to track forward from entries to see what already-published papers have relied on data that's now known to be incorrect. One of the many goals of this course is to give scientists the understanding they need to tackle this problem.

Ready for Proofreading

Mon, 29 Jun 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We're ready for feedback — if you check out the new slides, you'll see a little feedback bubble at the bottom of each topic. Clicking that will give you a chance to send us email to tell us about formatting glitches, factual errors, or anything else you'd like fixed. Please let us know what you think...

Updating the License

Thu, 25 Jun 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

In response to several requests, we have updated the license on the course material: the course content is now covered by the Creative Commons Attribution license, while the example code is (still) covered by an open source MIT license. In plain English, this means that you can re-use course content however you want, as long as you give us credit.

Another New Version of the Slides

Tue, 23 Jun 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A new(er) version of the slides has been posted at http://software-carpentry.org that includes styling changes courtesy of Ryan Feeley. There are still many minor formatting glitches; we'll fix them in the coming week, and post a schedule showing which lectures are going to be given when.

We've also updated the Guest Speakers page with bios and abstracts for the people who'll be talking at the MaRS Centre in Toronto on July 29. Talks will run 1-6 pm, and will be followed by a wine and cheese. The event is free, but will require advance registration—we'll post details here as soon as we have them.

And Speaking of Sightings...

Mon, 15 Jun 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A new (but very rough) version of the slides for the course is now up at http://software-carpentry.org. This uses Eric Meyer's S5 package for formatting and pagination (along with a short Python script to insert cross-references). We were planning to use LaTeX, but after messing around with packages, dependencies, and the like, HTML started to look at lot simpler. We still have a lot of reformatting to do (particularly with tables and code inclusions); if you'd like to help, or if you are a CSS expert and can help make the slides look good with Internet Explorer (right now they're styled for Firefox), please let us know.

SECSE Workshop

Mon, 01 Jun 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The latest in a series of workshops on "Software Engineering for Computational Science and Engineering" was held in Vancouver on May 23, just after ICSE'09. Steve Easterbrook has written a good summary of what was discussed, and Jeffrey Carver's longer summary will appear in a future issue of Computing in Science and Engineering.

Error Handling

Tue, 12 May 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One topic that isn't currently in the curriculum that I'd really like to add is detecting, handling, reporting, and recovering from errors. This makes up is 10-30% of the code in real applications, but dealing with errors is almost omitted from textbook examples and tutorials for the sake of clarity (Tanenbaum's Minix book being a laudable exception). I have asked elsewhere for someone to write an entire book on the subject; if anyone wants to take a crack at an hour-long lecture, please get in touch.

Links for Summer Interns

Mon, 11 May 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our summer interns started today; our first job is to define exactly what they'll be working on this summer, so it seems like a good time to round up a few links on interesting topics. My apologies for those hidden behind paywalls...

Steve's Project Ideas

Social network analysis for scientists
Electronic lab notebooks

Reproducible Research

If I said, "I just got a really interesting result in the lab, but I didn't record the steps I took or the settings on the machine," no reputable journal would publish my paper. If I said, "I just got a really interesting computational result," most reviewers and editors wouldn't even ask if I'd archived my code and the parameters I used, or whether that code would run on someone else's machine. Reproducible research (RR) is the idea of making computational science as trustworthy as experimental science by creating tools and working practices that will allow scientists to re-create past results.

How Scientists Use Computers: Survey Part 2

Sat, 09 May 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thank you once again for taking part in our Fall 2008 survey of how scientists use computers in their research. We will present a paper describing our findings at ICSE'09 in Vancouver on May 23, and will make the results public as soon after that as possible. There will also be an article in American Scientist magazine discussing what you've told us some time this summer.

Topics and Schedule Posted

Wed, 06 May 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've put up a list of the topics we intend to cover, and the order in which we intend to cover them. It's very provisional; we'll update it regularly, and your comments would be very welcome.

Note: the topics page has been retired. Please see the course outline instead.

Empirical Software Engineering and Scientific Computing

Tue, 28 Apr 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The slides for my talk at the National Research Council on empirical software engineering and how scientists actually use computers are now up on SlideShare. The colors in some of the embedded images were messed up during upload, but the result should still be readable.

Firming Up Course Goals

Mon, 27 Apr 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The best way to design a course is to describe the things students will be able to do when it's over; the best way to do that is to specify graduation exercises. Ours are listed in the Goals page on this blog. We would be grateful for feedback: are these the things you want to be able to do? What did we forget? What could we take out to make room for things you care about more? Please leave your comments on the page itself; we'll update it regularly based on what you say.

What Supervisors Need To Know

Thu, 23 Apr 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I received an interesting email yesterday from a grad student who took this course the last time it was offered at the University of Toronto. It said in part:

My supervisor could better advise students doing computational work if they had more background knowledge. They are routinely faced with questions like:

Is a project possible, given the background of the student and the difficulty of the tasks?
How long should a project take, and what can be considered good progress?
What training should a student have?
How to manage collaboration between students, data archives, etc?
How to make sense of and build upon work done by previous students?

On a more personal note—I would enjoy my supervisor having a clearer idea of what I do.

We've Started a FAQ

Wed, 08 Apr 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have started a FAQ [no longer online] for the July 2009 offerings of the course in Edmonton and Toronto. Please let us know if you have any questions that it doesn't answer yet.

User Stories

Mon, 30 Mar 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the tricks I teach my undergraduates is to create fictional personas to describe the intended users of a system—or in this case, a course. Here are three of the "people" I've had in mind while developing Software Carpentry to date; my goal is to update these stories to better reflect how scientists work today.

Bhargan Basepair

Bhargan Basepair received a B.Sc. in biochemistry five years ago. He has been working since then for Genes'R'Us, a biotech firm with labs in four countries. He did a Java programming course as a freshman, and a bioinformatics course using Perl as a senior.

Legal Frameworks for Reproducible Research

Tue, 17 Mar 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

My grad students and I met Victoria Stodden for the first time yesterday, and had a great time talking about reproducible research, science 2.0, and most particularly the kind of legal/copyright frameworks needed to move science forward. She has two papers up that anyone interested in the subject should read: a short one that appeared in Computing in Science and Engineering titled The Legal Framework for Reproducible Research in the Sciences: Licensing and Copyright, anda longer one due out soon called Enabling Reproducible Research: Open Licensing For Scientific Innovation. If you have thoughts on the subject, I'm sure she'd enjoy hearing from you.

Computer Supported Collaborative Science

Wed, 18 Feb 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've used the term "CSCS" a few times now; time to start groping toward a definition. "Computer supported collaborative science" (CSCS) is a specialization of computer supported collaborative work, which is the study of "how collaborative activities and their coordination can be supported by means of computer systems". Insert the word "scientific", and you have CSCS. More specifically, CSCS includes science 2.0, open notebook science, reproducible research, workflow & provenance, and other things modern computing technology can do to help scientists find and share information.

Enough Players to Hand Out Medals

Mon, 16 Feb 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I blogged last August about the first and second Provenance Challenge, in which the creators of systems for tracking scientific data and workflows were given sample problems, then asked to have their tools answer a variety of questions. (Results from the first were reported in Concurrency and Computation, but ironically, those articles are not openly available; the third challenge will kick off soon.) Chasing down one of those references again, I came across the Open Notebook Science Challenge, which "...calls upon people with access to materials and equipment to measure the solubility of compounds (aldehydes, amines and carboxylic acids are a priority) in organic solvents and report their findings using Open Notebook Science". This isn't quite the same thing as automatically tracking data provenance; instead, it is "...the practice of making the entire primary record of a research project publicly available online as it is recorded". There are lots of interesting research questions for computer scientists here, ranging from privacy and security issues to notification, peripheral awareness, ontological engineering, and more — for example, see Cameron Neylon's latest post synthesizing discussion about using OpenID to identify scientific researchers and their contributions.

MTEST

Wed, 11 Feb 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Steve Eddins has posted an xUnit-style testing harness for MATLAB called MTEST on the MATLAB Central File Exchange. It's a nice piece of work, and I hope numerical programmers will make heavy use of it.

Python Textbooks for Biotech

Wed, 11 Feb 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Jan Erik Moström recently posted a request for textbook recommendations for teaching programming (using Python) to biotechnology students. He has now posted the responses [no longer online] he received — might be interesting to some readers of this blog.

Sharing Data Isn't That Easy

Fri, 06 Feb 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Interesting post from Systeme D called "ShareAlike considered harmful" (for geo data, anyway). I should give this to my students and ask them to think it through...

Web Native Lab Notebooks

Tue, 27 Jan 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Good post from Cameron Neylon about the lab notebooks of the future: "The traditional paper notebook is to the fully integrated web based lab record as a card index is to Google."

Time to Freshen It Up

Wed, 31 Dec 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I tell the students in my software engineering classes that the absolute value of code coverage in testing isn't as important as the trend: if you're testing a smaller percentage of your software as time goes by, you're headed for trouble. The same is true of site stats: I don't care much about the absolute number of visitors, but if the curve is down and to the right, it's time to give the site a polish, or retire it.

Igor, Connect the Electrodes!

Sun, 30 Nov 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The Software Carpentry course site is still getting a fair bit of traffic, although readership is definitely tailing off:

I'm hoping to run an intensive three-week version of the course in June 2009 in Toronto (details to follow); hope I can find time between now and then to finish wikifying the course notes, get the MATLAB material online, and generally freshen the site up.

One Good Survey Deserves Another

Tue, 04 Nov 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

While we're running our survey of ~~how scientists use computers~~ [link no longer active], the folks at MATLAB are asking their users a few questions too. If you use any MathWorks products, and have a few minutes, they'd be grateful for your help.

1731 People

Sun, 02 Nov 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

1731 people have completed our survey of how scientists use computers since it went online three weeks ago. That's pretty cool, but I'd like to double the number (at least). If you consider yourself a working scientist, and haven't taken the survey yet, please take a moment and do so. If you aren't a scientist, but know some, please pass on the link:
~~http://softwareresearch.ca/seg/SCS/scientific-computing-survey.html~~
[link no longer active]

Surveying Scientists' Use of Computers

Wed, 15 Oct 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Computers are as important to modern scientists as test tubes, but we know surprisingly little about how scientists develop and use software in their research. To find out, the University of Toronto, Simula Research Laboratory, and the National Research Council of Canada have launched an online survey in conjunction with American Scientist magazine. If you have 20 minutes to take part, please go to:

~~http://softwareresearch.ca/seg/SCS/scientific-computing-survey.html~~ [link no longer active]

Data Provenance Challenge

Wed, 13 Aug 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

John's summary of our discussion about what to teach scientists about reproducible research if they already believe it's a good thing, and want to start doing it reminded me that I never posted about the Provenance Challenge. It has been run twice so far; each time, authors of tools to track the provenance (or lineage) of scientific data have to implemented some workflows, then answers questions about where data came from, what was done to it, and so on. The results of the first challenge are described system-by-system in these papers (sorry, but it's behind a wall — if you google for combinations of the authors' names, you can find PDF preprints). This is a very cool research area, and I hope one of my incoming grad students will want to do something with it.

Next Lecture?

Mon, 28 Jul 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The Software Carpentry course currently contains the following lectures:

Introduction
The Unix Shell (2 lectures)
Version Control
Automated Builds
Basic Scripting (bool/int/float, for/while/if)
Strings, Lists, and Files
Functions and Libraries
Programming Style
Quality Assurance (basic testing)
Sets, Dictionaries, and Complexity
Debugging
Object-Oriented Programming (2 lectures)
Unit Testing (unittest — should switch this to nose)
Regular Expressions
Binary Data
XML
Relational Databases
Spreadsheets
Numerical Programming (the basics of NumPy)
Integration (subprocess+pipes and wrapping C functions)
Web Client Programming (HTTP request/response, URL encoding)
Web Server Programming (basic CGI processing)
Security (the weakest lecture of the bunch)
The Development Process (a mish-mash of sturdy and agile)
Teamware (introduces portals like DrProject)
Conclusion (various "where to look next" suggestions)

Between now and Christmas, I want to tidy them up, duplicate the examples in MATLAB, and add some of the content I wrote for "CSC301: Introduction to Software Engineering". Since I won't have time to do everything, I'd like your help prioritizing. Which of the following topics do you think is most important to add? And what have I forgotten entirely?

Quick Quiz to Measure What Scientists Know

Wed, 23 Jul 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Suppose you have a room full of scientists—hundreds of 'em—and want to find out how they actually use computers in their work. There isn't time to interview them individually, or to record their desktops during a typical working week, so you've decided to ask them to self-assess their understanding of some key terms on a scale of:

No idea what it is.
Use it/have used it infrequently.
Use it regularly.
Couldn't get through the day without it.

My list is below; what have I forgotten, and (more importantly) how would you criticize this assessment method?

Reviving the Software Carpentry Mailing List

Tue, 22 Jul 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Luke Petrolekas and I are thiiiiis close to having the Software Carpentry notes converted to a wiki. Once they are, I'm going to be working with Tina Yee to update them, do the examples in MATLAB as well as Python, and fix some longstanding bugs. I'm also going to resurrect the project's two mailing lists (one for occasional announcements, the other for people interested in developing new material and/or teaching the course). If you'd like to be on either or both, please let us know.

Faking Results

Fri, 06 Jun 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via BoingBoing, a story about scientists Photoshopping experimental results. Sometimes it's outright fakery; sometimes they're just "cleaning up" or "correcting". Either way, it raises an interesting question: how often are people doing this with computational results? Without scientists' code, or any other way to reproduce their work, we'll probably never know.

Programming and Scientific Education on Slashdot

Fri, 30 May 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via Adam Goucher, a Slashdot thread about whether programming should be part of science education. 300+ comments and counting, almost all relating personal experiences.

Reminded of the Difference Once Again

Tue, 27 May 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via Irving Reid, a great quote from Henrik Kniberg about how to tell when you're done:

So when a team member says that a story is Done and moves the story card into the Done column, a customer could run into the room at that moment and say "Great Lets go live now", and nobody in the team will say "no, but wait".

Ironically, Kniberg's post is called "Version Control for Multiple Agile Teams". Why ironic? Because I was talking yesterday to a professor in my department — young guy, smart, already has a better research and publication record than I'll ever have. I brought up version control (as in, "Why do less than 10% of faculty and grad students use it?"); his reply was, "Well, we're just going to back everything up once a day."

SE-CSE Workshop

Thu, 15 May 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the downsides of being in Texas is that I couldn't attend the First International Workshop on Software Engineering for Computational Science and Engineering, which was held at ICSE'08 in Leipzig this week. Papers are here (I'll be reading them on the flight home); they look interesting, but the biggest thing for me is the change in the workshop's name—it used to be "Software Engineering for High-Performance Computing Applications" (~~2004~~, ~~2005~~, 2006, 2007), and I'm hoping the change of name reflects a genuine broadening of focus.

Summer Plans for Software Carpentry

Wed, 02 Apr 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The Software Carpentry site is still getting a lot of traffic, despite my neglect:

This summer, I'd like to:

Convert the site from static HTML pages to a wiki to make it easier for people to contribute content and fix bugs.
Translate the examples into MATLAB to make them accessible to a larger audience. (Yes, Python is still my favorite language, and yes, the Python versions will remain—I just want it to be possible for the average mechanical engineer to follow the discussion of testing without first having to learn a new programming language).
Add some of the material that I developed for CSC301: Introduction to Software Engineering, and some of what Titus Brown wrote for Intermediate and Advanced Software Carpentry.

The odds of all three happening are close to zero: my grad students are going to be in the middle of real research, we're hoping to have half a dozen or ten undergraduate interns as well, we have a "CS-1 in Python" book to finish for Pragmatic, and oh yeah, I'm getting married twice. If you want to help out, now would be a good time to raise your hand... :-)

The Retractions Just Keep Coming In

Wed, 02 Apr 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via Titus Brown, yet another published result retracted because of a bug in code.

Nice Quote

Wed, 26 Mar 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

—David L. Donoho, WaveLab and Reproducible Research, 1995, p. 5.

(via Andrew Lumsdaine)

Python Supercomputing Statistics

Sun, 09 Dec 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I have a major grant application and (hopefully final) revisions to my next children's book due on Friday, so of course I'm reading white papers about Python-friendly supercomputing from Interactive Supercomputing, a Boston-area firm that's about three years old. IS offers several kinds of parallelism for MATLAB, Python, R, SAS, and other high-level languages; I don't know if their tools are any easier to use than anyone else's, but they have an impressive team (including Russ Barbour, ex-Apollo, and Steve Reinhardt, ex-Cray).

Openness and (the promise of) XML

Wed, 05 Sep 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Over at O'Reilly, people are gloating about ISO's decision not to ratify Microsoft's OOXML document "standard". At the same time, Jon Udell is saying much more interesting things about what smart document formats could do for the sciences. To quote him quoting Clifford Lynch, "Scientific literature that is computed upon, not merely read by humans."

Two Studies of ASCI (and no, that's not a typo)

Wed, 27 Jun 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

After spending ten years helping scientists write programs for massively-parallel computers, I realized that what scientists really needed was to learn how to program, full stop. It took me another eight years to (a) get up to speed with the theory and practice of modern software engineering, (b) realize how big the gap between the two was, and (c) accept that the only people who are entitled to have an opinion about how we ought to be building software are the ones who are studying how well their favorite tools and practices actually work in the field. I no longer care what you're pushing—formal methods and cleanroom development or agile adhocracy and pair programming—unless you've gone the extra mile and collected data to show what effect it's actually having.

Software Carpentry Screencasts by Chris Lasher

Wed, 20 Jun 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Chris Lasher, a grad student at the University of Virginia Virginia Tech (sorry, sorry), has posted some screencasts about version control based on the Software Carpentry notes at ShowMeDo.com; a second series about using the shell should be up shortly. It's just plain cool to see the material picked up and carried forward in ways I wouldn't even have thought of a couple of years ago.

Praising the Good

Mon, 11 Jun 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I frequently gripe about how backward scientific software development is, so it's only fair that I give praise where praise is due. Steve Easterbrook pointed me at the UK Met Office's Flexible Configuration Management system, which combines code management using Subversion and Trac with a GNU Make-based build and an extraction system that pulls data out of the repository and builds it for particular platforms. This image shows how the three parts fit together; it ain't rocket science, but it's very professional, and a good model for others to follow.

Reproducibility of Computational Results

Sat, 10 Mar 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via Titus Brown, a link to the Insight Journal, an open access online publication covering medical image processing. They have a very interesting process requirement: your source code must compile & be verifiable by an automatic system. I've been expecting something like this for a long time; glad to see it happening.

Later: and, via Gary Bader, Source Code for Biology and Medicine. Anyone know of a journal or journals like this for physics, chemistry, geology, and other non-life-science areas? Or (wistfully) computer science?

Lecture on Binary Data

Tue, 14 Feb 2006 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The Software Carpentry lecture on binary data is now up on the web. The content of this one has been fairly stable for a while, but that just means that all the bugs will be in the details—comments and corrections are greatly appreciated.