Assessment on The Carpentries

Share Your Feedback in The Carpentries 2026 Community Survey

Mon, 29 Jun 2026 00:00:00 +0000

Last week, we launched The Carpentries 2026 Community Survey and invited community members to share feedback about their experiences in our community. If you have not yet completed the survey, there is still time to participate.

The survey helps us better understand how community members engage with The Carpentries, the benefits they receive from participating, and how the Core Team can better support our global community. The results will inform future planning and help us identify areas for improvement in programs, communications, resources, and community support.

Incentivising the Completion of Workshop Surveys: Tips for Instructors

Mon, 18 Aug 2025 00:00:00 +0000

As all Carpentries Instructors probably know, it is very hard to get people to complete workshop surveys. It was time to find a solution, but what can we do to inspire folks to complete yet another boring survey. Despite all my promises to learners that we look at and consider all the comments and feedback we get, it has been a miracle every time we get more than 50 per cent of attendees to complete it. Giving people time, at the end of the lesson, to complete the survey helped a bit, but not much.

Sharing the 2023 Community Survey Evaluation Report

Thu, 14 Mar 2024 00:00:00 +0000

In 2023, The Carpentries launched a community survey to inform and prioritise our work and that of the Core Team so we can better serve the community. We asked community members to take 5 minutes to provide feedback on their experience as Carpentries community members, seeking feedback that will help us better understand the benefits community members get from their participation and inform areas of improvement.

We received 200 responses to the survey. It is difficult to identify a response rate without an exact count of the number of active community members. However, this represents 5% of The Carpentries Slack workspace members and 7% of those subscribed to Carpentries Clippings, our newsletter.

Announcing our new Director of Assessment

Thu, 17 Nov 2022 00:00:00 +0000

A Message from Erin Becker

It is with great pleasure that I announce that Dr. Kelly Barnes is advancing to a Senior Team Lead role and will now be serving as The Carpentries Director of Assessment. Over the past three years, Kelly has brought her expertise in assessment and evaluation to the Instructor Training team, improving and strengthening the Instructor Training and Trainer Training programs. In her new position, she will be able to expand her reach and bring high-quality assessment to all of The Carpentries core activities - including curriculum development and maintenance, workshops, and community development. Congratulations Kelly!

The 2021 Instructor Training Report

Wed, 01 Sep 2021 00:00:00 +0000

The Carpentries Instructor Training Team is excited to release the 2021 Instructor Training Report. This report is an in-depth look at the Instructor Training program and provides an update on its successes, challenges, and growth.

Quick Facts

Instructor Training Outcomes

Trainees rate Carpentries Trainers very highly.
- The majority of trainees thought Trainers gave clear answers to questions (97%), were good communicators (97%), considerate (99%), and enthusiastic (98%).
Trainees report higher levels of ability and knowledge after Instructor Training.
98% of trainees agreed that the atmosphere during Instructor Training was welcoming.
92% of trainees agreed that Instructor Training was worth their time.
74% of trainees report an increase in motivation to learn about teaching after Instructor Training.
Trainees are very likely to recommend Instructor Training to a friend or colleague.
- Recommendation scores have averaged 8.43 - 9 out of 10 for the last five years.
81% of Trainees hear about Instructor Training through word of mouth.

Instructor Training Growth

Our Long-Term Survey is Open for Responses

Mon, 28 Sep 2020 00:00:00 +0000

In April 2020 we released version 2 of The Carpentries Long-Term Survey Results. We learned that more than half of survey respondents use self-guided Data, Library, or Software Carpentry material months after attending a workshop. As the unprecedented Coronavirus pandemic has pushed our community to transition to virtual instruction, we are interested in receiving feedback from individuals who participated in a workshop in the last 6 months. There is much to learn about where learners are in their journey.

How We Use Feedback at The Carpentries

Fri, 04 Sep 2020 00:00:00 +0000

As you were filling out yet another survey or feedback form, you might have found yourself wondering, “What does The Carpentries do with all of this feedback anyway? Do they even use it? Do they really need to hear from me?” And the answer is yes, we definitely do!

If you have hung around The Carpentries for any period of time, you know that community is core to all we do. Everything is co-created by the community with support from The Carpentries Core Team. But to be able to co-create, we have to have strong lines of communication in all directions. As Core Team members, our job is to listen to you and use your feedback to inform the work we do. It is just as important, however, that we close that feedback loop by sharing back to you, to facilitate your work and to showcase your accomplishments. Since some of us really love concept maps here at The Carpentries, here is a little diagram to illustrate the many ways that we listen and share through our assessment work:

The Carpentries Long-Term Survey is Open

Thu, 28 Nov 2019 00:00:00 +0000

Our long-term survey was developed to collect information from learners at least six months after completing a Carpentries workshop. To date, we have received 633 responses to our long-term survey between March 2017 and September 2019. We are officially opening our Long-Term Survey and would like anyone who has taken a Data, Software, or Library Carpentry workshop between April 2018 and December 2019 to provide feedback by completing the survey.

Long-Term Survey

A Better Experience for Workshop Surveys

Mon, 21 Oct 2019 00:00:00 +0000

Assessment is a key element of our operations. Today, we are excited to announce that we are revamping the tools behind our pre- and post-workshop surveys to provide a better experience to our learners and to our instructors. This change also greatly reduces the time needed for the workshop administration team to generate the links they share with instructors to access the results. The changes outlined here only affect our regular workshops, but in early 2020, they will also affect Instructor Training events.

Shaping Our Programs through the Lens of Equity.

Tue, 15 Jan 2019 00:00:00 +0000

We have officially been operating as The Carpentries for one year now. Can you believe it?

I joined Data Carpentry’s staff as Deputy Director of Assessment in August of 2016. Since that time, and with the help of countless community members, I’ve been able to streamline our workshop surveys, develop and revise our instructor training surveys, expand our mentoring program, and liaise with the Code of Conduct and Instructor Development Committees to revise our Code of Conduct and improve our Community Discussions. Every day I get to do meaningful work and collaborate with a global network of amazing people.

The Carpentries' Programmatic Assessment

Tue, 19 Jun 2018 00:00:00 +0000

Starting this year, we are releasing quarterly programmatic assessment reports (see this blog post). These reports will share information about the growth rates of our workshops and instructor training programs, our geographic impact, and other metrics. We are happy to announce the publication of our first report.

This first report follows activity around Carpentries’ workshops, Instructors, and Trainers from 2012 through the first quarter of 2018 (January - March). From 2012 through March 2018, 58 Trainers badged 1,480 Instructors. Carpentry Instructors taught 1,332 workshops, reaching more than 37,000 learners in 44 countries. The report also highlights our significant geographic expansion, such as the growth of an active community of Instructors and Trainers in eight African countries.

Webinar with Rochelle Tractenberg: Debrief

Tue, 20 Mar 2018 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

On February 2, the Assessment Network held a well-attended webinar with Rochelle Tractenberg. Dr. Tractenberg holds two PhDs, one in Cognitive Sciences and the other in Statistics, and directs the Collaborative for Research on Outcomes and Metrics at Georgetown University, where she is a tenured professor in the Department of Neurology. It was a great privilege for our community to be able to engage in a conversation with her and to learn from her expertise.

State of the State: Carpentry Maintainers

Wed, 31 Jan 2018 00:00:00 +0000

This post originally appeared on the Data Carpentry website

All of the great work that we do as The Carpentries is dependent on the hard work and creativity of our community of volunteers. Each of you plays a vital role in helping us fulfill our mission of spreading data skills and computational literacy to researchers and other professionals worldwide. Within our Carpentry community, there are a number of subcommunities of like-minded folks carrying out particular aspects of the Carpentry mission. This blog post is the first in a series focusing on one of our sub-communities - the Maintainers.

Challenges Assessing Data Science

Mon, 11 Dec 2017 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The Assessment Network was established as a space for those working on assessment within the open source/research computing community to collaborate and share resources. During our quarterly meeting in November, we engaged one another in a conversation revolving around data science education. This meeting was organized and hosted online by Kari Jordan, and six community members attended.

First, we discussed the definitions of data scientist, data analyst, and data engineer; second, we worked in pairs on a set of questions about assessing data science education.

When Do Workshops Work? A Response to the 'Null Effects' paper from Feldon et al.

Mon, 11 Dec 2017 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Author: Karen R. Word

Contributors: Kari Jordan, Erin Becker, Jason Williams, Pamela Reynolds, Amy Hodge, Maxim Belkin, Ben Marwick, and Tracy Teal.

“Null effects of boot camps and short-format training for PhD students in life sciences” is the provocative title of a recent article in the Proceedings of the National Academy of Sciences. Those of us who enthusiastically design and deliver short-format training promptly took note, then scratched our heads a bit. We waited a little for a response, wondering if one or more of the programs that participated in the study might step up to their own defense. Nothing happened. We thought about letting it go - we’ve got our own programs, with distinct goals, and our own assessment data, so maybe this broad-brush study isn’t so important. But … it keeps being raised. Someone will bring it up here and there, asking what we think about it. Whenever this paper comes up in conversation, its title certainly throws some weight around.

Our long-term assessment results are in!

Wed, 18 Oct 2017 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A discussion of learner outcomes more than six months after attending a Carpentries workshop

What concrete changes are people implementing in their computational research practices as a result of completing a Carpentries workshop?

Our long term survey report shows that two-day Software or Data Carpentry workshops are effective for increasing skills and confidence, and the adoption of reproducible research perspectives. We see gains in our survey measures for learners’ motivation to continue their learning, change in reproducible research behavior, and frequency of use of computational skills and tools. We find this very exciting, especially since a recent general survey of bootcamps and short-format trainings reports no measurable impact on skill development or research productivity.

Reflections on Assessment

Mon, 07 Aug 2017 00:00:00 +0000

This post originally appeared on the Data Carpentry website As I approach my one-year anniversary on staff with Data Carpentry, I’m reflecting on what we’ve accomplished this year. I came into this role with no data science experience whatsoever: self-efficacy had been the focus of my doctoral dissertation and post-doctoral work. That background knowledge made for an exciting time exploring our workshop survey data and letting the community know what our learners think about our workshops.

Analysis of Software Carpentry Workshop Impact

Mon, 10 Jul 2017 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We’ve begun to look at our pre- and post- workshop surveys and are sharing the draft reports to stimulate conversation about our workshops and their impacts on learners. For this first cut, we looked at our post-workshop survey data.

From this survey data, we can see that learning to program can be quite intimidating for many learners. About 44.5% of Software Carpentry learners who responded to the post-workshop survey feel that at least one of the tools covered in the workshop they attended was either slightly or very intimidating to them before attending a workshop.

Long-Term Assessment Strategy

Mon, 27 Mar 2017 00:00:00 +0000

This post originally appeared on the Data Carpentry website A fair amount of discussion around assessment took place during the Software and Data Carpentry Instructor and Helper Retreat. One of the recommendations that came out of this retreat was to include skills-based questions in our pre- and post-workshop surveys to improve the accuracy of our survey results. We know that our learners come to our workshops having varying knowledge levels for programming. A way that we can capture this information, and determine whether our workshops make a difference in learner’s ability to work with data, is to add skills-based questions to our surveys. With the help of our community, we have done just that.

Analysis of Data Carpentry Workshop Impact

Fri, 04 Nov 2016 00:00:00 +0000

This post originally appeared on the Data Carpentry website It’s funny. When I first started working for Data Carpentry, I had never heard the phrase, “reproducible research”. I can tell you now that having attended a Software Carpentry workshop, Data Carpentry workshop, and a Software/Data Carpentry instructor training, I wish I had learned the skills we teach when I first began my PhD program.

I even confessed to my colleagues that the data I left behind for up and coming grad students is so disorganized, I sent them an e-mail to apologize! This community has made a believer out of me, and for good reason: Our workshops work.

Community Call on Assessment

Thu, 20 Oct 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Dr. Kari L. Jordan, Data Carpentry’s Deputy Director of Assessment, will be speaking on the Community Call on the impact our workshops have on our learners. Kari will cover the results of our post-workshop surveys including learner attitudes, learner data management and analysis skills prior to and following our workshops, and learner attitudes toward Data Carpentry instructors. Come and hear about our successes and share your thoughts. It’s going to be an informative community call.

Community Call on Assessment

Thu, 20 Oct 2016 00:00:00 +0000

This post originally appeared on the Data Carpentry website

Beth Duckles on the Practice of Measuring

Mon, 10 Oct 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Dr. Beth Duckles, who did a valuable study of our instructor community earlier this year, gave a talk at the recent Measuring the Impact of Workshops workshop titled “The Practice of Measuring”. It’s a very useful 50 minutes, particularly for those of us who have backgrounds in the physical rather than social sciences.

Responding to your Learners

Fri, 09 Sep 2016 00:00:00 +0000

This post originally appeared on the Data Carpentry website

Picture for a moment one of your undergraduate university courses, or a recent meeting of your department’s research seminar. If your experience is typical, more likely than not the instructor or presenter came in with a prepared set of materials on a slide deck (or for the old-school, on projector overheads), talked at you for an hour, maybe asked if there were any questions, and then packed up and left. If this sounds familiar, you’ve probably had lots of experience with the standard lecture.

Announcing our new Deputy Director of Assessment

Tue, 02 Aug 2016 00:00:00 +0000

This post originally appeared on the Data Carpentry website

We are very happy to announce that Dr. Kari L. Jordan will be joining us as the new Deputy Director of Assessment! Kari is coming from Embry-Riddle Aeronautical University in Daytona Beach, FL where her postdoctoral research was focused on understanding the factors that influence faculty adoption of evidence-based instructional practices. She is also adjunct faculty at Embry-Riddle and teaches a Graphical Communications course. Prior to her postdoc, Kari completed a PhD in STEM Education at Ohio State. Her thesis focused on strategies for improving self-efficacy and sense of belonging in first-year engineering students. She holds a BS and MS in mechanical engineering from Michigan Tech. Kari’s experiences in educational research are closely aligned with Data Carpentry’s efforts to create a welcoming and inclusive workshop environment that enhances learners’ confidence and helps them build identities as data-driven researchers. As the Deputy Director of Assessment, Kari will be focusing on assessment of workshop learning outcomes, including both skill-based and identity-based outcomes.

Data Carpentry's New Deputy Director of Assessment

Tue, 02 Aug 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Data Carpentry has just announced that Dr. Kari L. Jordan will be joining them as the Deputy Director of Assessment. Kari holds a PhD in STEM Education from Ohio State University, and has worked most recently at Embry-Riddle Aeronautical University in Florida, where her postdoctoral research focused on understanding factors that influence faculty adoption of evidence-based instructional practices. To learn more, please see the post on the Data Carpentry website or follow Kari as @drkariljordan on Twitter.

Assessment Update - 2015

Tue, 22 Dec 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

About a year ago, I hoped to help answer the question, "Is our students learning?" We are now a bit closer to an answer, and I wanted to update the community about what we have accomplished this year. First, I want to thank and acknowledge the assessment subcommittee and Blake Joyce, who helped to define tasks, possibilities, and provided significant feedback. Especially here I wanted to thank Daniel Chen, Jeramia Ory, and Rayna Harris who put in a lot of hours moving things along.

A Practical Computing Course

Sun, 15 Nov 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

While it was not a SWC course, this summer Casey Dunn and I taught a 12-day Practical Computing summer class at Friday Harbor Labs. It was a great group of students—mostly somewhat beginner level—and we covered regular expressions, the shell, Python, R + ggplot, Git, graphics, and bit of electronics. The centerpiece (which the students really got into) was a personal project that was applicable to their own interests and which they solved with some combination of the tools we covered. Many of the students went from zero experience to having a script that actually gave them insight into the real-life research back home.

Assessing Assessment

Sat, 10 Oct 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Jason Williams and I met with two consultants at the University of Texas at the end of September to get feedback on Software Carpentry's post-workshop survey. They gave us detailed suggestions for improving six of the questions, and felt the rest were OK as they are. The feedback is given below; even without the whole questionnaire (which we will post shortly), we hope it's helpful.

Assessing Our Learners Part I

Tue, 23 Jun 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Three weeks ago, Jason Williams, Jeramia Ory, and Daniel Chen met at the New York Public Library to work out an initial survey to assess our learners. Greg Wilson and Katerena Kuksenok joined virtually to provide feedback. The goal was to take the comments from the various initial GitHub issues and create a draft of an assessment survey for everyone to provide input. Our first draft is up, so please provide feedback at {{site.github_url}}/assessment/issues/6.

Plan to Assess Our Learners

Thu, 21 May 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The assessment subcommittee seeks to assess the effectiveness of the activities of the Software Carpentry Foundation (SCF). It met a few weeks ago and drafted an action plan on how to move forward to create a series of assessments for our learners. This action plan will be the basis on how further assessment tools will be developed.

Learning in Both Directions

Tue, 21 Apr 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have spent a lot of time thinking about how to assess the impact that Software Carpentry is having. We've done some small studies and collected a few testimonials, but it's been small potatoes compared to the 5000 people we taught last year alone.

After some back and forth with a colleague whose work I have admired for years, though, I realize that I've been trying to do this the wrong way. My training as an engineer taught me that only controlled, quantitative experiments were "real" science—that as Ernest Rutherford said, it's either physics or stamp collecting. I now understand that there are other rigorous ways to generate actionable insights, some of which are better suited to our needs than something like randomized control trials. More than that, I finally understand what one of my first teachers told me:

Teaching only works well when the teacher is also learning.

What Do People Want to Learn?

Sun, 15 Mar 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

In the planning phase of organizing a Software Carpentry workshop for my home department of Molecular Biology & Biochemistry I started to wonder what participants want to learn. I designed a short survey to answer this question, and from my small department, ~20% (30 people) filled it in. Here's what they said:

Data Carpentry Genomics and Assessment Hackathon

Wed, 28 Jan 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

If you're working or interested in genomics or assessment, we hope you'll consider applying for our upcoming Data Carpentry Genomics and Assessment hackathon. We're very excited about this event and the opportunity to develop lessons targeting genomics researchers and build assessment into the Data Carpentry curriculum. Travel support is available, so please apply to participate! It's a short application, and the deadline is this Friday, January 30th. If you have any questions about the event, please let us know.

Dates: March 23-25, 2015
Location: Cold Spring Harbor Labs, NY
Call for Participation
Application

The New Instructor Post-Assessment Questionnaire

Wed, 19 Nov 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As mentioned two weeks ago, a group led by Daniel Chen have been revising the post-workshop survey for instructors so that we can get a better idea of who's actually teaching what, how long it's taking, and how it's going. The latest draft can be viewed here, and is included below. We'd be grateful for one last round of feedback—in particular, we're concerned about (a) the overall length and (b) how closely the headings are tied to our current curriculum. Please let us know what you think in comments on this post.

The Fifth ANGUS Course

Thu, 21 Aug 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Titus Brown recently blogged a summary of the fifth run of the Analyzing Next Generation Sequencing (ANGUS) course at Michigan State. It includes some interesting observations on what's working and what needs to be improved, and some thoughts on assessment—he'd welcome feedback.

Data Science Study Invitation

Sat, 24 May 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Katie Kuksenok, a graduate student at the University of Washington, is interviewing academic researchers who do data science to explore the barriers and challenges they face. If you would be willing to take 20-30 minutes to be interviewed, please get in touch: among other things, her findings will be used to help improve curriculum for projects like Software Carpentry.

Assessment Results: First Batch

Thu, 08 May 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As many of our instructors and participants know Software Carpentry has been giving surveys for a number of months at every possible workshop. This takes a surprising amount of coordination and attention, much of which we owe to our wonderful two administrators: Amy Brown and Arliss Collins. The surveys are being administered before and after each workshop in an effort to gather difference data which could answer questions such as "How useful was this workshop to participants?", "Who benefits most from these workshops?", or "What ways could we improve the workshop?" We can also use this information to ask about the professional demographics such as "What fields are most represented by our participants?" and "What's the typical amount of programming experience?" both of which could help inform a tailored workshop experience to better suit the audience.

Oxford, One Year On

Wed, 25 Dec 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Philip Fowler recently posted a nice study of the impact last year's Software Carpentry bootcamp at Oxford had on its participants. The key finding for me is that 10 of the 13 respondents to his survey agreed or strongly agreed with the statement, "A year on, the workshop really encouraged me to change how I do my research." If you'd like to help us gather similar information from participants at other bootcamps, please get in touch.

Moving Forward with Assessment: Interviews

Sun, 17 Nov 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We would like to quantify realistic impact after the dust settles from a bootcamp. In particular, we'd like to know how (or whether) Software Carpentry has improved scientists' workflow three months or more after the bootcamp, on the theory that if people are still using things then, they must have found them useful. Our most recent steps toward this goal include obtaining Internal Review Board for Human Subjects Research (IRB) approval through Michigan State University and beginning to recruit volunteers for interviewing at the three month mark.

Bootcamp Student Composition by Domain

Mon, 30 Sep 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Recently I was challenged to show that computational science is important to biologists, and, more importantly, that biologists care about learning how to develop quality software. I am not a biologist. While I find the subject fascinating I cannot make any broad claims on the topic or hope to speak for all biologists. So I turned to Software Carpentry.

My experience teaching Software Carpentry and related bootcamps has been that biologists are always a significant portion of the student population. In order to demonstrate this I asked to have access to anonymized answers to part of the pre-assessment survey. Specifically, I was interested in the "Which of the following most closely aligns with your primary discipline?" question. With a big assist from Amy Brown I obtained the data I was looking for.

Scientific Computing at AIMS

Sun, 15 Sep 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Close on the heels of another bootcamp in Madison, WI, for the past two weeks I had the pleasure of teaching a 30 hour scientific computing course at the African Institute for Mathematical Sciences (AIMS) in Cape Town, South Africa. The course was an expansion and refactor of Software Carpentry material to fit the AIMS need. You can find the repository of source material here.

I could gush about how I fell in love with Cape Town, the unexpected preponderance of vegan cafes in Muizenberg, the sea, the sand, the mountains, and la dolce vita I am otherwise used to from Santa Barbara. I will not. AIMS is about the students. Let's examine them after a brief tour of AIMS itself.

Creating Assessments and What to do With the Data

Tue, 20 Aug 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Our new assessment questionnaires are up for your review, so I thought I'd talk a little bit about the work behind the scenes, and some lessons learned. Over the past couple months I've had the opportunity to interview bootcamp participants who attended the Michigan State University Software Carpentry Bootcamp a little over one year ago, May of 2012. The goal for interviews were two-fold: develop assessment questionnaires, and gather data.

Bootcamp Questionnaires

Mon, 19 Aug 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thanks to some hard work by Jory Schossau and others, we have developed three standard questionnaires for bootcamp participants:

Pre-assessment for Learners
Post-assessment for Learners
Post-assessment for Instructors

We hope to deploy these questionnaires starting in September, so we would be very grateful for feedback. Do you understand what we're asking? Do the responses provided include the answer you would give? As an instructor, would you find it useful to know these things about your learners before a bootcamp started? And what have we missed entirely?

Computational Competence for Biologists

Tue, 16 Jul 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

On July 8 and 9, I had the pleasure of taking part in a two-day workshop at SESYNC to discuss what we ought to teach biologists about computing. It was a relatively small meeting, but the participants spanned the range from computer scientists and systems engineers to bioinformaticians, field biologists, and a few odd ducks like me.

Software Carpentry: Lessons Learned

Thu, 20 Jun 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

With contributions from:

Azalee Bostroem (Space Telescope Science Institute)
Chris Cannam (Queen Mary, University of London)
Stephen Crouch (Software Sustainability Institute)
Matt Davis (Space Telescope Science Institute)
Luis Figueira (Queen Mary, University of London)
Richard "Tommy" Guy (Wave Accounting)
Edmund Hart (University of British Columbia)
Neil Chue Hong (Software Sustainability Institute)
Katy Huff (University of Wisconsin)

Michael Jackson (Edinburgh Parallel Computing Centre)
W. Trevor King (Drexel University)
Justin Kitzes (University of California, Berkeley)
Stephen McGough (University of Newcastle)
Lex Nederbragt (University of Oslo)
Tracy Teal (Michigan State University)
Ben Waugh (University College London)
Lynne J. Williams (Rotman Research Institute)
Ethan White (Utah State University)

Abstract

Over the last 15 years, Software Carpentry has evolved from a week-long training course at the US national laboratories into a worldwide volunteer effort to raise standards in scientific computing. This article explains what we have learned along the way the challenges we now face, and our plans for the future.

Git vs. Subversion and Feedback in General

Fri, 10 May 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Software Carpentry's mission is to help scientists teach other scientists how to be better programmers. If we want to do that successfully, we need to be scientists ourselves. In particular, we need to base what we teach on evidence, not anecdotes or personal preferences.

For example: we taught Git at the Toronto bootcamp last week, and once again I think our learners would have absorbed more if we'd taught Subversion. Why? Well, take a look at this diagram by Oliver Steele (which I found on this page written by Nick Quaranto):

Pre-Assessment Results

Tue, 30 Apr 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Of the 29 people who responded to a brief questionnaire before a recent bootcamp, we have:

18	graduate students
4	postdocs
4	staff
1	faculty member
1	general public
1	special student

They describe their current expertise as:

1	master
10	have written a few programs
17	have written a little code
1	has no programming experience

Here's how they thought they could do on some simple tasks:

Write a short program to read a file containing columns of numbers separated by commas, average the non-negative values in the second and fifth columns, and print the results.

11 could do it easily

15 could probably struggle through

3 wouldn't know where to start
In a directory with 1000 text files, create a list of all files that contain the word "Drosophila", and redirect the output to a file called results.txt.

12 could do it easily

10 could struggle through

7 wouldn't know where to start
Check out a working copy of a project from version control, add a file called paper.txt, and commit the change.

3 could do it easily

4 could struggle through

22 wouldn't know where to start
A tab-delimited file has two columns: the date, and the highest temperature on that day. Produce a graph showing the average highest temperature for each month.

19 could do it easily

0 could struggle through

10 wouldn't know where to start

Evaluation Revisited

Mon, 08 Apr 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Caitlyn Pickens (a graduate student at Michigan State who is interning with us this summer) has just posted an article about the validity of different kinds of assessment. Caitlyn's goal is to build some tools we can use to gauge the impact Software Carpentry is having—like almost everything involving human beings, it turns out to be more complicated than it first appears, and her post helps lay out the background knowledge needed to understand why.

Cumulative Enrollment

Sun, 17 Mar 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One measure of how well we're doing is the number of people we've helped. Here's what the last 15 months have looked like:

Pre-Assessment

Tue, 13 Nov 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the recurring problems in our bootcamps is that at any point, about 1/4 of people are lost, and 1/4 are bored. The only fix we can think of is to let people self-assess before arrival to determine whether this is the right thing for them or not (and to let instructors know more about who they're going to be teaching). We've used variations on the questionnaire below a couple of times with useful results; we'd appreciate feedback on:

Two Self-Assessments

Fri, 26 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A recurring problem with our workshops is the diversity of learners' backgrounds. Below are the results of two pre-workshop self-assessments (on different groups), one phrased in terms of content, the other in terms of specific tasks. The consistency is better than I expected: broadly speaking, it seems that people's impression of how much they know about X is more or less in line with their belief about whether they could do something that requires understanding X. The next step will be to follow up a few weeks after the workshop to see if they actually can do X.

Feedback from Newcastle

Wed, 24 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This week the Digital Institute at Newcastle University with The Software Sustainability Institute ran a second bootcamp at Newcastle University. This bootcamp was run for attendees from the North East of England, from whom there was demand after the first Newcastle bootcamp, back in May.

Feedback from the attendees was good though some wanted more time or detail on specific subjects. The most popular comment was the desire to have the slides and instructor scripts available so that attendees could catch up if they fell behind the instructor. The other major challenge, again, was Cygwin. We may want to provide a VM for attendees who typically use Linux but have a Windows laptop.

Post-Mortem on the NGS Course

Tue, 18 Sep 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Titus Brown has posted a post-mortem on his Next Generation Sequencing course, and the independent assessment that was done of it. As with Jorge Aranda's assessment of Software Carpentry, the results were very positive—what we and other like-minded people are doing really does make a difference.

Independent Assessment of the Past Six Months

Thu, 05 Jul 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As many of you know, Dr. Jorge Aranda has been doing an independent assessment of Software Carpentry's effectiveness over the past six months. His brief was to tell us whether we were having an impact on scientists' lives, and if so, what kind, and how we could do better. He just submitted his final report (PDF), and we would welcome your thoughts.

UCL Bootcamp: Version Control Wrap-Up

Fri, 04 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

For the bootcamp at UCL, we tried using Mercurial (with EasyMercurial) instead of Subversion in the version control segment.

You can see the plan for the segment on this EasyMercurial project page. Briefly, we opened with a few plain slides about the purpose of version control, followed by a hands-on example in three parts (working by yourself, working by yourself with an online remote repository, and working with others). We started at the beginning and got as far as "hg bisect", but did not cover branching.

The Good and the Bad of It

Wed, 02 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've come to the end of a hot, slightly sweaty, two days of learning in UCL and have just done a straw poll of participants of the good and bad points of this bootcamp. Something that was trialled at UCL was asking participants to sign up in small (3-4 people) teams drawn from their local research group. In total, we had a little over 40 learners and 10 helpers. We also taught the version control portion using EasyMercurial (thanks to Chris Cannam from SoundSoftware).

Halfway Home

Tue, 17 Apr 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We're half-way through our current round of work, so it's time to start thinking about what we've accomplished, what we've learned, and what we'd like to do next. Here's what I think we now know:

Our training makes scientists more productive.
We can prove it.
Our methods scale.
We can become self-sustaining in 2-3 years.

In more detail:

1. Our training makes scientists more productive.

Wrapping Up the STScI Course

Fri, 16 Mar 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The online portion of our work with learners at the Space Telescope Science Institute wound up today. 6 of the 14 people who took part in the on-site workshop submitted "graduation exercise" videos, and three more sent apologies (time pressure, technical difficulties, etc.), which I think is a pretty good completion rate. As always, we finished up by asking everyone to give us one good and one bad thing about the class:

Help Us Write Assessment Questions

Mon, 05 Mar 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have been asking people the questions below before the start of a bootcamp in order to get a handle on how much they already know. As part of evaluating the current round of work, we'd like to expand the list a little, and touch on a few more topics. Our rough categorization of how much someone should know about various things is in the competence matrix; ideally, we want another 3-4 questions on each topic. Our criteria are:

Toronto Bootcamp February 2012: How We Did

Thu, 01 Mar 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Last week's bootcamp at the University of Toronto was not the most successful one we've ever done: quite a few registrants didn't show up, and based on feedback from the instructors, we tried to cover too much, too fast. That said, the learners who stuck with us all the way through to Friday afternoon came away knowing a lot of useful things (or at least knowing that those things existed). We'll gear back a bit for next week's workshop at Indiana U. and see how that goes.

Badges (Finalized)

Mon, 27 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have finalized our first set of Software Carpentry badges—with luck, we'll list the first set of recipients later this week.

	for people who complete the core curriculum
	for helping to teach (either workshops or online)
	for organizing workshops
	for creating (or fixing) content

What Deep Thoughts Look Like

Wed, 22 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Before writing yesterday's post about assessment, I should have explained what I mean by"fundamental concepts". I'll start with Lewis Epstein's wonderful book Thinking Physics:

Here's a typical problem from the book. Put a block of ice in a bathtub, then fill the bathtub to the brim with water, so that the block is floating freely. When the ice melts, will the water level go up (causing a spill), go down, or stay the same? Hm... well, the ice displaces its own weight of water, so when it melts, it exactly fills the "hole" it made, so the water level stays the same.

Assessment Redux

Tue, 21 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The single biggest challenge Software Carpentry faces right now is how to tell what impact it's having. This is only partly to satisfy funders—as I said back in December, if we don't know how to tell if we succeeded, we're going to fail. It would be (relatively) easy to put together a multiple-choice quiz to see how much people have learned about basic shell commands, the syntax of Python, and so on, but that would only address the shallowest aspects of learning. We're trying to impart some fundamental principles, and what we need is questions that will tell us whether people have internalized them. (As many studies have shown, it's possible to get a decent score on a quiz without actually understanding the subject matter.)

Badges (Mark 1)

Tue, 21 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of our key deliverables for the Sloan Foundation-funded work is a badging program built on top of Mozilla's Open Badges Initiative. Riffing on our new logo, Carri Han has designed three badges for us:

	for people who have mastered our core content
	for people who have organized and run workshops
	for people who have created content

Please let us know what you think.

Pre-Workshop Questionnaire

Sun, 12 Feb 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've been asking people to fill in the short questionnaire below before our workshops in order to give us a better idea of what they want and what they already know. How do you score, and how could we improve the questions?

Roughly what fraction of your working time do you spend creating, modifying, or testing software?
What operating system will you be using in this class?
- Linux
- Mac OS X
- Windows
Do you understand the following Unix shell commands well enough to explain them to someone else?
- ls data/*.txt
- find ~ -name '*.py' -print
Do you understand the following Subversion commands well enough to explain them to someone else?
- svn update
- svn diff -r 1723
Do you understand the following Python statements well enough to explain them to someone else?
- {'east' : 5, 'west' : 11}
- __init__(self, *args)
Do you understand the following testing concepts well enough to explain them to someone else?
- fixture
- mock object
Do you understand the following Make terms and commands well enough to explain them to someone else?
- dependency
- cp $< $@
Do you understand the following SQL terms and commands well enough to explain them to someone else?
- select * from data where data.left < data.right;
- inner join
What do you hope to gain from this workshop? What could we do to make it most useful for you?

Spring 2011 Course Over

Wed, 30 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The Spring 2011 course is now over! We had a wonderful time, learned a lot, and hope the same is true for everyone who participated in the course.

We would love it if you would take a minute or two and give us any comments, feedback, and suggestions you have regarding the course and your experience in it.

To start the ball rolling, Erin Osborne gave us some awesome feedback, posted with permission:

Fall 2010: What Went Right, What Went Wrong

Thu, 02 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As we're getting close to wrapping up this semester's offering of Software Carpentry, we've been putting some thought into the things that worked and didn't work in terms of how we administered the course. Here's what we came up with:

What Worked Well

Version Control — Students really seemed to recognize the value of incorporating version control into their work practices. Of all the topics covered so far, we've received the most positive feedback about version control.
Midterm Quiz — This was useful for assessing the students' learning, and getting an idea of how many of the students were keeping up with the material.
Check-ins — About a month ago, we had each of the students meet with us on Skype for 10 minutes or so. Again, this helped us get an idea of how the students were doing with the course material, and also allowed the students to give us feedback on things they'd like to see changed/added to SWC.
TA Organization — Although we got off to a bit of a slow start, the Software Carpentry TAs managed to get organized in terms of the administrative aspects of the course. Division of labor, and explicitly assigning tasks to specific people seemed to work well.
Asking Students for their Problems — We spoke to a few students about the kinds of computer-related problems they are facing in their research. Next semester we hope to write weekly blog posts in which we will work through the solutions to the problems that the students are reporting.
They like us! — The feedback we've been getting from students indicates that they are enjoying the SWC course, and see the value of the material.

What Went Wrong

Mid-term Quiz Results

Sat, 20 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A few weeks ago we gave a quiz to the SWC students on the first three topics of the course: Version Control, Spreadsheets, and Databases. We did this in addition to our student check-ins in order to get a sense of how the students were doing with the material. Half of the class (24 students) submitted their answers.

Results:
Everyone did really, really well! The majority of the mistakes were small SQL syntax errors, or other similarly minor things.

Summary of student check-ins

Thu, 18 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Over the last two weeks we've had short one-on-one conversations with the current students about how the course is going for them, and what they've enjoyed and haven't. We've heard from 30 students of the ~48 signed up in the course. For most students, their feedback was about the first three lectures (Version Control, Databases, Spreadsheets) and to some extent the Python lectures.

Done In London

Fri, 05 Nov 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We wrapped up this week's class in London today: I think a lot of the students felt that they'd been in the wind tunnel most of the week, but the feedback was fairly positive:

Good

new content (hadn't seen before)
covered a lot (good to see things)
easy to apply / pragmatic
group exercises were fun / useful
Subversion
entertaining instructor (blah blah blah)
better understanding of best practices
persuasive arguments for best practices
nice to meet everybody!
accidental knowledge transfer (vertically and sideways)

Bad

Feedback at UKMO

Thu, 28 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've just finished teaching a four-day version of the course at the UK Met Office in Exeter. I think it went reasonably well, even if we did version control last instead of first :-). Here's the feedback the students gave me:

Good

Most of what the instructor told us was probably true!
Liked the emphasis on the human aspects of software development
Liked the emphasis on evidence-based software engineering
Lots of details on how to do testing
Liked it when the instructor went off on tangents
Liked having pointers to more material
Instructor enthusiastic
Lots of stuff on course web site
Enjoyed learning a new language
Finally had someone explain why to make code readable
Nice mix of concepts and details
Modular course design allowed me to catch up when I got lost

Bad

How We've Helped

Thu, 28 Oct 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We have started to collect testimonials from former students; if you have a story to share, please send it in.

Software Carpentry Usage in December

Thu, 18 Jan 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Visits to the Software Carpentry site were down quite a bit in December; I had expected a drop because of the holiday, but not this large. On the bright side, 47 students have signed up for the course at U of T this term; about 1/3 are from Computer Science, while the rest span Civil Engineering, Ecology, Physics, Ecology, and the life sciences.

11	could do it easily
15	could probably struggle through
3	wouldn't know where to start

12	could do it easily
10	could struggle through
7	wouldn't know where to start