Research on The Carpentries

Two Studies of Online Communities

Fri, 30 Sep 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Two recent papers may be of interest to this community. The first is from Adam Crymble at The Programming Historian a distributed group of digital humanities scholars that has built some excellent tutorials on software tools. Its title is Identifying and Removing Gender Barriers in Open Learning Communities, and its abstract reads:

Open online learning communities are susceptible to gender barriers if not carefully constructed. Gender barriers were identified in The Programming Historian, through an open online discussion, which informed an anonymous user survey. The initial discussion pointed towards two barriers in particular: a technically challenging submission system and open peer review, as factors that needed consideration. Findings are put in context of the literature on gender and online communication, abuse, and online learning communities. The evidence suggests that open online learning communities such as The Programming Historian should work actively to promote a civil environment, and should listen to their communities about technical and social barriers to participation. Whenever possible, barriers should be removed entirely, but when that is not feasible due to financial or technical constraints, alternatives should be offered.

Show Me Your Model

Sun, 18 Sep 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

As far as I can tell, there are no published studies showing that version control is better than mailing files around or sticking them in shared drives. I believe it is–I wouldn’t work on a project that didn’t use version control–but nobody’s ever gathered data, compared it to a model, and published the result.

One reason, I think, is that we don’t know how to measure the productivity of programmers. “Lines of code per hour” clearly isn’t right: good programmers often write less code, or spend their time on the parts of problems that have the highest thinking-to-coding ratio. Without some operationalization of “better” and “worse”, it’s hard to rank or compare alternatives.

Feedback Sought on Two Papers

Fri, 02 Sep 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We would be very grateful for feedback on two papers co-authored by members of our community:

Taschuk & Wilson: “Ten Simple Rules for Making Research Software More Robust”.
Wilson, Bryan, Cranston, Kitzes, Nederbragt, & Teal: “Good Enough Practices for Scientific Computing”.

Each paper has a link at the top to send us email; we look forward to hearing from you.

Heuristic Evaluation for Novice Programming Systems

Tue, 02 Aug 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I have recently been reading and enjoying a new paper by Michael Kölling and Fraser McKay titled “Heuristic Evaluation for Novice Programming Systems”. In it, the authors say:

With the proliferation of competing systems [for novices], the problem [of evaluation ] has become more complicated. Not only should we ask the question whether such kinds of tools are helpful at all (which many instructors strongly believe them to be, even in the absence of hard evidence), but we need to decide which of a significant number of competing systems is “better” for a given task in a given context. Educators have to make choices, not only between using an educational IDE or not, but between a number of direct competitors.

How Well Do Developers Understand Open Source Licenses?

Tue, 02 Aug 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

You are invited to participate in a survey on software licensing designed to investigate how well software developers understand common open source software licenses. We are looking for software developers that have built or are currently building on open source software in their projects (and I am personally interested in hearing from people building open source software for research). The study is being conducted by Prof. Gail Murphy (murphy@cs.ubc.ca) and graduate student Daniel Almeida (daa@cs.ubc.ca); participating in the anonymous online survey will take approximately 30 minutes. If you are interested in participating, please go to:

Correlations

Mon, 08 Feb 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We've run instructor training both online and in person for several years, so it's time to look at how they compare. The raw data shows:

the event's start date
its unique identifier (which we call a "slug")
whether it was online
how many people took part
how many completed training after this course (both as an absolute number and as a percentage)
how many completed after taking a later course (in both forms)
how many have never completed
how many have taught at least once since taking the course

The plot below then shows completion rates and follow-through teaching rates (as percentages) versus cohort size, tagged by whether the training event was online or in person, for all events that took place at least one year ago. (I've used that cutoff to give participants a fair chance to teach after completing their training.) It also shows the absolute number of participants and the follow-through teaching percentage by date.

A New Version of 'Lessons Learned'

Thu, 28 Jan 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A new version of "Software Carpentry: Lessons Learned" is [now available on F1000](http://f1000research.com/articles/3-62/v2). We think it's an interesting complement to the [instructor survey]({{site.baseurl}}/blog/2016/01/instructor-survey.html), and we hope you enjoy them both.

Change Strategies in STEM Education

Sat, 09 Jan 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I recently had a chance to read:

Maura Borrego and Charles Henderson: "Increasing the Use of Evidence-Based Teaching in STEM Higher Education: A Comparison of Eight Change Strategies". *Journal of Engineering Education*, 103(2), DOI 10.1002/jee.20040.

The abstract says:

Background Prior efforts have built a knowledge base of effective undergraduate STEM pedagogies, yet rates of implementation remain low. Theories from higher education, management, communication, and other fields can inform change efforts but remain largely inaccessible to STEM education leaders, who are just beginning to view change as a scholarly endeavor informed by the research literature.

What the Data Says About Novice Programming Mistakes

Sat, 09 Jan 2016 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I recently had a chance to catch up with this paper from 2014:

Neil C. C. Brown and Amjad Altadmri: “Investigating Novice Programming Mistakes: Educator Beliefs vs Student Data”. ICER'14, http://dx.doi.org/10.1145/2632320.2632343.

Its abstract says:

Educators often form opinions on which programming mistakes novices make most often – for example, in Java: “they always confuse equality with assignment”, or “they always call methods with the wrong types”. These opinions are generally based solely on personal experience. We report a study to determine if programming educators form a consensus about which Java programming mistakes are the most common. We used the Blackbox data set to check whether the educators’ opinions matched data from over 100,000 students – and checked whether this agreement was mediated by educators’ experience. We found that educators formed only a weak consensus about which mistakes are most frequent, that their rankings bore only a moderate correspondence to the students in the Blackbox data, and that educators’ experience had no effect on this level of agreement. These results raise questions about claims educators make regarding which errors students are most likely to commit.

Visualizing Repository Activity

Tue, 27 Oct 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I am updating the lessons learned paper, and would like to include histograms showing how many people have contributed how often to our lessons. More specifically, I have 9 data sets (one for each lesson), ranging in size from 5 to 16 records, in which each record shows a number of commits and how many people have committed that often. For example, the data for our SQL lesson is:

Software Engineering Practices in Science

Wed, 16 Sep 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Dustin Heaton and Jeffrey Carver have just published a paper titled Claims About the Use of Software Engineering Practices in Science: A Systematic Literature Review:

How Teaching Knowledge Is Transferred

Sun, 13 Sep 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I hesitate to say so, but I believe that we pedagogues tend to exaggerate greatly the amount of change in educational practice that results from reading what other people say should be done...

— Stephen Corey, 1951

Being back in Edinburgh thirty years on has occasioned much reflection upon lessons learned. This has occasioned a re-reading of some papers by Prof. Sally Fincher, whose research group at the University of Kent studies the teaching and learning of computer science. In particular, I have been looking at what they have discovered about how educators share teaching practices. I hope these excerpts and reflections are of interest. (Note: section titles link to papers.)

Announcing ReScience

Thu, 03 Sep 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It's our great pleasure to announce the creation of "ReScience" which is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible.

Better Teaching Practices

Tue, 01 Sep 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It wasn't part of our original plan, but over time Software Carpentry has come to be about better teaching as much as it is about better computing. In aid of that, I would like to offer the following:

Three Graphs I Would Like to See

Mon, 31 Aug 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I spent part of the weekend chatting with a friend in Cambridge who used to be science editor at The Independent and now edits Scientific Computing World. During those conversations, I realized that there are three graphs I'd really like to see:

Top 10 Myths about Teaching CS

Sat, 18 Jul 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Mark Guzdial (whose blog has been a frequent inspiration) recently wrote an article title Top 10 Myths about Teaching Computer Science:

The lack of women in Computer Science is just like all the other STEM fields.
To get more women in CS, we need more female CS faculty.
A good CS teacher is a good lecturer.
Clickers and the like are an add-on for a good teacher
Student evaluations are the best way to evaluate teaching.
Good teachers personalize education for students' learning styles.
High schools just can't teach CS well, so they shouldn't do it at all.
The real problem is to get more CS curriculum out into the hands of teachers.
All I need to do to be a good CS teacher is model good software development practice, because my job is to produce excellent software engineers.
Some people are just born to program.

The Open Research Glossary

Thu, 16 Jul 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

It's been knocking around for a while now, but this week saw a big new release of the Open Research Glossary, a crowdsourced glossary of terms, acronyms, tools and concepts in and around open science. There's a lot of jargon in this area and it's often a barrier to understanding for the uninitiated. For example: 'green', 'gold', 'diamond' and 'hybrid' open access are shorthand terms which aren't always fully understood.

The alphabet soup of acronyms are even worse! Take SHERPA/RoMEO for instance: it's a brilliant resource for checking author self-archiving rights relative to publisher-imposed restrictions and embargoes, but the name 'SHERPA/RoMEO' doesn't exactly make that clear. This glossary aims to elucidate all the need-to-know terms in open scholarship.

Research-Based Course Design

Sun, 21 Jun 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I've written before about the breadth and depth of Juha Sorva's work on computing education. His latest contribution is a paper co-authored with Otto Seppälä titled "Research-Based Design of the First Weeks of CS1". In it, they tie the specifics of the new intro programming class at Aalto University directly back to CS education research. More specifically (from their abstract):

Software Development Practices in Academia

Thu, 18 Jun 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Derek Groen, Xiaohu Guo, James Grogan, Ulf Schiller, and James Osborne have just submitted a paper to arXiv.org titled "Software development practices in academia: a case study comparison". From the abstract:

Academic software development practices often differ from those of commercial development settings, yet only limited research has been conducted on assessing software development practises in academia. Here we present a case study of software development practices in four open-source scientific codes over a period of nine years, characterizing the evolution of their respective development teams, their scientific productivity, and the adoption (or discontinuation) of specific software engineering practises as the team size changes. We show that the transient nature of the development team results in the adoption of different development strategies. We relate measures of publication output to accumulated numbers of developers and find that for the projects considered the time-scale for returns on expended development effort is approximately three years. We discuss the implications of our findings for evaluating the performance of research software development, and in general any computationally oriented scientific project.

Get More Done in Less Time

Wed, 17 Jun 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Over the past year, Alexandra Simperler has interviewed participants in Software Carpentry workshops to find out what impact we've actually had on their work. Her results are now available on arXiv.org:

The aim of this study was to investigate if participants of Software Carpentry (SC) get more done in less time. We asked 32 questions to assess 24 former participants to analyse if SC gave them the computing skills to accomplish this. Our research shows that time was already saved during the workshop as it could shorten the learning process of new skills. A majority of participants were able to use these new skills straight away and thus could speed up their day to day work.

Like Jory Schossau's study, Alexandra's work shows that workshop participants believe we're making their lives better.

Call for Chapter Proposals: Software Engineering for Science

Wed, 10 Jun 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

With the continuing increase in the importance and prevalence of software developed in support of science, there is a need to gather a set of best practices and case studies to serve as a standard reference book. We are producing a peer-reviewed, edited book to address this need. The book will be composed of chapters related to one of three topics that address the important needs. The book will have three sections, each related to one of those topics. The outline below enumerates those sections along with examples of the types of chapters that would fit within those sections. We solicit proposals from interested authors. Chapter proposals should fit into one of the following book sections.

A Few Articles on Education

Mon, 25 May 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Over the past year, I've come to realize that Software Carpentry will only work if knowledge flows in several directions. Scientists need to learn about software development, but software developers need to learn about science, too. In particular, they need to learn that it's possible to study software and programming scientifically, which is what motivated yesterday's post about my favorite papers from ICSE 2015.

And both groups need to learn about evidence-based teaching practices and the politics that education is embedded in (because without an understanding of the latter, no change is possible). While I don't have a snapshot like ICSE to offer, here are a few recent articles I've found illuminating:

ICSE 2015

Sun, 24 May 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Back when I was still trying to do science myself, my field of study was software engineering. The International Conference on Software Engineering is the big gathering for researchers in that area, and this year's has just wrapped up. Thanks to this Gist from Mike Hoye, I was able to browse the papers presented at ICSE and co-located workshops (like him, I'm outside the Great Paywall of Academia), and I've included titles and abstracts below from the ones I think readers of this blog might enjoy. They're only a fraction of what was presented, and I freely admit the sample is biased toward the things I understand and find interesting, but I hope they'll convince you that people are doing solid empirical studies in software engineering, and producing insights that we can and should act on.

Note: just over half of these papers (13 of 24) had an easily-findable version online. I'm not going to do the experiment, but I confidently predict that those 13 will be more widely read, and more influential, than the other 11.

Online Scientific Collaboration: The Sequel

Thu, 12 Feb 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Jon Udell's Internet Groupware for Scientific Collaboration taught me how to think about the web. He's now revisiting that report, and would like our help. Details are below; please give him a shout if you can help.

Rewarding Software Sharing by Mapping Scientific Software

Sat, 07 Feb 2015 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Sharing software you write with other scientists can magnify the impact of your research, but there can be a surprising amount of sometimes thankless extra work involved. I work with a group at Carnegie Mellon's Institute for Software Research who have been asking scientists what that extra work is, and what motivates them to do it—despite a sometimes uncertain link between that extra work and the ways many of them are evaluated in their jobs. We're looking at ways of measuring and mapping software and its impacts, in order to help scientists demonstrate the positive impact that their work on shared software has on science. We're running an experiment that you can help with.

Studying Impact

Sat, 04 Oct 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I am a Software Sustainability Institute Fellow, and am using my fellowship to work on making software training better. My main interest is in teaching computational chemistry software and making it accessible to a wide range of scientists. Like Software Carpentry, the courses I teach are one-off teaching events: basically, we have a maximum of two days to help your professional development.

Together with Greg Wilson, I am conducting research to analyse what impact Software Carpentry workshop have had on participants with respect to their professional skills. In particular, I would like to interview past participants in Software Carpentry workshops to analyse the impact those workshops have had on them. We cannot reimburse you for your time, but if you are willing to take part in an interview, your experience will help us make Software Carpentry better. The interviews can be done via telephone or Skype and can be arranged at a convenient time; we have ethics approval for this study, and every participant will be informed what exactly we do with their data.

Please contact me by email if you wish to take part, and pass on this request to other workshop alumni.

Thanks in advance,
Alexandra Simperler

Our IUSE Proposal Was Rejected

Fri, 27 Jun 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We got word a few days ago that our proposal to the NSF's Improving Undergraduate STEM Education program had been rejected. The panel summary agreed that software training is a good idea, but were not convinced about our plans to shift from training grad students to undergrads. In particular, they were not convinced that Unix-based workshops would be best for undergrads, and felt that not being embedded in the regular curriculum was a weakness. These are fair criticisms:

Most undergraduates only use GUIs and cannot navigate the terminal to save their lives. The perception that Software Carpentry is Unix-specific therefore hurt the proposal, and showed that we didn't clearly explain our focus on underlying concepts.
The fact that we wouldn't be part of the regular curriculum is more difficult to address. The fact is, it really is hard to get undergrads to do things that interfere with earning grades, just as it's often hard to get grad students to do anything that doesn't immediately lead to a publication. Some of their concern about impact also seemed to be due to us concentrating on REU students, a self-selected bunch that already are motivated.

At the same time, though, the panel did not connect the results from years of Software Carpentry workshops and the expected impact of this effort. This indicates that the proposal did not effectively communicate how well our experience to date has laid the groundwork for efforts like this.

We're obviously disappointed by this rejection, but we've some useful lessons, and we hope that they will inspire others to put forward proposals of their own.

Data Science Study Invitation

Sat, 24 May 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Katie Kuksenok, a graduate student at the University of Washington, is interviewing academic researchers who do data science to explore the barriers and challenges they face. If you would be willing to take 20-30 minutes to be interviewed, please get in touch: among other things, her findings will be used to help improve curriculum for projects like Software Carpentry.

Empirical Software Engineering Papers

Wed, 19 Mar 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

When I teach scientists programming, I frequently cite empirical studies in software engineering to back up my claims about various tools and practices making people more productive. No good, short survey of those papers exists—writing one has been on my to-do list for several years—but I hope the pointers below will be a useful substitute.

Lessons Learned Has Been Published

Wed, 19 Feb 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I'm pleased to announce that "Software Carpentry: Lessons Learned" has been published on F1000Research. A paper like this is necessarily incomplete, and so is any acknowledgments list, but I'd like to thank the following for their feedback:

Code Review, Round 2

Tue, 14 Jan 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Short version: we will launch a second pilot study of code review for scientific software in February 2014, in which experienced mentors will teach small groups of scientists how and when to do reviews. We are currently looking for both mentors and groups to take part.

It's Not Just the Tools that Differ Between the Two Cultures

Fri, 10 Jan 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

In the conversation that followed Philip Guo's blog post on the two cultures of computing, most on focused comparing the tools used by those two cultures, as discussed by Philip to create a contract. Some even wondered if Software Carpentry instructors should perhaps become more versed in MS Excel to meet potential students where they are. In so doing, we lost the fact that even with the same tools, the two cultures would likely use them differently. It's not about the tools, but about the users.

Mental Models and Vicious Circles

Tue, 07 Jan 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A few days ago, Konrad Hinsen asked this on our discussion list:

Is anyone aware of teaching methods that aim at either developing or verifying the students' mental model of some non-trivial procedure? If so, have they been used for teaching the use of computers?

Test-Driven Development in Scientific Computing

Tue, 07 Jan 2014 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Aziz Nanthaamornphong and Dr. Jeffrey Carver at the University of Alabama are conducting a survey of software development practices among computational scientists. This survey seeks to understand test-driven development in computational science and engineering domains, and should take approximately 15 minutes to complete. The survey can be found at:

http://goo.gl/9T2BIq

This survey study has been approved by the University of Alabama Institutional Review Board (IRB); for questions contact Jeffrey Carver.

Report on the PLOS/Mozilla Code Review Pilot

Tue, 12 Nov 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Marian Petre and I have written a report on the pilot study of scientific code review done jointly by PLOS and Mozilla; you can read this summary on the Mozilla Science Lab blog, or grab the full report from arXiv. Our thanks to everyone who participated; we hope to be able to do a longer and more in-depth pilot early in 2014.

Software Carpentry: Lessons Learned

Thu, 20 Jun 2013 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

With contributions from:

Azalee Bostroem (Space Telescope Science Institute)
Chris Cannam (Queen Mary, University of London)
Stephen Crouch (Software Sustainability Institute)
Matt Davis (Space Telescope Science Institute)
Luis Figueira (Queen Mary, University of London)
Richard "Tommy" Guy (Wave Accounting)
Edmund Hart (University of British Columbia)
Neil Chue Hong (Software Sustainability Institute)
Katy Huff (University of Wisconsin)

Michael Jackson (Edinburgh Parallel Computing Centre)
W. Trevor King (Drexel University)
Justin Kitzes (University of California, Berkeley)
Stephen McGough (University of Newcastle)
Lex Nederbragt (University of Oslo)
Tracy Teal (Michigan State University)
Ben Waugh (University College London)
Lynne J. Williams (Rotman Research Institute)
Ethan White (Utah State University)

Abstract

Over the last 15 years, Software Carpentry has evolved from a week-long training course at the US national laboratories into a worldwide volunteer effort to raise standards in scientific computing. This article explains what we have learned along the way the challenges we now face, and our plans for the future.

Best Practices for Scientific Computing

Wed, 03 Oct 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The following pre-print is now available on arXiv:

Best Practices for Scientific Computing

D.A. Aruliah^a, C. Titus Brown^b, Neil P. Chue Hong^c, Matt Davis^d, Richard T. Guy^e, Steven H.D. Haddock^f, Katy Huff^g, Ian Mitchell^h, Mark Plumbleyⁱ, Ben Waugh^j, Ethan P. White^k, Greg Wilson^l, and Paul Wilson^g.

^aUniversity of Ontario Institute of Technology, ^bMichigan State University, ^cSoftware Sustainability Institute, ^dSpace Telescope Science Institute, ^eUniversity of Toronto, ^fMonterey Bay Aquarium Research Institute, ^gUniversity of Wisconsin, ^hUniversity of British Columbia, ⁱQueen Mary University London, ^jUniversity College London, ^kUtah State University, and ^lSoftware Carpentry.

Linking Forward From a Bibliography?

Wed, 29 Aug 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This web site's bibliography lists 116 papers related in some way to the practice of scientific computing. I'd like to know:

what related papers I've missed, and
when a new related paper appears.

It seems to me that I ought to be able to upload the entire bibliography into some tool, then say, "Find everything that isn't in this list that cites things that are, minus everything in this other list over here that I've already looked at and decided aren't relevant." Going a step further, I ought to be able to prime some kind of robot to do this check automatically every week or so, and mail me when there's a hit so that I can either read it or add it to the exclusion list.

SoundSoftware 2012: Workshop on Software and Data for Audio and Music Research

Wed, 30 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The SoundSoftware project — a UK based project aiming towards software sustainability in the Audio and Music Research community — is organising a one day workshop on "Software and Data for Audio and Music Research". The aim of this workshop is to discuss issues such as robust software development for audio and music research, reproducible research, management of research data, and open access.

No CT Without PL

Thu, 24 May 2012 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

In a blog post earlier today, Mark Guzdial argues that computational thinking requires learning with a programming language. Unlike many such claims and counter-claims, his is based on a wealth of research, most recently an excellent dissertation by Juha Sorva. I strongly agree with Mark's position: our real goal in Software Carpentry is to teach computational thinking, but the only way to do that successfully is to teach some basic programming skills and use those to convey larger ideas. If we're successful in getting follow-on funding at the end of our current grant, we're going to work hard to re-cast everything in this way.

Yet Another Survey

Thu, 29 Dec 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Prakash Prabhu and others recently published "A Survey of the Practice of Computational Science" based on information from 114 researchers at Princeton University. The emphasis is different from that of the survey Hannay and others (including me) did in 2008-09, but the findings are broadly similar.

Building a Bibliography

Fri, 25 Nov 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

With help from several of our regular readers, we have assembled a bibliography of research related to software engineering and computational science to go with our recommended reading list for students. We hope you find it useful, and we would welcome corrections and additions.

Show Me the Data

Fri, 18 Nov 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

I got mail from a colleague at a prominent US university yesterday saying (in part, and elided to protect the guilty):

...the graduate student representative to the curriculum committee reported that the students did not want a scientific computing course, that they would instead figure it out themselves.... How does one respond to statements like this...that have...basically frozen skill levels? The options I see are formal ("in curriculum") training, bootcamps and /workshops, and letting them "figure it out themselves". Are there arguments about the successes of each?

Call for Participation

Tue, 15 Mar 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Are you a computational scientist (but not a computer scientist) who develops scientific software? That is, do you use mathematical models to describe scientific processes and then implement these models in your software?

Is your main reason for software development to advance science?

Is your software used by a wider community of scientists (and maybe not only scientists)?

Would you be willing to spare about one hour of your time to participate in a PhD study?

Research Study: How Do You Test Your MATLAB?

Thu, 27 Jan 2011 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Have you ever wondered how scientists test their code? We have, and we'd like you to help us find out. If you use MATLAB, please have a look at our new research project—and please forward the link to other groups and lists where we might be able to recruit people. (Of course, we'd also welcome input from NumPy users...)

See also these posts from Steve Eddins, who built the MATLAB xUnit testing framework, and thanks in advance for your help.

Open Research Computation

Mon, 27 Dec 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

By now, many of you have (hopefully) seen the announcement of Open Research Computation, a new journal devoted to "...peer reviewed articles that describe the development, capacities, and uses of software designed for use by researchers in any field." The editorial board includes several friends of this course; as one of them, Titus Brown, observed in his blog:

...the problem with the online world for scientists [is] there's no real systematized incentive to any of this online stuff. And that makes it really tough. I'm going through Reappointment right now... Nowhere on there is there a place for "influential blog posts" — how would you measure that, anyway? Same with software — I listed my various software releases on the "scientific products" page of the form, and have since been asked to describe and discuss the impact of my software. Since I don't track downloads, and half or more of the software hasn't been published yet and can't easily be cited, and people don't seem to reliably cite open source software anyway, I'm not sure how to document the impact.

Survey: Help Needed

Thu, 16 Sep 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Dr. Roscoe Bartlett (Sandia National Laboratory), Dr. Jeffrey Carver (University of Alabama), and Dr. Lorin Hochstein (University of Southern California) are conducting a survey of software development practices among computational scientists. This survey seeks to understand current software development practices and identify areas of need. The goal is to produce a report on the status of scientific software development which can serve as the basis for future work in this area. The survey should take approximately 15 minutes to complete.

Survey Update

Thu, 29 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Here's an update on responses to the survey I posted a couple of weeks ago. 172 people have responded at this point; it's encouraging that priorities are relatively stable as numbers increase.

Education
77.3%		Graduate degree
22.1%		Undergraduate degree
0.6%		High school
Field
41.0%		Computer Science
30.1%		Earth Sciences
28.9%		Physics
25.4%		Mathematics and Statistics
11.0%		Microbiology
9.2%		Biomedical Engineering
6.9%		Macrobiology
5.2%		Medicine and Health Care
5.2%		Electrical Engineering
5.2%		Astronomy
4.6%		Mechanical Engineering
4.6%		Aerospace Engineering
4.0%		Chemical Engineering
2.9%		Psychology
2.3%		Economics
2.3%		Business/Finance
1.2%		Linguistics
1.2%		Civil Engineering
0.6%		Social Sciences
0.6%		Arts and Humanities
Role
44.8%		Academic Researcher
32.8%		Software Developer
16.7%		Graduate Student
16.7%		Government Research Scientist
10.3%		Engineer
9.8%		Manager/Supervisor
8.6%		System Administrator
3.4%		Teacher
2.9%		Industrial Research Scientist
1.1%		Undergraduate student
1.1%		Laboratory Technician
Priorities
2.51		Automating Repetitive Tasks
2.50		Reproducible Research
2.49		Data Visualization
2.46		Version Control
2.43		Performance Optimization
2.41		Data Structures
2.41		Coding Style
2.38		Basic Programming
2.37		Testing and Quality Assurance
2.35		Parallel Programming
2.34		Debugging with a Debugger
2.33		Using the Unix Shell
2.29		Computational Complexity
2.21		Object-Oriented Programming
2.21		Designing a Data Model
2.19		Working in Teams/on Large Projects
2.14		Refactoring
2.10		Static and Dynamic Code Analysis Tools
2.09		Matrix Algebra
2.06		Systems Programming
2.06		Integrating with C and Fortran
2.03		Design Patterns
2.01		Packaging Code for Release
1.95		Functional Languages
1.93		Handling Binary Data
1.80		Image Processing
1.77		Introduction
1.75		Build a Desktop User Interface
1.73		XML
1.64		Create a Web Service
1.39		Geographic Information Systems

Survey Results

Thu, 15 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Here are the results of the survey that we announced a couple of days ago. I'm a bit surprised that so many computer scientists responded, and equally surprised by the popularity of "biomedical engineering" — who knew? The scores for various topics hold a few surprises as well: I would have predicted that something with the word "web" in it would have scored near the top of the list, rather than at the bottom.

HPC and Programmability

Sat, 10 Jul 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Via Andrew Lumsdaine, a pointer to an interesting article in Communications of the ACM by Eugene Loh titled "The Ideal HPC Programming Language". A few key quotes:

These programmability studies began with a focus on programming languages, but the focus quickly shifted to other topics. Existing languages—notably Fortran...—proved remarkably adequate. Programming challenges stem mostly from other factors.

The DARPA HPCS program...sponsored the development of new programming languages: Chapel from Cray, Fortress from Sun, and X10 from IBM. Proponents of those languages would show early on how rewriting familiar HPC benchmarks in the new languages could reduce source-code volume substantially—tenfold reductions were not surprising—but rewriting these benchmarks even in Fortran achieved similar source-code reductions and corresponding improvements in expressivity.

The Cowichan Problems

Sat, 12 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Back in the 1990s, as the first wave of euphoria about parallel computing was topping out, I had a crazy idea: why don't we actually try to measure, or at least compare, the usability of different parallel programming systems? I left the field before taking the idea very far, but with talk about clouds and GPUs growing louder by the day, I think the idea is worth revisiting. My proposal is below the cut; I'd be interested in feedback.

Who Reports On The Other 97 Per Cent?

Tue, 01 Jun 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The BBC has posted a nice graphic of the latest Top 500 list of supercomputers around the world. It's pretty impressive, particularly if you're a Linux fan (check out the display by operating system). However, nobody would think that a list of the world's 500 richest people told them much about the state of the economy; I'd be much more interested in statistics on how much computing power is employed by the 97% or so of scientists who don't use a supercomputer.

A Question About Documentation

Wed, 05 May 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

What kind of documentation do you use when you're programming? How useful do you find it? If have three minutes to fill in a very short survey on the topic (it's literally half a dozen questions), we'd be very grateful for your feedback.

Measuring Science

Sun, 11 Apr 2010 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Julia Lane, the director of the Science of Science & Innovation Policy program at the National Science Foundation, wrote an article for Nature a couple of weeks ago titled "Let's make science metrics more scientific". As the summary at the start says:

Existing metrics have known flaws
A reliable, open, joined-up data infrastructure is needed
Data should be collected on the full range of scientists' work
Social scientists and economists should be involved

The same points could be made about evaluating software developers (or any other kind of knowledge worker). The devil, as always, is in the details, and unfortunately I have to start doing evaluations before those details are worked out. Several of the supporters for this course need me to demonstrate its impact on the productivity of the scientists who take it (so that they can in turn justify their contribution to their funders). It isn't enough to ask students who have completed the course whether they think they know more about programming than they used to: ignoring the obvious problems of survivor bias and self-assessment, I would still have to demonstrate that making people better programmers also makes them better scientists. I believe it does, but belief is not evidence, and doesn't convey scale.

Special Issue of Computing in Science and Engineering

Wed, 18 Nov 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

A special issue of Computing in Science & Engineering that Andy Lumsdaine and I edited, devoted to software engineering in computational science, is now available. We'd like to thank everyone who contributed:

Report on the Second International Workshop on Software Engineering for CSE, by Jeffrey Carver (University of Alabama)
Managing Chaos: Lessons Learned Developing Software in the Life Sciences, by Sarah Killcoyne and John Boyle (Institute for Systems Biology)
Scientific Computing's Productivity Gridlock: How Software Engineering Can Help, by Stuart Faulk (University of Oregon), Eugene Loh and Michael L. Van De Vanter (Sun Microsystems), Susan Squires (Tactics), and Lawrence G. Votta, (Brincos)
Mutation Sensitivity Testing, by Daniel Hook (Engineering Seismology Group Solutions) and Diane Kelly (Royal Military College of Canada)
Automated Software Testing for MATLAB, by Steve Eddins (The MathWorks)
The libflame Library for Dense Matrix Computations, by Field G. Van Zee, Ernie Chan, and Robert A. van de Geijn (University of Texas at Austin), and Enrique S. Quintana-Ortí and Gregorio Quintana-Ortí (Universidad Jaime I de Castellón)
Engineering the Software for Understanding Climate Change, by Steve Easterbrook (University of Toronto) and Timothy Johns (Hadley Centre for Climate Prediction and Research)

American Scientist Article on How Scientists Use Computers

Thu, 06 Aug 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

American Scientist has just published a short article summarizing the results from the survey we did last year of how scientists actually use computers.

SECSE Workshop

Mon, 01 Jun 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The latest in a series of workshops on "Software Engineering for Computational Science and Engineering" was held in Vancouver on May 23, just after ICSE'09. Steve Easterbrook has written a good summary of what was discussed, and Jeffrey Carver's longer summary will appear in a future issue of Computing in Science and Engineering.

How Scientists Use Computers: Survey Part 2

Sat, 09 May 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Thank you once again for taking part in our Fall 2008 survey of how scientists use computers in their research. We will present a paper describing our findings at ICSE'09 in Vancouver on May 23, and will make the results public as soon after that as possible. There will also be an article in American Scientist magazine discussing what you've told us some time this summer.

Empirical Software Engineering and Scientific Computing

Tue, 28 Apr 2009 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

The slides for my talk at the National Research Council on empirical software engineering and how scientists actually use computers are now up on SlideShare. The colors in some of the embedded images were messed up during upload, but the result should still be readable.

SECSE'09 Call for Papers

Fri, 21 Nov 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Second International Workshop on Software Engineering for Computational Science and Engineering

Saturday, May 23, 2009

Co-located with ICSE 2009 — Vancouver, Canada

http://www.cs.ua.edu/~SECSE09

Overview

This workshop is concerned with the development of:

Scientific software applications, where the focus is on directly solving scientific problems. These applications include, but are not limited to, large parallel models/simulations of the physical world (high performance computing systems).
Applications that support scientific endeavors. Such applications include, but are not limited to, systems for managing and/or manipulating large amounts of data.

A particular software application might fit into both categories (for example, a weather forecasting system might both run climatology models and produce visualisations of big data sets) or just one (for example, nuclear simulations fit into the first category and laboratory information management software into the second). For brevity, we refer to both categories under the umbrella title of "Computational Science and Engineering (CS&E)".

What Sciences Are There?

Sun, 16 Nov 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Over 1900 people have already responded to our survey of how scientists use computers, and it still has two weeks left to run. Our next task will be to analyze the data we've collected, which (among other things) means coding people's free-form descriptions of their specialties so that we can talk about physicists and chemists as opposed to "this one person who's doing N-brane quantum foam approximations to multiversal steady-state thingummies".

One Good Survey Deserves Another

Tue, 04 Nov 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

While we're running our survey of ~~how scientists use computers~~ [link no longer active], the folks at MATLAB are asking their users a few questions too. If you use any MathWorks products, and have a few minutes, they'd be grateful for your help.

1731 People

Sun, 02 Nov 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

1731 people have completed our survey of how scientists use computers since it went online three weeks ago. That's pretty cool, but I'd like to double the number (at least). If you consider yourself a working scientist, and haven't taken the survey yet, please take a moment and do so. If you aren't a scientist, but know some, please pass on the link:
~~http://softwareresearch.ca/seg/SCS/scientific-computing-survey.html~~
[link no longer active]

Surveying Scientists' Use of Computers

Wed, 15 Oct 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Computers are as important to modern scientists as test tubes, but we know surprisingly little about how scientists develop and use software in their research. To find out, the University of Toronto, Simula Research Laboratory, and the National Research Council of Canada have launched an online survey in conjunction with American Scientist magazine. If you have 20 minutes to take part, please go to:

~~http://softwareresearch.ca/seg/SCS/scientific-computing-survey.html~~ [link no longer active]

Quick Quiz to Measure What Scientists Know

Wed, 23 Jul 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Suppose you have a room full of scientists—hundreds of 'em—and want to find out how they actually use computers in their work. There isn't time to interview them individually, or to record their desktops during a typical working week, so you've decided to ask them to self-assess their understanding of some key terms on a scale of:

No idea what it is.
Use it/have used it infrequently.
Use it regularly.
Couldn't get through the day without it.

My list is below; what have I forgotten, and (more importantly) how would you criticize this assessment method?

SE-CSE Workshop

Thu, 15 May 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

One of the downsides of being in Texas is that I couldn't attend the First International Workshop on Software Engineering for Computational Science and Engineering, which was held at ICSE'08 in Leipzig this week. Papers are here (I'll be reading them on the flight home); they look interesting, but the biggest thing for me is the change in the workshop's name—it used to be "Software Engineering for High-Performance Computing Applications" (~~2004~~, ~~2005~~, 2006, 2007), and I'm hoping the change of name reflects a genuine broadening of focus.

Three Studies (Maybe Four)

Thu, 10 Apr 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

We're in the thick of picking students and projects for Google Summer of Code, which has inspired some less-random-than-usual thoughts. Here are two studies I'd like to do (or see done):

What has happened to previous students? How many are still involved in open source? How many have gone on to {start a company, grad school, prison}? What do they think they learned from the program? How much of the software they wrote is still in use? Etc.
Every one of the 175 organizations blessed by Google this year is using the same web application for collecting and voting on projects. From what I can tell, they're all using it in different ways: +4 means something very different to the Python Software Foundation than it does to Eclipse or SWIG. They're also using a bewildering variety of other channels for communication: wikis, IRC, Skype chat sessions, mailing lists (the most popular), and so on. Why? Is this another reflection of Jorge Aranda's finding that every small development group evolves a different process, but all those processes "work" in some sense, or is it—actually, I don't have any competing hypotheses right now, but I'm sure there are some.

And while we're on the subject of studies, I just read Hochstein et al's paper "Experiments to Understand HPC Time to Development" (CT Watch Quarterly, 2(4A), November 2006). They watched a bunch of grad students at different universities develop some simple parallel applications using a variety of tools, and measured productivity as (relative speedup)/(relative effort), where relative speedup is (reference execution time)/(parallel execution time), and relative effort is (parallel effort)/(reference effort). The speedup measure is unproblematic, but as far as I can tell, they don't explain where their "reference effort" measure comes from. I suspect it's the effort required to build a serial solution to the problem, and that "parallel effort" is then the additional time required to parallelize; I've mailed the authors to ask, but haven't heard back yet.

Survey: Silent Errors in Scientific Code

Fri, 07 Mar 2008 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Posted on behalf of Daniel Hook and Diane Kelly:

We are members of a software research group from Queen's University who are investigating ideas and tools to assist with the development of scientific software. We are starting a project focused on finding silent or hidden errors in scientific code. (Silent errors are errors that don't result in a crash, an error message or an other obvious indicator of a problem.) To create a catalogue of common silent errors, we would like to hear debugging "war stories" from computational scientists. Using these stories we hope to provide improved code testing techniques specifically for scientists.

Random Survey about HPC

Fri, 31 Aug 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

This was just forwarded by a friend:

The High Performance Computing (HPC) community is initiating a study to develop education and training pathways in order to enable scientists and engineers to advance the pace of discovery by taking advantage of high performance computing (HPC) and grids.

In order to inform our study, we invite you to complete a survey about your experiences. We would like you to evaluate a list of categories of expertise according to how relevant they are for your work. We also would like to collect additional information about your training experiences.

Two Studies of ASCI (and no, that's not a typo)

Wed, 27 Jun 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

After spending ten years helping scientists write programs for massively-parallel computers, I realized that what scientists really needed was to learn how to program, full stop. It took me another eight years to (a) get up to speed with the theory and practice of modern software engineering, (b) realize how big the gap between the two was, and (c) accept that the only people who are entitled to have an opinion about how we ought to be building software are the ones who are studying how well their favorite tools and practices actually work in the field. I no longer care what you're pushing—formal methods and cleanroom development or agile adhocracy and pair programming—unless you've gone the extra mile and collected data to show what effect it's actually having.

Nature Precedings

Mon, 18 Jun 2007 00:00:00 +0000

This post originally appeared on the Software Carpentry website.

Tim O'Reilly posted a note from Timo Hannay, of Nature magazine, about their new "Precedings", which:

giv[es] researchers a place to post documents such as preprints and presentations in a way that makes them globally visible and citable. Submissions are filtered by a team of curators to weed out obviously inappropriate material, but there's no peer-review so accepted contributions appear online very quickly — usually within a couple of hours.