The Commercial Haskell SIG members want to help people adopt Haskell. What would help? Data beats speculation, so FP Complete recently emailed surveys to over 16000 people interested in Haskell. The questions were aimed at identifying needs rather than celebrating past successes, and at helping applied users rather than researchers.
Over 1240 people sent detailed replies, spending over 250 person-hours to provide their answers.
This rich data set includes extensive information on Haskell user needs. We are open-sourcing the entire anonymized data set, downloadable by clicking here [.zip]. There are numeric ratings and extensive textual comments. Feel free to analyze the data — or just read the summaries — and share your most helpful and actionable conclusions with the community. We will too.
Although we have completed only basic analysis, here are some of our first findings — those so clear that they show up on even the most basic examination of the aggregate data.
We have started deeper statistical analysis of the data, and we hope that some readers of this post will perform — and share — even better analyses than we can. New issues may become clearer by clustering or segmenting users, or through other statistical techniques. Also, we may find more clarity about needs through deeper study of textual responses. Follow-up studies are also a possibility.
We propose that the community, given this large and detailed data set, should set some of its priorities in a data-driven manner focused on user-expressed needs. This effort should be complementary to the ongoing research on issues of historic Haskell strength such as language design and runtime architecture.
We request that useful findings or insights derived from the open-sourced data set be shared with the community, including attribution of the source of the data and the methods of analysis used.
The data collected strongly invites clustering or segmentation, so as to identify needs of different sub-populations. FP Complete has already begun one such study.
The data collected includes extensive textual remarks which should be studied by knowledgeable people for insights. Automated text analysis methods may also be applicable.
Cost-benefit analysis seems worthwhile: based on identified needs, what improvements would help the most people, to the greatest degree, at the least cost and/or the least delay? A method to match volunteer contributors with identified high-payoff projects also seems worthwhile.
It would be useful to merge the data from versions 0.1 and 0.2 with version 1.0 of the survey, since 1.0 includes only 71% of the total answers received. Differences between the questions and scales make this a nontrivial, but still realistic, goal.
If important new hypotheses require testing, or if further detail is needed, we intend to conduct follow-up research at some future date among users who volunteered their email addresses for follow-up questions.
A future repeat survey could determine which improvement efforts are successful.
This was not a survey of the general public, but of a population motivated to provide feedback on Haskell. Invitees included 16165 non-opted-out email addresses gathered from FP Complete’s website, in randomized order. Due to privacy considerations this list will not be published, but FP Complete was able to use it to contact these users since the survey was directly related to their demonstrated interest in Haskell. The high quality of the list is reflected in the extremely high response rate (7.7%), the low bounce rate (1.9%), and the low unsubscribe rate (also 1.9%).
Surveys were conducted using SurveyGizmo.com, with an email inviting each participant to click a link to a four-page Web-based survey. Survey form 0.1 invitations went to 1999 users of whom 190 completed the survey. Survey form 0.2, incorporating some edits, went to 2000 users of whom 170 completed the survey. Survey form 1.0, incorporating further edits, went to 12166 users of whom 894 completed the survey.
Form 0.2 incorporated edits to eliminate questions yielding little information about how to help users, either because satisfaction was very high (the language itself, the compiler, the community) or because two questions were redundant. Also, new questions inspired by textual responses to form 0.1 were included.
Form 1.0 incorporated further such edits. Also, the rating scale was changed to ask about helping the user’s (and team’s) future choice of Haskell rather than current usefulness/difficulty. The ratings questions were displayed under the heading “Would improvements help you and your group to choose Haskell for your future work?”
Responses were processed anonymously, but users were given the option to fill in their email address if they would accept follow-up questions, and the option to name their company/organization. Users were informed that the survey results, without these fields, would be shared with the community.
We are grateful to the many, many people who spent their valuable time and expertise completing and returning their survey forms. Thanks to Dr. Tristan Webb and Ms. Noelle McCool-Smiley, both of FP Complete, for their material help in formulating and conducting the survey. Thanks to FP Complete’s corporate customers for providing the revenues that allow us to fund this and other community projects. Thanks to the Commercial Haskell SIG for providing the motivation behind this project. Thanks to the many volunteers who’ve spent absolutely huge amounts of time and expertise making Haskell as good as it is today, and who continue to make improvements like those requested by the survey participants. Thanks to the companies that allow some of their staff to spend company time making such contributions to the common good. Special thanks to the late Professor Paul Hudak; may we all strive to live up to his example.
Subscribe to our blog via email
Email subscriptions come from our Atom feed and are handled by Blogtrottr. You will only receive notifications of blog posts, and can unsubscribe any time.
