We have a problem. We are professional software developers who work with open-source software. The problem is that we are in the minority. Most open-source software is written by amateurs.
Every time a hot new technology comes on the scene, developers flock to it like ants to a picnic. Those early adopters are, by definition, people for whom choosing a new technology is less risky. Which means, mostly, that their work doesn’t really matter. Students, hobbyists, “personal” projects: nobody’s life or career is on the line. It doesn’t matter if the program is entirely correct, efficient, or scalable. It doesn’t matter if it ignores lots of edge cases.
I’ve been one of those amateurs. It’s fun. New technologies need amateurs. But as a technology matures, it attracts professionals with real jobs who do care about those details. And those professionals are immediately confronted with a world of open-source software written by amateurs.
I used to write code for myself. Since I started getting paid to write code for other people, I’ve become wary of code written by people writing for themselves. Every time I see a README that begins “X is a dead simple way to do Y,” I shudder. Nothing in software is simple. “Dead simple” tells me the author has “simplified” by “deadening” vast swaths of the problem space, either by making unfounded assumptions or by ignoring them completely.
We like to carp about “bloated” APIs in “mainstream” languages like Java. Truly, lots of APIs are more complicated than they need to be. But just because an API is big doesn’t mean it’s bloated. I like big APIs: they show me that someone has thought about, and probably encountered, all of the edges and corners in the problem space.
Simplifying assumptions do not belong in libraries; they belong in applications, where you know the boundaries of the problem space. On rare occasions, the ground of one problem is trod often enough to warrant a framework. Emphasis on rare. A framework is almost always unnecessary, and, in these days of rapidly-changing technological capabilities, likely to be obsolete before it’s finished.
Frameworks written by amateurs are the worst of the worst: brittle constructs that assume everything in service of one or two “dead simple” demos but collapse under the weight of a real-world application.
I don’t want to be a code snob. Let’s be amateurs. Let’s have fun. Explore. Learn. Publish code as we go. Show a solution to a problem without assuming it’s the solution. Be cognizant of and vocal about what assumptions we’re making. Don’t call something a library unless it really attempts to reach every nook and cranny of the problem space.
And don’t write frameworks. Ever. ;)
Update August 8, 2013: Based on the comments, I feel like too many people have gotten hung up on the words amateur and professional. Those were just convenient labels which I found amusing. The important difference is between “easy” single-purpose code and thorough, general-purpose code.
Thanks Stuart, it is very well written, and resonates with me.
Re: frameworks
http://www.codinghorror.com/blog/2013/07/rule-of-three.html
tl;dr; (plus my take) Until you find three, very different users of the same basic construct that each having their own mature implementation, which you have studied enough to understand why their authors made the decisions they did, don’t try to write general code, i.e. frameworks.
Very true!
This essay reminded me of http://en.wikipedia.org/wiki/Worse_is_Better. Quite often in software the amateurs win mindshare and the professional solutions are abandoned.
I guess the real problem is amateurs pretending to be the professionals, either because they overrate their own skills or (perhaps mostly) because they don’t know the differences.
As long as amateurs (1) don’t pretend to be professionals, (2) keep their learning loop open by assuming professionals may have their say on their pet projects, then it’s a win-win situation : everyone enjoys more code, and amateurs enjoy learning more.
One paradoxon here is that the need to stick to this right attitude is more important for amateurs than for professionals… but this attitude would be good for professionals too!
“nobody’s life … is on the line”
When the contrary is the case you should use a language like Coq and formally proof correctness of your programs.
And some of us do. I’ve worked in settings where induction engines are being used for lemma generation and code proofs.
Thanks for your post, a couple of thoughts:
I believe there lies a false dichotomy in dividing into amateurs and professionals. I’m not sure I see that distinction as clearly, professionals can be amateurs and amateurs can be professionals. In fact everyone is an amateur.
Open source is free, and hence It is hard to judge value and quality. If someone was giving away free code on a street corner I doubt I would think its good quality. What quantifies a open source project being professional? Quality? Upon whose judgement? Maybe all code projects start as amateur? We rarely know all the unknowns or tech upfront and our decisions and code contain amateur mistakes that help us and the community learn.
“Amateurs” also feels slightly like a straw-man, in reality software is written by people who no matter how professional are prone to mistakes and flaws.
One final note: the discussion starts by dividing the groups into them and us (or “we” as you say). This gives the impression if I disagree with you I am not one of you and hence I must be an amateur.
Thanks.
Joseph Wilk- The word “dichotomy” shows you’re taking this too seriously. ;)
Open Source is driven by natural selection process. It’s awesome!
I ll always consider myself an amateur.
I don’t consider my code good quality to shared, but I learn a lot from snippets.
The way it is today is a bless.
I remember when internet connection was a dream, when I imported Object Pascal Library for Turbo Pascal.
Knowlegde is at hand nowadays. The valuable skill is to be able to filter it.
The irony is that in an ideal world, open source developers need to be *more* disciplined and professional than enterprise or in-house developers. Enterprise developers can change/expand/refine their APIs when they find issues. They can follow their preferred choice of agile methodology and build minimal products that hardly work, and grow them iteratively in response to customer feedback. In-house developers can gradually evolve their frameworks from working code.
If an open source library becomes widely used, then changes to it affect a lot of people. So the early API design has to be rather careful.
John Hurst- I’d like to visit that ideal world some time.
A wise man at a conference said, any framework is the first day in jail, you never know quite what’s going to happen! I consider myself to be professinel because I live well by developing software and keep me up to date with what is happening in the field. I try as much as possible to avoid framework. First, I do not often need more than one or two operations from the framework and its partly annoy me when a framework is often updated or, on the other hand, is totally stalled. Another reason to avoid frameworks, especially in an expressive platform as Clojure, is that you can avoid different abstraction levels in the software, which is important if the software has to be easy to maintain over many years.
Thanks for a great article, dead simple problems is like turtles, all the way down :-)
While I do agree fully with Joseph Wilk’s post, the author does have a good point here. Though, I do not see the main distinction between professional and amateur, but rather the reasons for creating the software. It’s more important that software is created for real world use than by experienced or paid professional developers.
As you say, it can be hard to find quality libraries especially if you’re evaluating a new language or a platform. The question becomes how do you identify what libraries to use, which ones work well together, or are well maintained. If you are a professional you probably don’t have a lot of time to try this and that till something works.
I agree that frameworks in a traditional sense don’t add much value. However, I think there is certainly a need for having sets of curated libraries and patterns for doing things. Another advantage is in standardizing patterns.
For example, let’s say you want to evaluate Clojure for web development. Where do you start? What libraries do you use, how do you organize them, what are your deployment options, and so on. Also, without any standardization it becomes difficult to move from one project to another.
While I agree that simplifying assumptions in libraries can lead to brittle, simplistic code that doesn’t work under realistic conditions, my experience re. professional vs hobbyist developers has been quite the opposite. Most of the code I’ve seen, written by professionals, has been far worse, both in design and actual code quality, than most open-source code I’ve seen.