Contents

Preface

Programming-Language Religions

Author’s Note: Tour de Babel

Tour de Babel

Author’s Note: Execution in the Kingdom of Nouns

Execution in the Kingdom of Nouns

Author’s Note: Notes from the Mystery Machine Bus

Notes from the Mystery Machine Bus

Author’s Note: Moore’s Law Is Crap

Moore’s Law Is Crap

Author’s Note: Transformation

Transformation

Author’s Note: Is Weak Typing Strong Enough?

Is Weak Typing Strong Enough?

Code Philosophies

Author’s Note: Software Needs Philosophers

Software Needs Philosophers

Author’s Note: Code's Worst Enemy

Code's Worst Enemy

Author’s Note: A Little Anti-Anti-Hype

A Little Anti-Anti-Hype

Author’s Note: Bambi Meets Godzilla

Bambi Meets Godzilla

Author’s Note: Math For Programmers

Math For Programmers

Author’s Note: Rich Programmer Food

Rich Programmer Food

On Google

Author’s Note: Get that job at Google

Get that job at Google

Author’s Note: Good Agile, Bad Agile

Good Agile, Bad Agile

Author’s Note: Is Google going to stay relevant?

Is Google Going to Stay Relevant?

Author’s Note: Google Platforms Rant

Google Platforms Rant

Conclusion

About the Authors

About the Publisher

A-Programmers-Rantings-On-Programminglanguage-Religions-Code-Philosophies-Google-Work-Culture-And-Other-Stuff

Steve Yegge

I.

Preface

This book grew out of a lot of angst. Well, and wine. Put enough angst in me, and I’ll start ranting. Pour in some wine, and the rants get mean—and funny. I still go back and read these posts now and then, and I always laugh. I was so mean.

My angst grew out of traveling different roads than most programmers. Those roads forced me to see the world differently. Now I see all sorts of patterns that many experienced programmers fail to see—because, well, to put it bluntly, they’re stuck in ruts.

Over the past 25 years I’ve done a bunch of dramatically different types of programming, and I’ve also written far more code than any programmer ever should. The long roads I’ve traveled have basically given me a sixth sense. I see dead people. And it sucks. If you’re ever unlucky enough to acquire a dreadful sixth sense, there are really only two choices: you can be angry and depressed about it, or you can laugh about it.

So I try to laugh. It’s hard, but I’m getting better at it. The wine helps. Practice helps, too. You need to get in the habit of laughing—at yourself, at others, at the crazy world we live in—or in time you’ll just stop laughing altogether.

Take driving. I laugh, nervously, with a bit of an eye twitch, when I see drivers with vehicle damage who obviously didn’t learn from their accidents. I’m always noticing tailgaters with crushed front fenders. Or cars braking randomly with crushed rear fenders. Cars with damaged front-left quarter panels poking their noses too far out onto busy streets. Cars with huge sideswipe marks swerving perilously close to a median.

Once you start noticing this pattern, you’ll see it everywhere. A lot of drivers just don’t seem to learn from their mistakes. They think, I _know_ how to drive. That accident wasn’t my fault!

How many adults do you know who spend time trying to become a better driver? How many drivers do you know who have some annoying driving habit that you really want to point out to them, but you know it’s going to piss them off if you bring it up?

The thing is, everyone thinks they know how to drive. As if it’s some simple thing, like knowing how to microwave a burrito. And anyone daring to suggest that their driving is less than perfect is a patronizing jerk.

It’s actually a little more subtle than that. The problem with most drivers is that we think knowing how to drive and being a good driver are the same thing. Once you have your license and a few years of real-world driving under your belt, well, that’s pretty much it: you’re an experienced driver! You know what you need to know. There might be some oddball edge-case situations here and there, like driving in snow or sand or water or smoke or extreme heat or extreme cold or strong winds or on steep hills or in crowds of people or whatever—places where you just go really slow and hope for the best. Sure, those edge-case situations do exist, but you can generally avoid them. And you could learn more specialized driving techniques if you ever needed them. Right?

Well, sort of… except that by the time you actually need the skill, it’s usually too late to go off and take some driver-training course. Often your first hint is when you’re hydroplaning into a telephone pole. If you’re lucky, you can walk away. But after your accident, do you go off and study and train to become a better driver? Hells, no. Not unless a judge forces you to. Because you know how to drive.

If you stop to think about all the specialized driving out there—professional racing, or police techniques for physically trapping cars, or park rangers on fire roads, or very large vehicles, or any of the other bazillion kinds of specialized driving out there—if you think about it, you should realize that you’re not a very good driver. At best, you’re an adequate driver. And since half of all drivers are below average, there’s a pretty good chance that you actually suck at driving, and everyone but you knows it.

This is one of the reasons a zombie apocalypse seems so plausible. It’s because we know everyone’s totally unprepared for it. We know most drivers are going to be zombie chow, because they all think they know how to drive, and nobody’s gonna tell them otherwise.

Programming, as it turns out, is a lot like driving. Programmers get a few years of experience under their belts, and bam! They know how to program, just like microwaving a burrito. Most programmers then fall into a comfort zone and try to stay there for their whole careers, just the way drivers try to avoid unfamiliar situations. Anyone who tries to suggest that they could stand a little improvement is a patronizing jerk or simply misinformed.

But that’s the same ass-backwards argument that so many families make about swimming lessons, isn’t it? All those millions of families who don’t let their children learn to swim because they might drown?

Programmers and drivers are always telling themselves they can learn new skills and disciplines “on demand” if and when they ever need them. But they all know deep down that on demand is too late. In the real world, non-swimmers avoid water, drivers avoid dirt roads, and programmers stake out their comfort zone and start building fences. Then everyone keeps their fingers crossed and hopes for an uneventful life.

You might think that’s just ducky—as long as each job is filled with some specialist, then nothing should ever go wrong. But if we dig into that programming-as-driving metaphor a little further, it’s not so cut-and-dried. Driving is a complicated and opinionated affair. We all like to pretend that it’s not, and that causes big real-life problems.

Take regional driving differences. People in Seattle drive very differently from people in Los Angeles, even though the laws are pretty much the same. A civil engineer once told me that how fast or aggressively people in a region drive is a function of how long the traffic has been bad in the region. The traffic in L.A. has been terrible for generations, so everyone knows they have to drive aggressively. The traffic in Seattle has been bad for only a decade or two, and it’s still not as bad as in L.A., so Seattle drivers drive like Grandma.

And ironically enough, Seattle drivers are terrible at driving in the rain. They slow down to a near-standstill and still get in tons of accidents.

Why is that? Well, it’s because Seattle’s driving philosophy is currently in flux. The overall driving style is transforming from leisurely to aggressive. Part of this change is due to a literal influx of Californians moving up the coast and bringing their driving habits with them. And part of it is driven by the rapid population expansion itself, which is putting more cars on the road than Seattle has ever seen before. So Seattle today has a mix of timid and aggressive drivers.

When driving philosophies collide, so do vehicles. And when philosophies align, with everyone in a region driving “the same way,” then accidents are minimized—even if the drivers all look completely insane to an outsider.

The programming world is just as intricately complicated as the world of driving. Programming has the equivalents of drivers, mechanics, auto manufacturers, traffic engineers, regional traffic laws, traffic law enforcement, regional driving styles, and of course massive fatal multi-vehicle pile-ups. Programming has subcultures, and they collide whenever you put them together.

That’s how and why this book happened. I went off the beaten path as programmer and spent years and years adventuring in exotic places that most programmers never visit. I learned how people drive software development in different styles and cultures. I drove a bunch of weird specialist rigs with their own memorable quirks. I wrote in dozens of languages, got to know dozens of communities, and watched huge empires of projects rise and conquer and fade and die. I doubt I’ve seen it all in the programming world, but I’ve seen a lot.

And now I see parochialism everywhere. Just like drivers in geographic regions, programmers cluster together in regions of the industry and academia and develop local accents, customs, taboos, lore, and other cultural outgrowths. They form intellectual enclaves that think their way is the best way, the only way.

Whenever there’s a big accident or traffic problem, people blame bad drivers, and programmers blame bad coders. And sometimes it really is bad drivers or bad coders. But often as not, the collisions are caused by philosophical conflicts stemming from cultural differences.

Unfortunately, most programmers, because they know how to program, are basically parochial types who prefer to label anyone who differs from them as “wrong.” It’s human nature. It’s one of the easiest traps we can fall into; I’ve been there.

When I first started ranting, I was the ugly American, stomping around in my posts, and essentially yelling “What the hell is wrong with all you people?” But over the next ten years or so, I like to think I’ve grown into more of an amateur software anthropologist. I now take cultural relativism seriously, and I try hard not to judge people who think differently from me.

Of course I don’t mind poking fun at them, because I don’t mind people poking fun at me. And ultimately I would like to convince undecided programmers to share my view of the programming world, because—just like driving—programming works best if everyone nearby does it the same way. So I’ll continue to argue that my view, which I’ve recently taken to calling “software liberalism,” is a perfectly valid and perhaps even preferable way to do a lot of software development.

Converting everyone to be more liberal is doomed to fail, of course. But even so, I hope I can still help people in radically different software cultures to understand each other better.

I’m going to keep ranting, because it appears to be the only way to make a message sink in to a very large audience. Some people still tell me that my blog posts are too long. They tell me I could have made my “point” in under a hundred words. I have noticed that this complaint comes most often from people who disagree with me. They’re really just saying they want less work to voice their disagreement. But even some folks who agree with me find the posts too long to carry their attention, and they complain too.

They’re missing the point, though. The posts aren’t too long. You need a certain minimum “heft” to penetrate. Through years of trial and error, I’ve found that the best way to get a lot of people to listen to you is to tell them a story. And you can’t spin a good yarn without settling in and enjoying the ride.

So that’s what this book is. It’s really a bunch of stories. Each might take the form of an article, essay, guide, rant, or occasionally a fiction tale. But behind the structure, each one of them is sharing a story. Even if you don’t always agree, I’m hoping you’ll at least find the stories entertaining and, with luck, sometimes even eye-opening.

The guys at Hyperink chose which of my posts to include, by and large, and they also came up with the overall chapter organization. I made a couple of tweaks, but what you’re looking at is largely their vision of how to curate this stuff into a cohesive book. I think they did an admirable job.

I hope you enjoy the journey as much as I did.

Steve Yegge

August 2012

Become a Hyperink reader. Get a special surprise.

Like the book? Support our author and leave a comment!

II.

Programming-Language Religions

Author’s Note: Tour de Babel

This is by far the oldest post in this book, written back in September 2004. I had been at Amazon.com for almost six years. During that time, as Amazon struggled with their massive code base, it had gradually been dawning on me that many of Amazon’s code-base scaling issues could be classified as language problems. And the root cause was their culture.

The first culture issue was that they had two verbose languages (C++ and Java), and one succinct one (Perl). But Perl was getting squeezed out of the picture, falling out of fashion. I think this is self-perpetuating: Perl programmers can get things done with fewer engineers, so they are doomed to lose—via population warfare—to Java and C++ programmers. So Amazon’s code was bloating faster than their feature set, according to our measurements.

The second contributor was that Amazon had a bunch of engineering problems that would have been best solved with custom domain-specific languages (DSLs). These included large-scale queries, distributed computations, prod configuration, and other domains where they were just writing WAY too much code. A year later, after leaving to join Google, I found that Google had written powerful custom DSLs for exactly these domains. This confirmed my suspicions that Amazon engineers had just been hacking around quasi-cluelessly in these spaces. There were exceptions, but they were rare.

The last piece of the puzzle was that Amazon—like most companies—was heavily biased against solving problems using new languages. They were biased against using more expressive general-purpose ones like Ruby or Erlang, and they almost never created DSLs.

So I could see the problems they were having, and how to solve them, but I found myself utterly unable to get buy-in from the lead engineers and other managers. With only a few exceptions, the vast majority were hear-no-evil-see-no-evil: C++ for everything; problem solved. Or else Java for everything, if you swing that way. Nothing else would do. They just wouldn’t hear it.

Anyway, one day I finally got fed up—and liquored up—and I decided to unleash my frustrations on my Amazon colleagues all at once. I think this may have been my first-ever bona fide blog “rant,” where I just lost it and started ranting about halfway through. Nobody read it at the time, of course. But over the years I’ve had many people tell me it started them on a lifelong journey to mastering Emacs and Lisp—some starting as early as their high school freshman year!

Tour de Babel

This is my whirlwind languages tour—the one I was going to write for the Amazon Developers Journal this month, but couldn't find a way to do it that was… presentable.

For one thing, I lapse occasionally into coarseness and profanity here, so it wasn't appropriate for an official-ish Amazon publication. Instead, I'm stuffing it into my blog, which nobody reads. Except for you. Yep, just you. Hiya!

For another, it's really a work in progress, just a few snippets here and there, not polished at all. Another great reason to make it a blog entry. Doesn't have to be good or complete. It's just what I have today. Enjoy!

My whirlwind tour will cover C, C++, Lisp, Java, Perl—all languages we use at Amazon—Ruby (which I just plain like), and Python, which is in there because… well, no sense getting ahead of ourselves now.

C

You just have to know C. Why? Because for all practical purposes, every computer in the world you'll ever use is a von Neumann machine, and C is a lightweight, expressive syntax for the von Neumann machine's capabilities.

The von Neumann architecture is the standard computer architecture you use every day: a CPU, RAM, a disk, a bus. Multi-CPU doesn't really change it that much. The von Neumann machine is a convenient, cost-effective, 1950s realization of a Turing Machine, which is a famous abstract model for performing computations.

There are other kinds of machines, too. For instance, there are Lisp Machines, which are convenient 1950s realizations of Lisp, a programming language notation based on the lambda calculus, which is another model for performing computations. Unlike Turing machines, the lambda calculus can be read and written by humans. But the two models are equivalent in power. They both model precisely what computers are capable of.

Lisp Machines aren't very common though, except at garage sales. von Neumann machines won the popularity race. There are various other kinds of computers, such as convenient realizations of neural networks or cellular automata, but they're nowhere as popular either, at least not yet.

So you have to know C.

You also have to know C because it's the language that Unix is written in. It also happens to be the language that Windows and virtually all other operating systems are written in, because they're OSes for von Neumann machines, so what else would you use? Anything significantly different from C is going to be too far removed from the actual capabilities of the hardware to perform well enough, at least for an OS—at least in the last century, which is when they were all written.

You should also know Lisp. You don't have to use it for real work, although it comes in quite handy for lots of GNU applications. In particular, you should learn Scheme, which is a small, pure dialect of Lisp. The GNU version is called Guile.

They teach Scheme at MIT and Berkeley to new students for a semester or two, and the students have absolutely no clue why they're learning this weird language. It's a lousy first language, to be honest, and probably a lousy second one, too. You should learn it, eventually, but not as your first or second language.

It's hard, though. It's a big jump. It's not sufficient to learn how to write C-like programs in Lisp. That's pointless. C and Lisp stand at opposite ends of the spectrum; they're each great at what the other one sucks at doing.

If C is the closest language to modeling how computers work, Lisp is the closest to modeling how computation works. You don't need to know a lot of Lisp, really. Stick with Scheme, since it's the simplest and cleanest. Other Lisps have grown into big, complex programming environments, just like C++ and Java have, with libraries and tools and stuff. That, you don't need to know. But you should be able to write programs in Scheme. If you can make your way through all the exercises in The Little Schemer and The Seasoned Schemer, you'll know enough, I think.

But you choose a language for day-to-day programming based on its libraries, documentation, tools support, OS integration, resources, and a host of other things that have very little to do with how computers work, and a whole lot to do with how people work.

People still write stuff in straight C. Lots of stuff. You should know it!

C++

C++ is the dumbest language on earth, in the very real sense of being the least sentient. It doesn't know about itself. It is not introspective. Neither is C, but C isn't "Object-Oriented," and object orientation is in no small measure about making your programs know about themselves. Objects are actors. So OO languages need to have runtime reflection and typing. C++ doesn't, not really, not that you'd ever use.

As for C: it's so easy to write a C compiler that you can build tools on top of C that act like introspection. C++, on the other hand, is essentially unparseable, so if you want to write smart tools that can, for example, tell you the signatures of your virtual functions, or refactor your code for you, you're stuck using someone else's toolset, since you sure as heck aren't gonna parse it. And all the toolsets for parsing C++ out there just plain suck. (Author’s note: clang is pretty good now, some eight years after I wrote this. Finally!)

C++ is dumb, and you can't write smart systems in a dumb language. Languages shape the world. Dumb languages make for dumb worlds.

All of computing is based on abstractions. You build higher-level things on lower-level ones. You don't try to build a city out of molecules. Trying to use too low-level an abstraction gets you into trouble.

We are in trouble.

The biggest thing you can reasonably write in C is an operating system, and they're not very big, not really. They look big because of all their apps, but kernels are small.

The biggest thing you can write in C++ is… also an operating system. Well, maybe a little bigger. Let's say three times bigger. Or even ten times. But operating system kernels are at most, what, maybe a million lines of code? So I'd argue the biggest system you can reasonably write in C++ is maybe 10 million lines, and then it starts to break down and become this emergent thing that you have no hope of controlling, like the plant in Little Shop of Horrors. Feeeeeed meeeeeee…

If you can get it to compile by then, that is.

We have 50 million lines of C++ code. No, it's more than that now. I don't know what it is anymore. It was 50 million last Christmas, nine months ago, and was expanding at 8 million lines a quarter. The expansion rate was increasing as well. Ouch.

Stuff takes forever to do around here. An Amazon engineer once described our code base as "a huge mountain of poop, the biggest mountain you've ever seen, and your job is to crawl into the very center of it, every time you need to fix something."

That was four years ago, folks. That engineer has moved on to greener pastures. Too bad; he was really good.

It's all C++'s fault. Don't argue. It is. We're using the dumbest language in the world. That's kind of meta-dumb, don't you think?

With that said, it is obviously possible to write nice C++ code, by which I mean, code that's mostly C with some C++ features mixed in tastefully and minimally. But it almost never happens. C++ is a vast playground, and it makes you feel smart when you know all of it, so you're always tempted to use all of it. But that's really, really hard to do well, because it's such a crap language to begin with. In the end, you just make a mess, even if you're good.

I know, this is heresy with a capital “H.” Whatever. I loved C++ in college, because it's all I knew. When I heard that my languages prof, Craig Chambers, absolutely detested C++, I thought: "Why? I like it just fine." And when I heard that the inventor of STL was on record as saying he hated OOP, I thought he was cracked. How could anyone hate OOP, especially the inventor of STL?

Familiarity breeds contempt in most cases, but not with computer languages. You have to become an expert with a better language before you can start to have contempt for the one you're most familiar with.

So if you don't like what I'm saying about about C++, go become an expert at a better language (I recommend Lisp), and then you'll be armed to disagree with me. You won't, though. I'll have tricked you. You won't like C++ anymore, and you might be irked that I tricked you into disliking your ex-favorite language. So maybe you'd better just forget about all this. C++ is great. Really. It's just ducky. Forget what I said about it. It's fine.

Lisp

(I'm betting this next section will astonish you, even if you've been here a while.)

When Amazon got its start, we had brilliant engineers. I didn't know all of them, but I knew some of them.

Examples? Shel Kaphan. Brilliant. Greg Linden. Brilliant. Eric Benson. Independently famous in his own right, before he ever even came to Amazon. Also brilliant.

They wrote the Obidos web server. Obidos made Amazon successful. It was only later that poop-making engineers and web devs, front-end folks mostly—schedule-driven people who could make their managers happy by delivering crap fast—it was only later that these people made Obidos bad. Clogged the river, so to speak. But Obidos was a key cornerstone of Amazon's initial success.

The original brilliant guys and gals here only allowed two languages in Amazon's hallowed source repository: C and Lisp.

Go figure.

They all used Emacs, of course. Hell, Eric Benson was one of the authors of XEmacs [1]. All of the greatest engineers in the world use Emacs. The world-changer types. Not the great gal in the cube next to you. Not Fred, the amazing guy down the hall. I'm talking about the greatest software developers of our profession, the ones who changed the face of the industry. The James Goslings, the Donald Knuths, the Paul Grahams [2], the Jamie Zawinskis, the Eric Bensons. Real engineers use Emacs. You have to be way smart to use it well, and it makes you incredibly powerful if you can master it. If you don’t believe me, go look over Paul Nordstrom's shoulder while he works sometime. It's a real eye-opener for someone who's used Visual Blub .NET-like IDEs their whole career.

Emacs is the 100-year editor.

Shel, Eric, Greg, and others like them whom I wasn't fortunate enough to work with directly: they didn't allow C++ here, and they didn't allow Perl. (Or Java, for that matter). They knew better.

Now C++, Java, and Perl are all we write in. The elders have moved on to greener pastures too.

Shel wrote Mailman in C, and Customer Service wrapped it in Lisp. Emacs-Lisp. You don't know what Mailman is. Not unless you're a longtime Amazon employee, probably non-technical, and you've had to make our customers happy. Not indirectly, because some bullshit feature you wrote broke (because it was in C++) and pissed off our customers, so you had to go and fix it to restore happiness. No, I mean directly; i.e. you had to talk to them. Our lovely, illiterate, eloquent, well-meaning, hopeful, confused, helpful, angry, happy customers, the real ones, the ones buying stuff from us, our customers. Then you know Mailman.

Mailman was the Customer Service customer-email processing application for… four, five years? A long time, anyway. It was written in Emacs. Everyone loved it.

People still love it. To this very day, I still have to listen to long stories from our non-technical folks about how much they miss Mailman. I'm not shitting you. Last Christmas I was at an Amazon party, some party I have no idea how I got invited to, filled with business people, all of them much prettier and more charming than I am and the folks I work with here in the Furnace, the Boiler Room of Amazon. Four young women found out I was in Customer Service, cornered me, and talked for fifteen minutes about how much they missed Mailman and Emacs, and how Arizona (the JSP replacement we'd spent years developing) still just wasn't doing it for them.

It was truly surreal. I think they may have spiked the eggnog.

Shel's a genius. Emacs is a genius. Even non-technical people love Emacs. I'm typing in Emacs right now. I'd never voluntarily type anywhere else. It's more than just a productivity boost from having great typing shortcuts and text-editing features found nowhere else on the planet. I type 130 to 140 WPM, error-free, in Emacs, when I'm doing free-form text. I've timed it, with a typing-test Emacs application I wrote. But it's more than that.

Emacs has the Quality Without a Name.

We retired Mailman. That's because we have the Quality With a Name—namely, Suckiness. We suck. We couldn't find anyone who was good enough at Emacs-Lisp to make it work. Nowadays it would be easy; Amazon's filled up with Emacs Lisp hackers, but back then, CS Apps couldn't get the time of day from anyone, so they did what they could with what they had, and there weren't enough Emacs-Lisp folks. For a while, they even had Bob Glickstein on contract, the guy who wrote the O'Reilly "giraffe" book Writing Gnu Emacs Extensions, sitting there writing Gnu Emacs Extensions for Mailman in this little office in the Securities building.

CS Apps was Amazon's first two-pizza team, you know. They're completely autonomous—then and now. Nobody talks to them, nobody helps them, they build everything themselves. They don't have web devs, they don't have support engineers, they don't have squat, except for rock-solid engineers and a mentoring culture. And that's all they've ever needed.

But they had to retire Mailman. Alas. Alackaday. And I still get to hear about how much people miss it. At parties, even.

I think there may still be more Lisp hackers, per capita, in CS Apps than in any other group at Amazon. Not that they get to use it much, but as Eric Raymond said, even if you don't program in it much, learning Lisp will be a profound experience that will make you a better engineer for the rest of your life.

Religion isn't the opiate of the masses anymore, Karl. IDEs are.

Java

Java is simultaneously the best and the worst thing that has happened to computing in the past 10 years.

On the one hand, Java frees you up from many mundane and error-prone details of C++ coding. No more bounds errors, no more core dumps. Exceptions thrown point you to the exact line of code that erred, and are right 99 percent of the time. Objects print themselves intelligently on demand. Etc., etc.

On the other hand, in addition to being a language, a virtual machine, a huge set of class libraries, a security model, and a portable bytecode format, Java is a religion. So you can't trust anyone who loves it too much. It's a tricky business to hire good Java programmers.

But Java really has been a big step forward for software engineering in general.

Going from C++ to Java isn't just changing syntax. It's a shocking paradigm shift that takes a while to sink in. It's like suddenly getting your own executive assistant. You know how VPs always seem to have all this time to be in meetings, and know how the company's running, and write cool documents, and stuff like that? VPs tend to forget that they're actually two full-time people: their self and their EA. Having an EA frees you up to think about the problems you need to solve; not having one forces you to spend half your time on mundane tasks. Switching to Java turns you into two programmers—one taking care of all this stuff that you no longer have to think much about, and another one focused on the problem domain. It's a staggering difference, and one you can get used to in a hurry.

As Jamie Zawinski said in his famous "java sucks" article: "First the good stuff: Java doesn't have free(). I have to admit right off that, after that, all else is gravy. That one point makes me able to forgive just about anything else, no matter how egregious. Given this one point, everything else in this document fades nearly into insignificance."

Jamie's article was written in 1997, which in Java years is a long time ago, and Java has improved a lot since he wrote it; some of the things he complains about are even fixed now.

Most of them aren't. Java does still kind of suck, as a language. But as Jamie points out, it's "the best language going today, which is to say, it's the marginally acceptable one among the set of complete bagbiting loser languages that we have to work with out here in the real world."

Really, you should read it.

Java is truly wonderful along almost every dimension except for the language itself, which is mostly what JWZ was griping about. But that's a lot to gripe about. Libraries can only get you so far if your language sucks. Trust me: you may know many, many things better than I do, but I know that libraries can't really save a sucky language. Five years of assembly-language hell at Geoworks taught me that.

Compared to C++, Java as a language is about even. Well, scratch that, it's a lot better, because it has strings. Oh man, how can you use a language with lousy string support.

But Java's missing some nice features from C++, such as pass-by-reference(-to-stack-object), typedefs, macros, and operator overloading. Stuff that comes in handy now and then.

Oh, and multiple inheritance, which now I've come to appreciate in my old age. If you think my Opinionated Elf was a good counterpoint to polymorphism dogma, I have several brilliant examples of why you need multiple inheritance, or at least Ruby-style mixins or automatic delegation. Ask me about the Glowing Sword or Cloak of Thieving sometime. Interfaces suck.

Gosling even said, a few years ago, that if he had to do it all over again, he wouldn't have used interfaces.

But that's exactly the problem with Java. When James said that, people were shocked. I could feel the shock waves, could feel the marketing and legal folks at Sun maneuvering to hush him up, brush it off, and say it wasn't so.

The problem with Java is that people are blinded by the marketing hype. That's the problem with C++, with Perl, with any language that's popular, and it's a serious one, because languages can't become popular without hype. So if the language designer suggests innocently that the language might not have been designed perfectly, it's time to shoot the language designer full of horse tranquilizers and shut down the conference.

Languages need hype to survive; I just wish people didn't have to be blinded by it.

I drank the OOP Kool-Aid, and even regurgitated the hype myself. When I started at Amazon, I could recite for you all the incantations, psalms, and voodoo chants that I'd learned, all in lieu of intelligence or experience, the ones that told me Multiple Inheritance is Evil 'cuz Everyone Says So, and Operator Overloading Is Evil, and so on. I even vaguely sort of knew why, but not really. Since then I've come to realize that it's not MI that sucks—it's developers who suck. I sucked, and I still do, although hopefully less every year.

I had an interview candidate last week tell me that MI is Evil because, for instance, you could make a Human class that multiply-inherits from Head, Arm, Leg, and Torso. He was both right and wrong. That MI situation was evil, sure, but it was all him. Stupid from a distance, evil if he'd made it in through the front door.

Bad developers, who constitute the majority of all developers worldwide, can write bad code in any language you throw at them.

That said, though, MI is no picnic; mixins seem to be a better solution, but nobody has solved the problem perfectly yet. But I'll still take Java over C++, even without MI, because I know that no matter how good my intentions are, I will at some point be surrounded by people who don't know how to code, and they will do far less damage with Java than with C++.

Besides, there's way more to Java than the core language. And even the language is evolving, albeit glacially, so there's hope. It's what we should be using at Amazon.

You just have to be careful, because as with any other language, you can easily find people who know a lot about the language environment, and very little about taste, computing, or anything else that's important.

When in doubt, hire Java programmers who are polyglots, who detest large spongy frameworks like J2EE and EJB, and who use Emacs. All good rules of thumb.

Perl

Perl. Where to start?

Perl is an old friend. Perl and I go way back. I started writing Perl stuff in maybe 1995, and it's served me well for nearly a decade.

It's like that old bicycle you've put 100 or 200 thousand miles on, and you'll always have a warm fuzzy spot for it, even though you've since moved on to a more modern bike that weighs five pounds and doesn't make your ass hurt so much.

Perl is popular for three reasons:

1. You can get stuff done really fast in Perl, which is what really matters, in the end.

2. Perl has the best marketing in the world. You could write a book about how brilliant their marketing is. Sun has marketed Java with money, and Perl is almost keeping up, popularity-wise, purely on the on sheer marketing brilliance of Larry Wall and his buddies. Folks at Harvard Business School should study Perl's marketing. It's astonishing.

3. Until roughly, oh, now, it had no real competitors.

There are "better" languages than Perl—hell, there are lots of them, if you define "better" as "not being insane." Lisp, Smalltalk, Python, gosh, I could probably name 20 or 30 languages that are "better" than Perl, inasmuch as they don't look like that Sperm Whale that exploded in the streets of Taiwan over the summer. Whale guts everywhere—covering cars, motorcycles, pedestrians. That's Perl. It's charming, really.

But Perl has many, many things going for it that, until recently, no other language had, and they compensated for its exo-intestinal qualities. You can make all sorts of useful things out of exploded whale, including perfume. It's quite useful. And so is Perl.

While all those other languages (Lisp and Smalltalk being particularly noteworthy offenders) tried to pretend that operating systems don't exist, and that lists (for Lisp) or objects (for Smalltalk) are the be-all, end-all of getting shit done, Perl did exactly the opposite.

Larry said: Unix and string processing are the be-all, end-all of getting shit done.

And for many tasks, he was absolutely right. So Perl is better at Unix integration and string processing than any language on the planet, save one, and that one only arrived on the scene recently, from the land of Godzilla. I'll get to that one later.

Sadly, Larry focused so hard on Unix integration and string processing that he totally forgot about lists and objects until it was far too late to implement them properly. In fact, a few key mistakes he made early on in Perl's… well, I hesitate to use the word "design" for whale guts, but let's call it Perl's "lifecycle"—those mistakes made it so hard to do lists and objects correctly that Perl has evolved into a genuine Rube Goldberg machine, at least if you want to use lists or objects.

Lists and objects are pretty farging important too, Larry!

Perl can't do lists because Larry made the tragically stupid decision early on to flatten them automatically. So (1, 2, (3, 4)) magically becomes (1, 2, 3, 4). Not that you ever want it to work this way. But Larry happened to be working on some problem for which it was convenient on that particular day, and Perl's data structures have been pure exploded whale ever since.

Now you can't read a book, tutorial, or PowerPoint on Perl without spending at least a third of your time learning about "references," which are Larry's pathetic, broken, Goldbergian fix for his list-flattening insanity. But Perl's marketing is so incredibly good that it makes you feel as if references are the best thing that ever happened to you. You can take a reference to anything! It's fun! Smells good, too!

Perl can't do objects because Larry never really believed in them. Maybe that's okay; I'm still not quite sure if I believe in them either. But then why did he try adding them? Perl's OO is a halfhearted add-on that never caught on with the Perl community. It's just not as inspired as the string-processing or Unix integration stuff.

And of course, Perl has plenty of other crackpot design features. Take its "contexts," for instance, which are a horrid outgrowth of Larry's comical decision to have N variable namespaces, dereferenced by sigils, which he sort of copied from shell-script. In Perl, every operator, every function, every operation in the language behaves randomly in one of six different ways, depending on the current "context." There are no rules or heuristics governing how a particular operation will behave in a given context. You just have to commit it all to memory.

Need an example? Here's one: accessing a hash in a scalar context gives you a string containing a fraction whose numerator is the number of allocated keys, and the denominator is the number of buckets. Whale guts, I'm telling you.

Like I said, though—until recently, nothing could get the job done like Perl could.

Ruby

Every 15 years or so, languages are replaced with better ones. C was replaced by C++, at least for large-scale application development by people who needed performance but desperately wanted data types too. C++ is being replaced by Java, and Java will doubtless be replaced with something better in seven years—well, seven years after it finishes replacing C++, which evidently hasn't fully happened yet, mostly because Microsoft was able to stall it before it became ubiquitous on the desktop. But for server-side applications, C++ is basically on its way out.

Perl will be gone soon, too. That's because a new language called Ruby has finally been translated into English. Yep, it was invented in Japan, of all places—everyone else was as surprised as you are, since Japan's known for its hardware and manufacturing, but not for its software development. Why, is anyone's guess, but I'm thinking it's the whole typing thing; I just can't imagine they were able to type fast enough before, what with having an alphabet with ten thousand characters in it. But Emacs got multibyte support a few years ago, so I can imagine they're pretty dang fast with it now. (And yes, they use Emacs—in fact Japanese folks did the majority of the Mule [multibyte] support for Emacs, and it's rock-solid.)

Anyway, Ruby stole everything good from Perl; in fact, Matz, Ruby's author (Yukihiro Matsumoto, if I recall correctly, but he goes by "Matz"), feels he may have stolen a little too much from Perl and got some whale guts on his shoes. But only a little.

For the most part, Ruby took Perl's string processing and Unix integration as-is, meaning the syntax is identical, and so right there, before anything else happens, you already have the Best of Perl. And that's a great start, especially if you don't take the Rest of Perl.

But then Matz took the best of list processing from Lisp, and the best of OO from Smalltalk and other languages, and the best of iterators from CLU, and pretty much the best of everything from everyone.

And he somehow made it all work together so well that you don't even notice that it has all that stuff. I learned Ruby faster than any other language, out of maybe 30 or 40 total; it took me about three days before I was more comfortable using Ruby than I was in Perl, after eight years of Perl hacking. It's so consistent that you start being able to guess how things will work, and you're right most of the time. It's beautiful. And fun. And practical.

If languages are bicycles, then Awk is a pink kiddie bike with a white basket and streamers coming off the handlebars, Perl is a beach cruiser (remember how cool they were? Gosh.) and Ruby is a $7,500 titanium mountain bike. The leap from Perl to Ruby is as significant as the leap from C++ to Java, but without any of the downsides, because Ruby's essentially a proper superset of Perl's functionality, whereas Java took some things away that people missed, and didn't offer real replacements for them.

I'll write more about Ruby sometime. I need to be inspired first. Read Why the Lucky Stiff's (poignant) guide to Ruby. That is an inspired piece of work. Really. Read it. It's amazing. I don't understand the kind of mind that could produce it, but it's funny, and poignant, and all about Ruby. Sort of. You'll see.

Python

Well gosh, what about Python, a nice language that has patiently been waiting in the wings for all these years? The Python community has long been the refuge for folks who finally took the red pill and woke up from the Perl Matrix.

Well, they're just like the Smalltalk folks, who waited forever to replace C++, and then Java came along and screwed them royally and permanently. Oops. Ruby's doing exactly that to Python, right now, today. Practically overnight.

Python would have taken over the world, but it has two fatal flaws: the whitespace thing, and the permafrost thing.

The whitespace thing is simply that Python uses indentation to determine block nesting. It forces you to indent everything a certain way, and they do this so that everyone's code will look the same. A surprising number of programmers hate this, because it feels to them like their freedom is being taken away; it feels as if Python is trampling their constitutional right to use shotgun formatting and obfuscated one-liners [3].

Python's author, Guido Van Rossum, also made some boneheaded technical blunders early on—none quite as extravagant as Larry's blunders, but a few were real doozies nonetheless. For instance, Python originally had no lexical scoping. But it didn't have dynamic scoping either, and dynamic scoping may have its share of problems, but it at least sort of works. Python had nothing except for global and local (function) scope, so even though it had a "real" OO system, classes couldn't even access their own damned instance variables. You have to pass a "self" parameter to every instance method and then get to your instance data by accessing it through self. So everything in Python is self, selfself, selfselfself, selfSELFselfSELF__SELF__, and it drives you friggin’ nuts, even if you don't mind the whitespace thing.

Etc.

But in my opinion, it's really the frost thing that killed Python, and has prevented it from ever achieving its wish to be the premier scripting language, or the premier anything language, for that matter. Heck, people still use Tcl as an embedded interpreter, even though Python is far superior to Tcl in every conceivable way—except, that is, for the frost thing.

What's the frost thing, you ask? Well, I used to have a lot of exceptionally mean stuff written here, but since Python's actually quite pleasant to work with (if you can overlook its warts), I no longer think it's such a great idea to bash on Pythonistas. The "frost thing" is just that they used to have a tendency to be a bit, well, frosty. Why?

Because they were so tired of hearing about the whitespace thing!

I think that's why Python never reached Perl's level of popularity, but maybe I'm just imagining things.

Coda

That was the ADJ article I really wanted to write. Or at least something like it. For some reason, though, my true feelings only seem to come out during insomniac attacks between 3 A.M. and 6 A.M. Time for bed! Two hours 'til my next meeting.

Notes

1. Eric tells me it was actually almost all Jamie Zawinski, when they worked at Lucid together.

2. It's been pointed out many times since I wrote this that Paul Graham actually uses vi. Go figure!

3. For the record, I personally don't mind the whitespace thing at all. I think it's silly to dislike Python for that reason. What I'm saying is that a surprising percentage of *other* programmers hate it.

Author’s Note: Execution in the Kingdom of Nouns

This was my first “famous” rant. I remember it clearly. I’d left Amazon in mid-2005 to join Google, and published most of my old internal Amazon rants on my public blog. They had started getting some attention around December, via reddit and Hacker News—just enough to where everything I posted was immediately being read by thousands of people.

Meanwhile, inside Google I had been working with a particular Google Java code base that disagreed with me. We just didn’t see eye to eye. I’ve been in a few Java code bases that were tolerable, but this one was grossly over-engineered. Not gonna say which one. But it was horrible. It was almost a spoof of itself. It was a programming style that has become increasingly popular in Java, favoring composability and immutability in a language with almost no direct support for either. And this was pre-Java 5 code, making it even worse.

I got the impression the team just loved their code, since they kept on churning out more of it. You’ll get a feel for what it was like when you read my actual spoof of it in this post.

Anyway, I was depressed about it, and I was ranting about it to my buddy and fellow Googler Todd Stumpf at the gym every night. I was trying to find a way, some way, any way to paint a picture for them that would show the team what their code looked and smelled like to me.

Eventually it came together, and one night I sat down and wrote a satirical bedtime story, surrounded by a little exposition that I probably should have just omitted. I posted; it went viral; people went nuts. Most Java folks at the time were shocked, and often too confused to be angry. Everyone else jeered and hooted and laughed it up, not knowing they were next. (I’ve made fun of almost every language in my posts over the years.)

For what it’s worth, they still haven’t added lambdas, as of this writing, over six years later. I like to think my constant ridicule played at least some small role in motivating them to add lambdas to Java 8 (assuming they make it in this time around—historically it’s not looking good). The Java community’s pressure against lambdas, fueled by the fear of this scary “unknown” that’s been around for decades in other languages, has sent the proponents back to the drawing board repeatedly.

But now it looks like Java might finally get some verbs, right around seven years after I wrote this story.

Execution in the Kingdom of Nouns

“They've a temper, some of them—particularly verbs, they're the proudest—adjectives you can do anything with, but not verbs—however, I can manage the whole lot of them! Impenetrability! That's what I say!” -Humpty Dumpty

Hello, world! Today we're going to hear the story of Evil King Java and his quest for worldwide verb stamp-outage. (Beginning with the verb "to stamp out," which is being replaced by a call to VerbEliminatorFactory.createVerbEliminator(currentContext).operate(). But that's getting way ahead of ourselves…)

Caution: This story does not have a happy ending. It is neither a story for the faint of heart nor for the critical of mouth. If you're easily offended, or prone to being a disagreeable knave in blog comments, please stop reading now.

Before we begin the story, let's get some conceptual gunk out of the way.

The Garbage Overfloweth

All Java people love "use cases," so let's begin with a use case: namely, taking out the garbage. As in, "Johnny, take out that garbage! It's overflowing!"

If you're a normal, everyday, garden-variety, English-speaking person, and you're asked to describe the act of taking out the garbage, you probably think about it roughly along these lines:

Even if you don't think in English, you probably still thought of a similar set of actions, except in your favorite language. Regardless of the language you chose, or the exact steps you took, taking out the garbage is a series of actions that terminates in the garbage being outside, and you being back inside, because of the actions you took.

Our thoughts are filled with brave, fierce, passionate actions: we live, we breathe, we walk, we talk, we laugh, we cry, we hope, we fear, we eat, we drink, we stop, we go, we take out the garbage. Above all else, we are free to do and to act. If we were all just rocks sitting in the sun, life might still be okay, but we wouldn't be free. Our freedom comes precisely from our ability to do things.

Of course our thoughts are also filled with nouns. We eat nouns, and buy nouns from the store, and we sit on nouns, and sleep on them. Nouns can fall on your head, creating a big noun on your noun. Nouns are things, and where would we be without things? But they're just things, that's all: the means to an end, or the ends themselves, or precious possessions, or names for the objects we observe around around us. There's a building. Here's a rock. Any child can point out the nouns. It's the changes happening to those nouns that make them interesting.

Change requires action. Action is what gives life its spice. Action even gives spices their spice! After all, they're not spicy until you eat them. Nouns may be everywhere, but life's constant change, and constant interest, is all in the verbs.

And of course in addition to verbs and nouns, we also have our adjectives, our prepositions, our pronouns, our articles, the inevitable conjunctions, the yummy expletives, and all the other lovely parts of speech that let us think and say interesting things. I think we can all agree that the parts of speech each play a role, and all of them are important. It would be a shame to lose any of them.

Wouldn't it be strange if we suddenly decided that we could no longer use verbs?

Let me tell you a story about a place that did exactly that…

The Kingdom of Nouns

In the Kingdom of Javaland, where King Java rules with a silicon fist, people aren't allowed to think the way you and I do. In Javaland, you see, nouns are very important, by order of the King himself. Nouns are the most important citizens in the Kingdom. They parade around looking distinguished in their showy finery, which is provided by the Adjectives, who are quite relieved at their lot in life. The Adjectives are nowhere near as high-class as the Nouns, but they consider themselves quite lucky that they weren't born Verbs.

Because the Verb citizens in this Kingdom have it very, very bad.

In Javaland, by King Java's royal decree, Verbs are owned by Nouns. But they're not mere pets; no, Verbs in Javaland perform all the chores and manual labor in the entire kingdom. They are, in effect, the kingdom's slaves, or at very least the serfs and indentured servants. The residents of Javaland are quite content with this situation, and are indeed scarcely aware that things could be any different.

Verbs in Javaland are responsible for all the work, but as they are held in contempt by all, no Verb is ever permitted to wander about freely. If a Verb is to be seen in public at all, it must be escorted at all times by a Noun.

Of course "escort," being a Verb itself, is hardly allowed to run around naked; one must procure a Verb Escorter to facilitate the escorting. But what about "procure" and "facilitate?" As it happens, Facilitators and Procurers are both rather important Nouns whose job is chaperoning the lowly Verbs "facilitate" and "procure," via Facilitation and Procurement, respectively.

The King, consulting with the Sun God on the matter, has at times threatened to banish entirely all Verbs from the Kingdom of Java. If this should ever to come to pass, the inhabitants would surely need at least one Verb to do all the chores, and the King, who possesses a rather cruel sense of humor, has indicated that his choice would be most assuredly be "execute."

The Verb "execute," and its synonymous cousins "run," "start," "go," "justDoIt," "makeItSo," and the like, can perform the work of any other Verb by replacing it with an appropriate Executioner and a call to execute(). Need to wait? Waiter.execute(). Brush your teeth? ToothBrusher(myTeeth).go(). Take out the garbage? TrashDisposalPlanExecutor.doIt(). No Verb is safe; all can be replaced by a Noun on the run.

In the more patriotic corners of Javaland, the Nouns have entirely ousted the Verbs. It may appear to casual inspection that there are still Verbs here and there, tilling the fields and emptying the chamber pots. But if one looks more closely, the secret is soon revealed: Nouns can rename their execute() Verb after themselves without changing its character in the slightest. When you observe the FieldTiller till(), the ChamberPotEmptier empty(), or the RegistrationManager register(), what you're really seeing is one of the evil King's army of executioners, masked in the clothes of its owner Noun.

Verbs in Neighboring Kingdoms

In the neighboring programming-language kingdoms, taking out the trash is a straightforward affair, very similar to the way we described it in English up above. As is the case in Java, data objects are nouns, and functions are verbs. (And variable names are proper nouns, attributes are adjectives, operators often serve as conjunctions, varargs are the pronoun "y'all," and so on. But this is beside the point of our story.)

But unlike in Javaland, citizens of other kingdoms may mix and match nouns and verbs however they please, in whatever way makes sense for conducting their business.

For instance, in the neighboring realms of C-land, JavaScript-land, Perl-land and Ruby-land, someone might model taking out the garbage as a series of actions—that is to say, verbs, or functions. Then if they apply the actions to the appropriate objects, in the appropriate order (get the trash, carry it outside, dump it in the can, etc.), the garbage-disposal task will complete successfully, with no superfluous escorts or chaperones required for any of the steps.

There's rarely any need in these kingdoms to create wrapper nouns to swaddle the verbs. They don't have GarbageDisposalStrategy nouns, nor GarbageDisposalDestinationLocator nouns for finding your way to the garage, nor PostGarbageActionCallback nouns for putting you back on your couch. They just write the verbs to operate on the nouns lying around, and then have a master verb, take_out_garbage(), that springs the subtasks to action in just the right order.

These neighboring kingdoms generally provide mechanisms for creating important nouns, when the need arises. If the diligent inventors in these kingdoms create an entirely new, useful concept that didn't exist before, such as a house, or a cart, or a machine for tilling fields faster than a person can, then they can give the concept a Class, which provides it with a name, a description, some state, and operating instructions.

The difference is that when Verbs are allowed to exist independently, you don't need to invent new Noun concepts to hold them.

Javalanders look upon their neighbors with disdain; this is the way of things in the Kingdoms of Programming.

If You Dig a Hole Deep Enough…

On the other side of the world is a sparsely inhabited region with kingdoms where Verbs are the citizens of eminence. These are the Functional Kingdoms, including Haskellia, Ocamlica, Schemeria, and several others. Their citizens rarely cross paths with the kingdoms near Javaland. Because there are few other kingdoms nearby, the Functional Kingdoms must look with disdain upon each other, and make mutual war when they have nothing better to do.

In the Functional Kingdoms, Nouns and Verbs are generally considered equal-caste citizens. However, the Nouns, being, well, Nouns, mostly sit around doing nothing at all. They don't see much point in running or executing anything, because the Verbs are quite active and see to all that for them. There are no strange laws mandating the creation of helper Nouns to escort each Verb, so there are only exactly as many Nouns as there are Things in each kingdom.

As a result of all this, the Verbs have the run of the place, if you'll pardon the expression. As an outsider, you could easily form the impression that Verbs (i.e., the functions) are the most important citizens by far. That, incidentally, is why they're called the Functional Kingdoms and not the Thingy Kingdoms.

In the remotest regions, beyond the Functional Kingdoms, lies a fabled realm called Lambda the Ultimate. In this place it is said that there are no nouns at all, only verbs! There are "things" there, but all things are created from verbs, even the very integers for counting lambs, which are the most popular form of trading currency there, if the rumors speak truth. The number zero is simply lambda(), and 1 is lambda(lambda()), 2 is lambda(lambda(lambda())), and so on. Every single Thing in this legendary region, be it noun, verb or otherwise, is constructed from the primal verb "lambda." (The meaning of the verb "lambda" is allegedly "to lambda.")

To be quite honest, most Javalanders are blissfully unaware of the existence of the other side of the world. Can you imagine their culture shock? They would find it so disorienting that they might have to invent some new nouns (such as "Xenophobia") to express their new feelings.

Are Javalanders Happy?

You might think daily life in Javaland would be at best a little strange, and at worst grossly inefficient. But you can tell how happy a society is through their nursery rhymes, and Javaland's are whimsically poetic. For instance, Javaland children often recite the famous cautionary tale:

It remains wonderful advice, even to this very day.

Although the telling of the tale in Javaland differs in some ways from Ben Franklin's original, Javalanders feel their rendition has a distinct charm all its own.

The main charm is that the architecture is there for all to see. Architecture is held in exceptionally high esteem by King Java, because architecture consists entirely of nouns. As we know, nouns are things, and things are prized beyond all actions in the Kingdom of Java. Architecture is made of things you can see and touch, things that tower over you imposingly, things that emit a satisfying clunk when you whack them with a stick. King Java dearly loves clunking noises; he draws immense satisfaction from kicking the wheels when he's trying out a new horse-drawn coach. Whatever its flaws may be, the tale above does not want for things.

One of our first instincts as human beings is to find shelter from the elements; the stronger the shelter, the safer we feel. In Javaland, there are many strong things to make the citizens feel safe. They marvel at the massive architectural creations and think "this must be a strong design." This feeling is reinforced when they try to make any changes to the structure; the architectural strength then becomes daunting enough that they feel nobody could bring this structure down.

In addition to the benefits of a strong architecture, everything in Javaland is nicely organized: you'll find every noun in its proper place. And the stories all take a definite shape: object construction is the dominant type of expression, with a manager for each abstraction and a run() method for each manager. With a little experience at this kind of conceptual modeling, Java citizens realize they can express any story in this style. There's a kind of "noun calculus" backing it that permits the expression of any abstraction, any computation you like. All one needs are sufficient nouns, constructors for those nouns, accessor methods for traversing the noun-graph, and the all-important execute() to carry out one's plans.

The residents of the Kingdom of Java aren't merely happy—they're bursting with pride!

StateManager.getConsiderationSetter("Noun Oriented Thinking," State.HARMFUL).run()

Or, as it is said outside the Kingdom of Java, "Noun Oriented Thinking Considered Harmful."

Object Oriented Programming puts the Nouns first and foremost. Why would you go to such lengths to put one part of speech on a pedestal? Why should one kind of concept take precedence over another? It's not as if OOP has suddenly made verbs less important in the way we actually think. It's a strangely skewed perspective. As my friend Jacob Gabrielson once put it, advocating Object-Oriented Programming is like advocating Pants-Oriented Clothing.

Java's static type system, like any other, has its share of problems. But the extreme emphasis on noun-oriented thought processes (and consequently, modeling processes) is more than a bit disturbing. Any type system will require you to re-shape your thoughts somewhat to fit the system, but eliminating standalone verbs seems a step beyond all rationale or reason.

C++ doesn't exhibit the problem, because C++, being a superset of C, allows you to define standalone functions. Moreover, C++ provides a distinct namespace abstraction; Java overloads the idea of a Class to represent namespaces, user-defined types, syntactic delegation mechanisms, some visibility and scoping mechanisms, and more besides.

Don't get me wrong—I'm not claiming C++ is "good." But I do find myself appreciating the flexibility of its type system, at least compared with Java's. C++ suffers from problems causing reasonable-looking sentences to cause listeners to snap and try to kill you (i.e., unexpected segfaults and other pitfalls for the unwary), and it can be extremely difficult to find the exact incantation for expressing a particular thought in C++. But the range of succinctly expressible thoughts far exceeds Java's, because C++ gives you verbs, and who'd want to speak in a language that doesn't?

Classes are really the only modeling tool Java provides you. So whenever a new idea occurs to you, you have to sculpt it or wrap it or smash at it until it becomes a thing, even if it began life as an action, a process, or any other non-"thing" concept.

I've really come around to what Perl folks were telling me eight or nine years ago: "Dude, not everything is an object."

It's odd, though, that Java (and arguably C#, due to its similar roots) appears to be the only mainstream object-oriented language that exhibits radically noun-centric behavior. You'll almost never find an AbstractProxyMediator, a NotificationStrategyFactory, nor any of their ilk in Python or Ruby.

Why do you find them everywhere in Java? It's a sure bet that the difference is in the verbs. Python, Ruby, JavaScript, Perl, and of course all Functional languages allow you to declare and pass around functions as distinct entities without wrapping them in a class.

It's certainly easier to do this in dynamically typed languages; you just pass a reference to the function, obtained from its name, and it's up to the caller to invoke the function with the proper arguments and use its return value correctly.

But many statically-typed languages have first-class functions as well. This includes verbosely-typed languages like C and C++, and also type-inferring languages like Haskell and ML. The languages just need to provide a syntax for creating, passing and invoking function literals with an appropriate type signature.

There's no reason Java couldn't simply add first-class functions and finally enter the grown-up, non-skewed world that allows people to use verbs as part of their thought processes. In fact there's a JVM language called “The Nice programming language” that sports a very Java-like syntax, but also includes expressive facilities for using verbs: standalone functions, which Java forces you to wrap with Callbacks or Runnables or other anonymous interface implementation classes to be able to refer to them.

Sun wouldn't even have to break their convention of requiring all functions to be "owned" by classes. Every anonymous function could carry an implicit "this" pointer to the class in which it was defined; problem solved.

I don't know why Sun insists on keeping Java squarely planted in the Kingdom of Nouns. I doubt it's a matter of underestimating their constituency; they added generics, which are a far more complex concept, so they clearly no longer care deeply about keeping the language simple. And that's not a bad thing, necessarily, because Java's established now: it makes more sense to start giving Java programmers tools that let them program the way they think.

I sure hope they fix this, so I can take the trash out and get back to my video game. Or whatever I was doing.

Author’s Note: Notes from the Mystery Machine Bus

I wrote this one just before this book went to press. It’s safe to say that I wrote it because the book was going to press. I didn’t feel like I could point to a single blog post that explained all my ranting, nor even a big fraction of it. To be honest, I don’t feel like I’d made much of an impact on anyone who didn’t already agree with me on these issues. The opposition was getting really good at pretending I didn’t exist—that we didn’t exist.

I was tired of being ignored.

So I struggled and pondered for a few days, and then wrote this post in a single day. It was a seventeen-hour day, to be sure, but it was basically one sitting with a few breaks here and there.

It’s a bit early to tell, but I think I made a splash. A lot of people disagree with me, both on the core issues and on my general characterization of the spectrum of views. But the model I’m presenting here is sort of magical, in that disagreeing with me just reinforces the model. It’s like a self-tightening philosophical noose.

I think it’s going to take years for it to settle in and become canon, but already, just in the last few weeks, a lot of people (both for and against my model) have acknowledged its likely inevitability.

It just works too damn well to be ignored. Thus solving my problem!

A few days later, I posted a sequel that’s not published here, and most of that is fairly irrelevant clarifications. But I did make a point there that I’d like to highlight here.

Before I created and published this conceptual model for software developers’ views, the ones that I’m calling “liberals” didn’t have a seat at the negotiating table in design discussions and implementation reviews. They were all too often simply dismissed as hacks, or lazy, or sloppy, or uncaring. We’re talking about a bald-faced dismissal of half the engineers in the world—including just about everyone at Facebook, the frontend engineers at Amazon and Yahoo, the engineering staff at reddit and many other favorite sites, the entire Rails community, and many many others. People said they were just getting lucky, dodging bullets.

I think that’s over now. I fixed it. It’ll take time, but it’s done. They have a seat at the table.

See what you think!

Oh, one final note: in the weeks following the post, we learned that the folks in one of the quadrants—political-liberal/software-conservative—were really unhappy with their characterization as “conservative”, to the point where they rejected the model entirely. I’ve thought long and hard about this, and my answer is: Tough shit. If you want to program conservatively, you need to live with the labels. Be proud and stand up for your beliefs!

Notes from the Mystery Machine Bus

I've spent the past eight years (starting back in June 2004) writing elaborate rants about a bunch of vaguely related software engineering issues.

I was doing all that ranting because I've been genuinely perplexed by a set of "bizarre" world-views held dear by—as far as I can tell—about half of all programmers I encounter, whether online or in person.

Last week, after nearly a decade of hurling myself against this problem, I've finally figured it out. I know exactly what's been bothering me.

In today's essay I'm going to present you with a new conceptual framework for thinking about software engineering. This set of ideas I present will be completely obvious to you. You will probably slap yourself for not having thought of it yourself. Or you might slap the person next to you. In fact you probably have thought of it yourself, because it is so blindingly obvious.

But in my thirty-odd years as a programmer I'm pretty sure this way of thinking about things, if it already existed, has never been mainstream. That assertion is backed by what has to be at least ten Google searches that turned up nothing. So I'm pretty confident.

I'm going to make it mainstream, right now. Watch!

And I suspect this conceptual framework I'm giving you will immediately become, and forever remain, one of the most important tools in your toolkit for talking with—and about—other programmers.

The punch line, a.k.a. TL;DR

I won't keep you in suspense. Here is the thesis of this long essay. It is the root cause that motivated over half of my ranting all these years, starting at Amazon and continuing here at Google.

(Note: I Do Not Speak For My Employer. This should be patently obvious. When employers want someone to speak for them, they hire a person like the Mouth of Sauron, to make absolutely sure everyone knows they are speaking for the Employer.)

My thesis:

1. Software engineering has its own political axis, ranging from conservative to liberal. Note: Technically, you could stop reading right here and be at pretty much 90 percent comprehension. In case you care.

2. The notions of "conservative" and "liberal" on this political axis are specialized to software engineering. But they exhibit some strong similarities to their counterparts in real-world politics.

3. Everyone in the software industry who does stuff related to programming computers falls somewhere fairly precise on this political spectrum, whether they realize it or not.

Put another way, you are either a liberal or a conservative software engineer. You may be more of a centrist, or maybe an extremist, but you fall somewhere on that left/right spectrum.

Just as in real-world politics, software conservatism and liberalism are radically different world views. Make no mistake: they are at odds. They have opposing value systems, priorities, core beliefs, and motivations. These value systems clash at design time, at implementation time, at diagnostic time, at recovery time. They get along like green eggs and ham.

I think it is important for us to recognize and understand the conservative/liberal distinction in our industry. It probably won't help us agree on anything, pretty much by definition. Any particular issue only makes it onto the political axis if there is a fundamental, irreconcilable difference of opinion about it. Programmers probably won't—or maybe even can't—change their core value systems.

But the political-axis framework gives us a familiar set of ideas and terms for identifying areas of fundamental disagreement. This can lead to faster problem resolution. Being able to identify something quickly as a well-defined political issue means we can stop wasting time trying to convince the other side to change their minds, and instead move directly into the resolution phase, which (just as in politics) generally boils down to negotiation and compromise. Or, you know, Watergate.

Are real politics and software politics correlated?

Does being a political conservative make you more likely to be a software conservative? I think all we have are hunches and intuition at this point. We'd need studies for an answer with any scientific basis.

But my hunch tells me that they are only loosely correlated, if at all. I suspect that the four possible combinations of political left/right and software left/right each contain around a quarter of the programming population, plus or minus maybe 8 percent. But that's a totally non-scientific hunch, and I'd love to see the real numbers.

The reason I suspect they're only loosely correlated is that I think a person's software-political views are largely formed by two forces:

1. Their personal early/formative experiences as a programmer—in particular, what kinds of decisions either burned them or made them wildly successful.

2. The software-political core values of their teachers, professors, mentors, role models, and coworkers.

But there are some factors that could increase correlation between political orientation and software orientation. One is geographic/regional influence—for instance, big tech universities being located primarily in red or blue states. Another might be basic personality traits that may incline people toward traditionally liberal or conservative value systems.

Regardless of how correlated they are, there are going to be a lot of programmers out there who are mildly shocked to find that they are political conservatives but software liberals, or vice-versa.

Isn't the two-party system a flawed model?

Well, yeah. Mapping something as nuanced as political viewpoints onto a one-dimensional axis is a well-known gross oversimplification. For instance, some U.S. voters may self-identify as fiscally conservative but socially liberal, and as a result may wind up not having a strong tie to either majority political party. These "swing votes" tend to be the reason that political moderates more often wind up winning elections, even though studies have shown that people tend to assign more credibility to extreme opinions.

The influence of centrists or moderates is just as important in the software engineering world, and we'll explore it more in a bit.

But having a single axis still allows for a wide distribution on both the right and the left. As in real politics, a moderately right-leaning viewpoint may seem dishearteningly liberal to someone who lies even further toward the extreme right. I will illustrate with some well-known real-world programming language scenarios in the examples section.

All in all, despite its oversimplifications, I think the two-party model is a good starting point for educating people about the politics of software. It'll be a good frame of reference for refining the model down the road, although that's out of scope for this essay.

So what's a Software Liberal slash Conservative?

If you ignore specific real-world issues, and just focus on the underlying traits and values of real-world conservatives and liberals, it boils down to just a few themes. I'll argue that those underlying themes are also the basis for software politics.

It's easiest to talk first about conservatives, and then define liberals in terms of what conservatives are not. This is because conservatives tend to have a unified and easily-articulated value system, whereas liberals tend to be more weakly organized and band together mostly as a reaction to conservatism. This applies to both real-world and software-world politics.

So we'll start with an operational definition of conservatism, from Jost et al.:

"We regard political conservatism as an ideological belief system that is significantly (but not completely) related to motivational concerns having to do with the psychological management of uncertainty and fear."

This theory is explored, re-tested, and affirmed in a 2008 study from Garney et. al, "The Secret Lives of Liberals and Conservatives: Personality Profiles, Interaction Styles, and the Things They Leave Behind."

I hope you'll agree that this definition is minimally controversial. After all, the adjective "conservative" is more or less synonymous with caution and risk aversion. Financial conservatism is frequently (and intuitively) associated with age and with wealth. Companies tend to grow more conservative with age as they weather the storms of lawsuits, technical fiascoes, dramatic swings of public opinion, and economic downturns. We even have fables about ants and grasshoppers to teach us about conserving food for the upcoming winter.

Conservatism, at its heart, is really about risk management.

Similarly, liberal views are often associated with youth, with idealism, with naivete. In the corporate world, we think of startups as being prototypically liberal—in part because they're setting out to change the world in some way (and liberalism is traditionally associated with change), and in part because they have to go all-out in order to hit their scheduled funding milestones, which can justify cutting corners on software safety.

Liberalism doesn't lend itself quite as conveniently to a primary root motivation. But for our purposes we can think of it as a belief system that is motivated by the desire above all else to effect change. In corporate terms, as we observed, it's about changing the world. In software terms, liberalism aims to maximize the speed of feature development, while simultaneously maximizing the flexibility of the systems being built, so that feature development never needs to slow down or be compromised.

To be sure, conservatives think that's what they're maximizing, too. But their approach is… well, conservative. Flexibility and productivity are still motivators, but they are not the primary motivators. Safety always trumps other considerations, and performance also tends to rank very highly in the software-conservative's value system.

The crux of the disagreement between liberals and conservatives in the software world is this: how much focus should you put on safety? Not just compile-time type-safety, but also broader kinds of "idiot-proofing" for systems spanning more than one machine.

Let's characterize this core disagreement with some example statements that would be rated with much higher importance by conservatives than by liberals:

- Software should aim to be bug free before it launches. (Banner claim: "Debugging Sucks!") Make sure your types and interfaces are all modeled, your tests are all written, and your system is fully specified before you launch. Or else be prepared for the worst!

- Programmers should be protected from errors. Many language features are inherently error-prone and dangerous, and should be disallowed for all the code we write. We can get by without these features, and our code will be that much safer.

- Programmers have difficulty learning new syntax. We should limit the number of languages used at our company, so that nobody has to learn a new syntax when a system goes down in the middle of the night on Christmas Eve. And we should never permit features that allow defining new syntax, nor changing the semantics of existing syntax. (Common examples: no operator overloading, and no metaprogramming!)

- Production code must be safety-checked by a compiler. Any code that cannot be statically checked should in general be avoided. In specific cases where it is strictly necessary, uses of it must be approved by a central committee. (Examples: eval, dynamic invocation, RTTI).

- Data stores must adhere to a well-defined, published schema. Relational databases must be in third-normal form and adhere to a UML or equivalent schema definition. XML should have a DTD. NoSQL databases and name/value stores—both of which should be avoided in general—must have a separate physical schema that defines all permissible keys and their corresponding value types.

- Public interfaces should be rigorously modeled. Data should never be stored in strings or untyped collections. All input and output entities should be thoroughly and explicitly specified via statically-checkable, ideally object-oriented models.

- Production systems should never have dangerous or risky backdoors. It should never be possible to connect to a live production system via a debugger, telnet shell, nor any other interface that allows the developer to manipulate the runtime operation of the code or data. The only ports into a production system should be read-only monitoring channels.

- If there is ANY doubt as to the safety of a component, it cannot be allowed in production—no matter how teams may cry and wail that they need it to make forward progress. (I'm talkin' to you, FUSE).

- Fast is better than slow. Everyone hates slow code. Code should perform well. You should engineer all your code for optimum speed up front, right out of the box. Otherwise it might not be fast enough. Avoid using languages or DSLs or libraries that have a reputation for being slow. Even if they're fast enough for your current purposes, the requirements (or callers) could change, and suddenly the software would be too slow!

Of course these examples are meant to be illustrative rather than comprehensive. And obviously not all of them may necessarily be espoused by everyone who self-identifies as a conservative. But barring minor exceptions, they are all very common conservative viewpoints.

To decide whether a system or a technology is liberal or conservative, just think of a bunch of independent "safety checkboxes." If a majority of them are checked, then the system is conservative as a whole. The more that are checked, the more conservative it is.

By way of comparison, here are the Liberal versions of the nine examples above. It might help to think of Scooby-Doo characters. The examples above are all from Fred, and the examples below are from Shaggy.

- Bugs are not a big deal. They happen anyway, no matter how hard you try to prevent them, and somehow life goes on. Good debuggers are awesome pieces of technology, and stepping through your code gives you insights you can't get any other way. Debugging and diagnosing are difficult arts, and every programmer should be competent with them. The Christmas Eve Outage scenario never, ever happens in practice—that's what code freeze is for. Bugs are not a big deal! (This belief really may be the key dividing philosophy between Conservative and Liberal philosophies.)

- Programmers are only newbies for a little while. The steady state for a programmer's career is being smart, knowledgeable, creative, resourceful, and experienced. Putting a bunch of rules in place to protect newbies from doing harm (or being harmed) is incorrectly optimizing for the transient case instead of the steady state.

- Programmers figure stuff out amazingly fast when their jobs depend on it. People learn to read sheet music, braille, sign language, and all sorts of other semiotic frameworks. Hell, even gorillas can apparently do all that. Programmers don't need protection from syntax. They just need documentation and a little slack time to read up on it.

- Succinctness is power. Code should be kept small. Period. If your static checking tools can't reason about the code, then the checking needs to be made smarter (e.g., by incorporating runtime data) rather than making the code dumber.

- Rigid schemas limit flexibility and slow down development. Lightweight/partial/optional schemas are a better tradeoff. Moreover, the schema is often not well-understood until a lot of data is collected and a lot of use cases are thoroughly exercised. So the schema should follow the code rather than precede it.

- Public interfaces should above all else be simple, backward-compatible, and future-compatible. Rigorous modeling is just guessing at how the interface will need to evolve. It makes both forward- and backward-compatibility almost impossible, resulting in interface churn and customer unhappiness. Public interfaces should always do the simplest thing that could possibly work, and grow only as needed.

- System flexibility can mean the difference between you getting the customer (or contract) vs. your competitor nabbing it instead. Security and safety risks in runtime production systems can be mitigated and controlled by logging, monitoring and auditing. There are plenty of existence-proofs of large systems with root-access backdoors and shells (e.g., RDBMS, online game servers) whose risk is controlled while still giving them world-class runtime flexibility.

- Companies should take risks, embrace progress, and fiercely resist ossification. It doesn't matter how big your business is: it must grow or die. If you want to stay competitive, you have to make a conscious, often painful effort to take risks. Which means you'll need good recovery techniques for the inevitable disasters. But you need those even if you don't take risks. So take risks!

- Premature optimization is the root of all evil. Get the code working first, focusing on correctness over performance, and on iterative prototyping over correctness. Only when your customers list latency as the top priority should you begin performing profiler-driven optimizations.

There you have it: Shaggy vs. Fred.

Just as in real-world politics, software liberals are viewed by conservatives as slovenly, undisciplined, naive, unprincipled, and downright "bad" engineers. And liberals view conservatives as paranoid, fear-mongering, self-defeating bureaucrats.

Once again, although I don't think the two camps will ever agree, I do think that mutual understanding of the underlying value systems may help the camps compromise.

Or at the very least, the conservative/liberal classification should help the two camps steer clear of each other. I think it is probably better to have a harmonious team of all-liberals or all-conservatives than a mixed team of constantly clashing ideologies. It's a lot like how vehicle-driving philosophies can differ regionally—it's okay if everyone drives in some crazy way, as long as they all drive that way.

So Security Engineers are the most conservative, then, right?

Wrong! The reality here goes against intuition, and I mention it to show how poorly our intuition fares when it comes to interpolating software-political views from job descriptions.

Security engineers are deeply concerned with risk assessment and attack-surface management, so you might well guess that they would naturally tend towards software conservatism.

In practice, however, security engineers tend to be keenly aware of the tradeoffs between caution and progress, since they have to spend big parts of their day meeting with software teams who are often too busy (not to mention woefully under-informed) to spend a lot of time on security. Security engineers learn quickly to make well-informed and practical risk-management decisions, rather than conservatively always trying to be as safe as humanly possible.

Hence many security engineers are in fact software liberals. Some just swing that way.

Many of the security engineers at Google happen to be fans of Ruby—both as an intrinsically secure language, and also as a nice, expressive language for writing auditing scripts and other security analysis tools. It wound up being fairly easy for me to get security sign-off for using Ruby in production for my first project at Google. In contrast, it was almost career-endingly difficult for me to get the same sign-off from our highly conservative systems programmers.

The reality is that almost any programming specialization is going to be divided into liberal and conservative camps. There are left- and right-wing versions of web developers, persistence engineers, protocol engineers, serving-system engineers, and most other sub-genres of programming.

I'm hard-pressed to think of any domains that tend to be mostly-liberal or mostly-conservative, with the sole exception of Site Reliability Engineering, which is sort of by definition a conservative role. For most of the other domains that initially came to mind—for instance, data analysts, which at first I thought were uniformly liberal—I've been able to think of specific teams or subdomains composed entirely of one or the other.

So in the end I think it comes down to personal preference. That's it. I don't think it's as domain-driven as much as it is personality-driven. Some people are just liberal, and some are just conservative, and that's how they are.

BIG FAT DISCLAIMER: Before I continue, I will now issue the important disclaimer that I am a hardcore software liberal, bordering on (but not quite) being a liberal extremist.

This fact, by and large, is what has driven me to pen many of my most infamous and controversial rants, including (among others) "Execution in the Kingdom of Nouns," "Portrait of a N00b," "Rhinos and Tigers," "Code's Worst Enemy," "Scheming is Believing," "Tour de Babel," "Dynamic Languages Strike Back," "Transformation," "Haskell Researchers Announce Discovery of Industry Programmer Who Gives a Shit," and many others besides.

Sure, I sometimes write about other stuff. My in-progress "Programmers View of the Universe" series is motivated by an entirely different bugbear that has nothing at all to do with software politics. My "Universal Design Pattern" article is about a technique that transcends software politics and can be applied with equal effectiveness in either a conservative or a liberal manner. And I've written about video games, Japanese anime, and random other stuff.

But I've come to understand that liberalism underlies a tremendous amount of my thinking. Even when I'm writing about management, I see that I am a liberal manager rather than a conservative one. And despite being both relatively old and relatively wealthy, I am also a political liberal—both socially and fiscally.

Nevertheless I am going to try to represent both sides fairly and accurately in this essay. You know, mostly. I think it's most important for you to buy in to the left/right distinction itself. I think it's far less important whether you happen to agree with my particular side.

I'll consider this essay a success if a sufficient number of you agree that the liberal/conservative distinction is valid for software engineering, and that my particular left/right classification of various technologies and philosophies below seems intuitively reasonable. I'm fine with declaring success if we disagree on a few small details, as long as we agree on the overall picture.

So… static typing enthusiasts are conservative, right?

Why yes. Yes, they are. Static typing is unquestionably one of the key dividing software-political issues of our time. And static typing is a hallmark of the conservative world-view.

In the conservative view, static typing (whether explicit or inferred) is taken on faith as an absolute necessity for modern software engineering. It is not something that one questions. It is a non-issue: a cornerstone of what constitutes the very definition of Acceptable Engineering Practice.

In the liberal's view, static typing is analogous to Security Theater. It exists solely to make people feel safe. People (and airports) have proven time and again that you can be just as statistically secure without it. But some people need it in order to feel "safe enough."

That's a pretty big difference of opinion—one I'm sure you can relate to, regardless of how you feel about it.

I'm not going to try to defend my view here, since that's what all those old blog posts were for. If I haven't convinced you by now, then there's not much point in continuing to try. I respect your opinion. Well, now. And I hope you now have a little better understanding of mine.

I should, however, mention that there is an unrelated (i.e., politically-neutral) point on which both camps agree: namely, that static types yield better toolchain support. This is undeniably true today, and I have made it my life's work to ensure that it is not true tomorrow.

I have spent the last four years championing an initiative within Google called the "Grok Project," one that will at some point burst beyond our big walled garden and into your world. The project's sole purpose in life is to bring toolchain feature parity to all languages, all clients, all build systems, and all platforms.

(Some technical details follow; feel free to skip to the next section heading… )

My project is accomplishing this lofty and almost insanely ambitious goal through the (A) normative, language-neutral, cross-language definitions of, and (B) subsequent standardization of, several distinct parts of the toolchain: (I) compiler and interpreter Intermediate Representations and metadata, (II) editor-client-to-server protocols, (III) source code indexing, analysis, and query languages, and (IV) fine-grained dependency specifications at the level of build systems, source files, and code symbols.

Okay, that's not the whole picture. But it's well over half of it.

Grok is not what you would call a "small" project. I will be working on it for quite some time to come. The project has gone through several distinct life cycle phases in its four years, from "VC funding" to "acceptance" to "cautious enthusiasm" to "OMG all these internal and even external projects now depend critically on us." Our team has recently doubled in size, from 6 engineers to 12. Every year—every quarter—we gain momentum, and our code index grows richer.

Grok is not a confidential project. But we have not yet talked openly about it, not much, not yet, because we don't want people to get over-excited prematurely. There is a lot of work and a lot of dogfooding left to do before we can start thinking about the process for opening it up.

For purposes of this essay, I'll assert that at some point in the next decade or so, static types will not be a prerequisite for world-class toolchain support.

I think people will still argue heatedly about type systems, and conservatives may never be able to agree amongst themselves as to which type system approach is best for modeling program behavior. But at least I will have helped that discussion be only about representations. The quality of the corresponding developer tools should no longer be a factor in the discussions.

Dividing up the Space

I'm going to go through and toss a bunch of random technologies, patterns, designs, and disciplines each into one of six buckets: "apolitical," "conservative," "centrist," "liberal" buckets, plus two buckets that start Centrist and head Left or Right in the presence of overuse. One bucket per thingy. Hey, it's not an exact science. But it should help set the rough foundation for the more detailed use cases below.

Non-political Stuff: Algorithms, data structures, concrete mathematics, complexity analysis, information theory, type theory, computation theory, and so on. Basically all CS theory. These disciplines occasionally inspire tempest-in-a-teapot butthurtedness in academia, but when it happens, it's just similar fish in too small a tank biting on each other. It's to be expected. Overall, these essentially mathematical disciplines are timeless, and they are all equally applicable to both the Liberal and Conservative programming worlds. Yes, even type theory.

Conservative Stuff: Provably sound type systems. Mandatory static type annotations. Nonpublic symbol visibility modifiers (private/protected/bewareofdog/etc.). Strict, comprehensive schemas. all-warnings-are-errors. Generics and templates. Avoidance of DSLs (XPath, regexps) in favor of explicit DOM manipulation and hand-rolled state machines. Build dependency restrictions. Forced API deprecation and retirement. No type equivalence (i.e., no automatic conversions) for numeric types. Checked exceptions. Single-pass compilers. Software Transactional Memory. Type-based function overloading. Explicit configuration in preference to convention. Pure-functional data structures. Any kind of programming with the word "Calculus" in it.

Centrist (or flat-out Neutral) Stuff: Unit testing. Documentation. Lambdas. Threads. Actors. Callbacks. Exceptions. Continuations and CPS. Byte-compilation. Just-in-time compilation. Expression-only languages (no statements). Multimethods. Declarative data structures. Literal syntax for data structures. Type dispatch.

Liberal Stuff: Eval. Metaprogramming. Dynamic scoping. all-errors-are-warnings. Reflection and dynamic invocation. RTTI. The C preprocessor. Lisp macros. Domain-specific languages (for the most part). Optional parameters. Extensible syntax. Downcasting. Auto-casting. reinterpret_cast. Automatic stringification. Automatic type conversions across dissimilar types. Nil/null as an overloaded semantic value (empty list, empty string, value-not-present). Debuggers. Bit fields. Implicit conversion operators (e.g., Scala's implicits). Sixty-pass compilers. Whole-namespace imports. Thread-local variables. Value dispatch. Arity-based function overloading. Mixed-type collections. API compatibility modes. Advice and AOP. Convention in preference to explicit configuration.

Centrist Stuff that Becomes Conservative If Taken Far Enough: Type modeling. Relational modeling. Object modeling. Interface modeling. Functional (i.e., side-effect-free) programming.

Centrist Stuff that Becomes Liberal if Taken Far Enough: Dynamic class loading and dynamic code loading. Virtual method dispatch. Buffer-oriented programming.

Woah, that exercise was surprisingly fun! It's far from complete, but hopefully you get the idea.

Some natural themes arise here:

- Implicit is generally liberal; explicit is generally conservative.

- Performance-oriented is generally conservative; late-optimized is generally liberal.

- Compile-time binding is generally conservative; runtime/late binding is generally liberal.

- Concurrency and parallelism in general seem to be politically charged topics, but the disagreement is orthogonal to the liberal/conservative camps.

I'd love to keep going with the classification. But I'll stop here, since we've got higher-level stuff to discuss.

Examples and Case Studies

I'll walk you through a bunch of examples to show you how just widespread and deep-rooted this political phenomenon is.

Example 1: Languages

Here are some very rough categorizations. Note that within each language camp there are typically liberal and conservative sub-camps. But as a whole, language usage tends to be dominated by what the language makes possible (and easy), so the culture tends to follow the features.

This list is just a few representative examples to give you the flavor. I'm only listing general-purpose languages here, since DSLs and query languages are typically feature-restricted enough to be hard to categorize.

Batshit liberal: Assembly language

Extremist liberal: Perl, Ruby, PHP, shell-script

Hardcore liberal: JavaScript, Visual Basic, Lua

Liberal: Python, Common Lisp, Smalltalk/Squeak

Moderate-liberal: C, Objective-C, Scheme

Moderate-conservative: C++, Java, C#, D, Go

Conservative: Clojure, Erlang, Pascal

Hardcore conservative: Scala, Ada, OCaml, Eiffel

Extremist conservative: Haskell, SML

These are my own categorizations based on my own personal experiences with these languages and their respective communities. Your mileage may vary. However, I'd be quite surprised if you chose to move any of these languages more than a step or two away from where I've positioned it.

One thing that jumps out is that a language doesn't have to be statically-typed or even strongly-typed in order to be conservative overall. More on that in a bit.

The next thing you might notice from the list is that the liberal and moderate languages are all pretty popular, and that popularity declines sharply as languages head into conservative territory.

I think this has a simple explanation: It's possible to write in a liberal language with a conservative accent, but it's very hard (and worse, discouraged) to write in a conservative language with a liberal accent.

For instance, it's straightforward to write JavaScript code in a way that eschews reflection, eval, most automatic type casting, prototype inheritance, and other dynamic features. You can write JavaScript that plods along as unadventurously as, say, Pascal. It doesn't have all the static type annotations, but you can replace them with assertions and unit tests and stereotypically stolid code organization.

But if you try writing your Haskell code with a bunch of dynamic features, well, you're in for a LOT of work. Haskell enthusiasts have managed to implement dynamic code loading and a ton of other ostensibly dynamic features, but it was only through herculean effort.

What's more, if you write your liberal-language code in a conservative way, people will just look at it and say: "Well, it's kinda boring, and you could have saved a lot of coding by using some dynamic features. But I guess it gets the job done. LGTM."

Whereas if you write your conservative-language code in a liberal way, you run the risk of being ostracized by your local language community, because… why are you doing all that dangerous dynamic stuff in the first place? I'll explore this cultural phenomenon further when I talk about Clojure below.

The last big, interesting observation from the list is that a lot of the most popular languages out there are only moderately conservative—even if they think of themselves as quite conservative compared to their ultra-dynamic cousins.

I've said it before, and it bears repeating here: the reason C++, C#, and Java have been particularly successful in the marketplace is that—just like effective politicians—they know how to play both sides.

C++ allows liberal-biased programmers to program in straight C. And it allows conservative-biased programmers to layer in arbitrary amounts of static type modeling, depending on how much work they want to expend in order to feel secure. Java? Pretty much the same story.

Playing to both the fast-and-loose and lock-your-doors mindsets has proven to be a key ingredient to market success. Also marketing, but it helps a LOT to be viewed as philosophically friendly by both the liberal and conservative camps.

There is a new crop of languages on the horizon (for instance, Google's Dart language, but also new specs for EcmaScript) that are deliberately courting the centrist crowd—and also delicately playing to grab both the liberals and conservatives—by offering optional static types. In principle this is a sound idea. In practice I think it will come down to whether the marketing is any good. Which it probably won't be.

Language designers always seem to underestimate the importance of marketing!

Example 2: Tech corporations

Just for fun, let's contrast four similar-ish tech companies in their software-political outlook.

1) Facebook—Diagnosis: Extremist Liberal. Despite their scale, they are still acting like a startup, and so far they've been getting away with it. They use primarily C++ and PHP, and they're prone to bragging about how their code calls back and forth from PHP to C++ and back into PHP, presumably bottoming out somewhere. Their datastore is memcached: just name-value pairs. No schema. They dump the data and logs into a backend Hive store and run Hadoop mapreduces for offline data analysis. They still hold all-night hackathons every other week or so, which will remain feasible for them as long as the majority of their programmers are very young males (as was the case last time I toured there) and their stock continues to promise great riches (as was not so much the case last I checked.) As a company they are tightly knit and strongly biased for action, placing a high value on the ability of individual programmers to launch features to their website with little to no bureaucracy or overhead. This is pretty remarkable for a company as big as they are, with as many users as they have. Conservatives no doubt regard them with something between horror and contempt. But Facebook is proving that programmers of any world-view can get a hell of a lot accomplished when they gang up on a problem.

2) Amazon.com—Diagnosis: Liberal. Which is surprising, given how long they've been in business, how much money is at stake, how mature their Operations division is, and how financially conservative they are. But "Liberal" is actually quite a retreat compared to their early days. Back in 1998 and 1999 they were almost exactly like Facebook is today, with the sole exception that they put everything in relational databases and did a ton of up-front relational data modeling. Well, except in Customer Service Apps, where we used a name/value store just to be flexible enough to keep up with the mad chaotic scramble of the business launches. All part of my multi-decade indoctrination as a Liberal. In any case, despite many corporate improvements with respect to work-life balance (which happened after several stock plunges and years of significant double-digit turnover in engineering), Amazon has retained its liberal, startup-like engineering core values. Every team owns their own data and makes their own decisions, more or less like independent business units. Amazon still launches and executes faster than just about anyone else out there, because they're still willing to take real risks (incurring occasional huge outages), and to make hard decisions in favor of launching early and often. Above all else, Amazon has proven conclusively that after fifteen years, they can still innovate like nobody else. They've still got it.

3) Google—Diagnosis: Conservative. They began life as slightly liberal and have grown more conservative ever since. Google was only software-liberal in the very very early days, back when the search engine itself was written in Python. As they grew, they quickly acquired a software conservatism driven entirely by the engineers themselves. Manifestos were written about the dangers of using multiple languages, and strict style guides were put in place to severely limit "risky" or "hard to read" language features of the few languages they did allow. Google's JavaScript code is written in an extremely conservative style with extensive static type annotations, and eval is forbidden. The Python style guide forbids metaprogramming and other dynamic features, which makes their Python look a lot like untyped Java. And they have severely limited the use of many C++ language features, with C++11 support rolling out literally one feature every few weeks. (There are over five hundred new features in C++11.) In internal surveys, Google engineers commonly cite bureaucracy, churn, and complexity as core obstacles to feature advancement and rapid launches. Google has made serious attempts on several occasions to reduce this bureaucracy, but they always get pushback from—surprise—the engineers themselves, who have grown so staunchly conservative that they actively (and even more often, passively) resist the introduction of more flexible stacks and technologies. Most of the major technological shifts within Google over the past half-decade have been overtly conservative. For a liberal like me, it has been a very sad process to observe. But at least I've found myself a niche that's widely regarded (by both camps) as valuable, and within my own org we can still be pretty liberal and get away with it.

4) Microsoft—Diagnosis: Batshit Conservative. Microsoft has two geese that lay golden eggs: Office and Windows. Microsoft has been reduced to a commercial farmer protecting the geese from all incursions. The golden eggs still have value, because customers are locked into the platform by the cost-ineffectiveness of retraining their fleets. But Microsoft can no longer innovate in Office or Windows precisely because of those corporate fleet retraining costs. Their OEMs are stretched as thin as they can go. Apple is dominating the handheld markets, and Microsoft is actively stifling their own innovation in Windows Phone because they're afraid it will cannibalize their core Windows business. Microsoft has not had a successful product-level innovation in 15, maybe 20 years. All of their successful products have been copies of competitors' products: IE, XBox, C#, .NET, Bing, Windows Phone, and so on ad infinitum. All great implementations of someone else's ideas. Microsoft's playbook is to embrace, extend, and leverage their brand to crush the competition—or at least it was, until the government put an end to that circa 2002. Now the company genuinely doesn't know what the fuck to do with themselves, and what's more, instead of Bill Gates, they now have a lunatic in charge. Employees are leaving in droves, all citing the same internal "existential crisis" and unbearable corporate politics caused by competing business units actively sabotaging one another. Microsoft has turned into a caricature of right-wing corporatism: sitting on their front porch with a shotgun cursing at passers-by, waiting for their government bribes to give them another few years of subsidies and shelters while they wait to die. I've personally chatted with close to four hundred current and ex-Microsoft employees over the past seven years. Oh, the stories I could tell you… someday, maybe.

5) Bonus company: Apple. Diagnosis: no idea, but they're so good at marketing that it's almost irrelevant. Would love to have more insight into their internal software culture, though. Any takers? Throwaway accounts? AMA?

Okay, that was a fun exercise too. But we need to move on! Almost done now.

Specific Case Study: The Clojure Language

I've been meaning to follow up on Clojure for a while now. Over a year, at least. But until recently I didn't have the conceptual tools to explain what I wanted to say about it.

Now I do!

Clojure is a new-ish Lisp dialect that runs on the JVM and .NET, and I was honored to write the Foreword to The Joy of Clojure a while back. For a few years I had been really excited to start learning Clojure, and my initial experiences with it were quite positive.

However, I eventually learned that the Clojure community is extremely conservative. That is is pretty unusual for a Lisp dialect. Lisp is widely regarded as one of the most liberal language families in existence. And Clojure has the superficial appearance of being a laissez-faire kind of language. It is quite expressive, including—ahem—a liberal dose of new syntax. And it eschews static type annotations and strong type modeling in favor of a small set of highly regular, composable data types and operations—not unlike, say, Scheme or Python.

But the resemblance to a liberal language ends there. Clojure's community came pre-populated with highly conservative programmers from the pure-functional world: basically Haskell/ML types (lots of puns today!) who happen to recognize the benefits of Lisp's tree syntax. So under its expressive covers, everything about Clojure is strongly conservative, with a core overriding bias towards protecting programmers from mistakes.

And the community follows suit. At a Clojure conference last year (or was it two years ago? time flies so fast these days…), there was a key presenter doing a talk on how Macros were basically harmful and should be avoided in modern Clojure code.

I trust that if you know anything about Lisp, your blood is basically boiling at this point. I know mine was.

But his argument is perfectly valid from the classic software-conservative's viewpoint. Macros allow you to invent domain-specific language abstractions. Those require documentation in order for users to figure out what they do, and what they mean. That means you can know Clojure and not really know enough to read someone's Clojure code without some documentation handy. Which is sort of the definition of a newbie. And there you have it. In a very real sense, conservatives fear being turned—in the blink of an eye—from masters into newbies by the application of macros. (Author’s note: This was an accidental misrepresentation of their position, and is thus not a good illustrative example. However, it’s irrelevant: I stand by my assertion, one later validated offline by important members of their own community, that the Clojure community is highly conservative.)

It's sort of scary if you have a lot of your personal identity invested in knowing some shit. And wouldn't you know it, real-world politics conservatives are shown in study after study to be "staunch" in sticking to their world views rather than compromising. That means they have a lot of identity tied up in those views.

So while I liked a lot of what I saw in Clojure, as a hardcore software liberal I was inevitably and inexorably driven away from the language. And that's good for me, and it's good for Clojure. I mean, why should they compromise?

I think that my conceptual framework gives us an "out"—a way to avoid being emotional about these subjects. Casting the problem as a clash between Liberalism and Conservatism gives us the ultimate ticket for "agreeing to disagree."

Hopefully it will also help language designers and communities do a better job of targeted marketing. Right now just about every language out there makes a claim along the lines of "This language is the best choice for everybody!" But now we know that is very unlikely to be true—or if it is, we can at least be assured that they're centrists, and they run the risk of being equally distasteful to everybody.

In the conservative/liberal framework, language designers can make more accurate, less bait-and-switchy claims; for instance: "Haskell is the best choice for every radical extremist conservative programmer!"

Well, we can work on the wording. But you get the idea.

Wrap-Up

I was thinking of going through a bunch more examples and stuff, but I see that I'm on my third (Editor's Note: fourth) glass of wine, which means my typing is about to give out any minute.

So let's wrap it up!

There's one kinda key point I wanted to get across, but didn't see a good place for it. That point is this: please do not be alarmed that I am calling you a (software) Conservative.

I worry that politically left-leaning programmers will hear the term "conservative" and will immediately associate it with all of the… uh, politically-charged connotations associated with far right-wing conservatism in the United States political arena today. You know, racism, sexism, religious fundamentalism, homophobia, warmongering, bear-shooting, that kind of thing.

I'm not saying they're bad, at least not in this essay. I'm just saying nobody in their right mind wants to be associated even remotely with those embarrassing wingnuts. See what fine-grained, nuanced distinctions three (Editor's Note: four) glasses of wine can produce? But I'm not saying their views are bad. No. Not here. I'm just observing that they're heavily politically charged viewpoints which have, for better or worse, recently come to be associated with the term "conservatism" in US politics.

So please do me a favor and try to dissociate those specific agendas and real-world political viewpoints from the generic term "Conservative," which here really just means risk averse.

It's perfectly okay, and normal, to be a programming conservative. You don't have to shoot any bears. I would actually like to see the terms liberal and conservative to become badges of honor in the programming world. People should stand behind their beliefs. I mean, we do already, I think, so it shouldn't be much of a stretch to allow our beliefs to be given convenient labels.

Ah, me. I can see the Euphemism Treadmill rearing its ugly head already. We'll see.

Anyway, tell me what you think! I welcome any and all viewpoints and comments. Even from bears!

Special thanks to Writer's Block Syrah for reviewing this post for career suicide.

(Special Note to my fellow Googlers: Yes, I meant to post this externally. BOTH times. No, I am not the Mouth of Sauron.)

Author’s Note: Moore’s Law Is Crap

This is a slightly different take on the language adoption problem, focused on parallel computing. This post has a very personal flavor. It’s one of my all-time personal favorites. Lots of neat stories in here.

I see that I published it six days before I published “Execution in the Kingdom of Nouns.” I must have been in a really ornery mood that month.

For what it’s worth, Google’s been making slow but steady progress on the problem I’m complaining about. They still use way too much C++, and way too much Java. But they’re getting a lot better over the years at getting more work out of more machines with a lot less code. At this rate we might actually have something nice in a few generations.

Moore’s Law Is Crap

Sometimes people ask me how I find time to go learn new stuff. Here's the answer: you make time.

Nobody ever likes that kind of answer, but we all know it's the real one.

My brother Dave put on some pretty serious weight after he graduated from high school. He'd gone from playing varsity football to working two jobs and going to college full time. It didn't help that one of his jobs was pizza delivery, and the other was waiting tables. Soon he was a fat, fat kid. Went from a buck-eighty to a deuce and a half, at least, maybe a deuce sixty.

One day he saw a truck with a bumper sticker on it that said: "Lose weight now, ask me how!" So he pulled up next to the truck at the next stoplight, and said to the two cowboy-types in it: "How do I lose weight?" They yelled back: "Just lose weight, ya fat pig! Haw haw haw HAW HAW HAW HAW!" and then drove off.

Dave was sad about this advice for a brief while, but eventually he brightened up, because he did know how to lose weight. Cripes, he'd been a champion varsity football player two years prior. It's not like there's any magic to it. He went and bought a mountain bike, started riding the crap out of it, joined a gym, watched what he ate, and lost about 85 pounds over the next year.

I was 145 pounds 2 years ago today, after a similar two-year diet and exercise kick that took me down from a deuce and a half myself. Then I fell off the wagon—it happens—and put on 50 pounds over the next 2 years. It sucks, but there's no magic. Two months ago I finally started going to the gym every day, seven days a week, and my legs are sore every day. My weight hasn't improved at all yet. But it will. You just have to work at these things consistently.

But you knew that, didn't you?

This isn't a self-help blog, by the way. I'm not in that business. I'm not here to help, since I don't actually know the answers. I'm just here to rant, and pose occasional questions. It's what I do, when I'm not doing other stuff.

I don't know why I blog. I'm just compelled; it just happens whether I like it or not. Don't read too much into my blogs. My opinions change from day to day. The only things I've learned, the only universal constants, are that I don't know very much, and that public whale explosions are just about the funniest thing human beings can experience during our stay on Earth. I don't know why that is, either.

Today's blog is truly a rant; I just need to get this particular one off my chest, so my gym partner Todd can listen to me rant about something different next week.

The Big Choice

We all have to choose how to spend the time we're given.

If you don't choose, it just slips right by you. I know. On a trip to Vegas not too long ago, I made a pit stop in a casino restroom, and as I was washing my hands, there was this older guy there, also washing his hands. On a whim, I asked, "Hey man, how old are you?"

His reply? "Seventy-two! I have a son. I remember the day he was born like it was yesterday! I was holding him just like so. Well, guess what, he turned forty years old just last week! It goes by in a flash! Before long, you'll be lookin' at THIS!"

He pointed at his wrinkled mug, and concluded his monologue with: "Haw, haw, haw! HAW HAW HAW *cough* *cough* HAW *cough* *hack* HAW HAW HAW HAW HAW!" and walked out. I think I made his day, although I can't exactly say he made mine.

When you graduate from college (or high school, for that matter) you have a simple choice facing you. You can either keep learning, or you can stop.

There is an almost unbelievably easy heuristic for knowing whether you're learning. It goes like this: no pain, no gain. Learning is hard. If it's easy, then you're coasting; you're not making yourself better at something fundamentally new that you couldn't do before.

In other words, it's just like working out. You've gotta mix it up. If you're sore every day, then you're getting good workouts (assuming you can tell the difference between "good" soreness and "injury" soreness; if you're not sure, go ask a professional.)

When you do study, there's a lot of appeal to studying what you already know, because it's less painful. And of course to become an expert at any field, you have to focus on it pretty hard for a long time. But cross-training is well established in sports; you don't typically become a world-class baseball player by just playing baseball all the time. You have to do other kinds of workouts and exercises to maximize your strength, agility, and endurance gains.

Cross-training improves you every bit as rapidly in other disciplines. That includes programming. If you're cranking out code as easily as breathing, then if you're getting better at all, it's so gradual that you'd never notice it happening. You won't have great insights until you get new perspectives from working hard, even if only occasionally, at stuff other than what you already know.

Being in school full-time is an amazing luxury, one that's hard to appreciate when you're actually there, because learning is painful. But trust me on this one: it's even more painful when all you have is scraps of time here and there.

While you're in school, assuming you make a reasonable effort at applying yourself once in a while, you learn a fantastic amount, and you learn it at a fantastic rate. Later you'll learn at a slower rate; it's pretty much guaranteed. Non-educational activities will inevitably intrude and consume the majority of your time.

Hence the choice. After you graduate, you can either learn a little, or not at all.

If you're in the "not at all" camp, well you've made your choice, and I respect it. You'll probably be happier than I am. I'm tormented by how slowly I have to move as a programmer. I now believe programming languages have failed. All of them. The programming world is a giant body shop, and we're building huge pyramids with millions of years of hard labor. I believe that stuff should be easier than it is, and it pisses me off that most people are so content with the current state of the art, because it means they're not helping make it better.

To me, mainstream languages have failed because they're still fundamentally serial, fundamentally early von Neumann, fundamentally single-box languages. They're all vying for the position of "least crappy," and the current winner (whether it's Python, Ruby, Lisp, name your favorite) is just that: the least crappy. Because they're all focused on finding more elegant ways to express mostly-serial computations for crap computers. That, or faking parallelism poorly with threads, if the language supports them at all.

Sure, there have been some interesting attempts at parallel languages. Erlang is one of the better-known ones, and it's actually quite cool. But Erlang has failed too, because you haven't heard of it.

Programming's Biggest Problem

Our industry is in a seriously ugly position, right now, as we speak. Most of the hardware designers are focused on keeping Moore's Law going, because that's where the money is. They want that doubling every 18 months. Today it's probably quite within our reach to get a tenfold increase every 18 months, if we'd agree to focus on parallelism (in the massively distributed computing sense).

But programmers like XML, to the point of focusing on it to an unhealthy extent. Same with C++. And Java. They like these things because they work, and because they like to minimize the amount of crap they have to learn. Because learning is painful. Remember? You might think I've gone way off track, off the deep end, even, but this is the same thread.

Let's face it: a parallel language will have to be radically different if it's to break free of the von Neumann Turing-machine rat race we're in. If we move to cellular automata, or in fact any other parallel computational model that's resilient to node failures, then we'll need a new language, because the current serial languages will perform badly, or be horribly hard to manage, or both.

Cell or grid (or whatever) parallel computing will have a radically different internal economy. It'll need new data structures, new algorithms, new instruction sets, new everything. You do realize that John von Neumann was an economist, right? Among (many) other things, he was an economist, and it influenced the design of his first computer, that one right there on your desk.

The computing machine JvN built was created in an environment very similar to the one in the movie Apollo 13, where the folks at Houston had to build a carbon-dioxide remover out of exactly the free junk available on the spacecraft, and then explain it to the crew so they could build their own copy of it.

Johnny went out and collected a bunch of engineers: materials engineers, electrical engineers, mechanical engineers, everyone who had some spare junk. They came up with a design for a computation machine that just barely sufficed, one that could be built out of the crap they had available at the time: vacuum tubes, magnetic drums, wire, duct tape.

As he was creating this thing, Johnny was focusing on what he was calling the "internal economy" of the resulting machine. Secondary storage accesses were painfully slow. Memory accesses were faster. Register accesses were very fast. And so on. So he designed representations, data structures, and algorithms that were appropriate for that particular machine, the one he was building from spare parts.

If he'd made the machine out of Brazilian rainforest army ants, or mechanical gears, or falling dominoes with marionettes to pick them up again, his data structures and algorithms would have been very, very different. There are some commonalities, sure—you need numbers, and arithmetic, and functions, and data storage, and sorting, and so on. But what's most efficient for army ant computers isn't going to be most efficient for vacuum tube computers.

You do realize you can make a computing machine out of just about anything, right? And they don't all have to work like Turing machines do. Turing was one of the greatest geniuses of the century, but he'd have been the first person to tell you that there are infinitely many machine designs capable of the same computations. His was just one model, and his professor's (which led to Lisp) was just one other model. But who's to say they're the best models?

Some computing machines are more efficient at certain computations than others. Some are more practical to build than others. Some are faster than others. Some are more robust, or more inherently parallel.

You do realize that your brain is such a machine, right? And that it's 100,000-times faster than today's computers at pattern-matching tasks, because while JvN's machine operates serially, your neurons can all fire independently.

Let me give you a hint: your brain's operating system isn't written in C++.

Is our industry ever going to get out of this amazing backwater of a gridlock, this evolutionary dead-end we're in, where we programmers act like army ants, churning out loops and SOAP calls and UML diagrams as if we're weaving cloth with the very fabric of the computational universe?

If it ever happens, and by God I hope to witness it in my lifetime, then the computers and languages and data structures and algorithms are all going to have to change simultaneously. Because a language optimized for a serial computer will perform like crap along some important dimension of performance, probably more than one. But we can't switch wholesale to parallel languages either, because they'll perform like crap on today's computers: again, for some value of "perform" that's not worth discussing here, but it'll be some form of either computer-performance or people-performance.

And programmers are nothing if not fanatically obsessed with performance. Kinda ironic, huh?

Half the irony stems from knowing that there are far more productive languages out there than the ones most of us are using. But most of them perform poorly on our hardware, because these languages are targeting meta-virtual machines, typically "defined" (informally) by the capabilities of the language itself. And if you're not targeting exactly the hardware you're on, the impedance mismatch will slow the language down.

That's the problem with most JVM languages other than Java: they need hardware (think ants! anything can be hardware!) to support operations like long-jumps and tail-call optimization, but the JVM doesn't export those facilities as part of its abstract machine definition.

Same goes for Lisp. It can't get the performance break it deserves because the hardware available today isn't a Lisp machine. They've built them, and I can assure you that C++ would be the loser slug of a language on a Lisp machine. But, alas, performance isn't the only thing programmers care about. They also care about not having to learn anything new.

That's the other half of the irony. Programmers are obsessed with performance, and they'll go to almost any length to fiddle with their algorithms and data representations in order to eek every last cycle and byte from their programs. Any length, that is, except for learning a new language on new hardware. Even if it would get them 1000-times performance improvement for the same productivity. Or a 1000-times productivity improvement for the same performance.

Because they don't want to learn anything hard. No gain, no pain, problem solved.

And that's where we're at. Moore's Law is crap. If we ever want to be 10-times as productive and computationally efficient, let alone 1000-times, then our whole computing model will have to change at once. Nothing else will do. The incremental approaches have all failed. It has to be a revolutionary change.

If everything changes all at once, that's going to pose a bit of a problem for the folks on the Zero Learning curve, wouldn't you say? Don't freak out and mail me about this, either, because I'm a pessimist now, at least about this particular topic, and I doubt we'll ever get out of our rut. We're ignoring the First Law of Holes.

You do realize that John von Neumann spent the last 10 years of his life single-handedly developing a theory of computing based on cellular automata? The computer you're reading this blog rant on was his frigging prototype! He was going to throw it out and make a better one! And then he died of cancer, just like my brother Dave did, just like so many people with so much more to give and so much more life to live. And we're not making headway on cancer, either, because our computers and languages are such miserable crap.

You have no idea the pain I feel when I sit down to program. I'm walking on razor blades and broken glass. You have no idea the contempt I feel for C++, for J2EE, for your favorite XML parser, for the pathetic junk we're using to perform computations today. There are a few diamonds in the rough, a few glimmers of beauty here and there, but most of what I feel is simply indescribable nausea.

Are you beginning to see why I prefer to work with programmers who stay on the Upward Curve after they get out of school? Because even while we're grubbing around in the dirt—just a sorry bunch of illiterate, innumerate cavemen, here in the very heart of the Stone Age of programming—at least these upward-curve programmers give me some hope. Hope that if something better comes along, they'll give it a try, a serious try, the old college try. Or hope, even, that they'll build that "something better" with me.

Fat chance. But hope can keep ya going for a good long while.

Baby Steps

It's all still fun, though. Broken glass and razor blades aren't so bad, when I think about how much worse my lot in life could be, had I been born in a different time or place. I've got it pretty good, and so have you, in all probability.

At my current job, they feed us and massage us like Kobe cows, and I'm surrounded by unbelievably brilliant people, all way smarter than me, and we're doing great stuff together. Make no mistake: my blog whining is all relative to a totally imaginary future, one which in all likelihood, should it ever come to pass, will be filled with even more whining about totally imagined new futures. It's just in our nature to whine. But really, I have no complaints.

I put a lot of stock in fun. And family. And trying to live my life in such a way that I won't have any major regrets when the game's over. So there's the first part of my schedule: having fun.

If you want to be on an upward curve, just make some time for it, and make it a habit. That's all there is to it. It doesn't matter if you're trying to get better at programming, or math, or fitness, or flying kites, or even humanity's Number One Fear, even worse than the fear of Death: public speaking. You just work your way up, a little at a time.

I can't promise you any satisfaction from the upward curve. You'll get better at a lot of things, and you'll have plenty of interesting insights. You may even get a better job, or build some software that makes you famous, or just have more fun doing what you do. But you won't have much time for television. Something will have to give. We all have to choose how to play our time, and it's a zero-sum game.

If, like me, you're dissatisfied with the current state of affairs, well, believe you me, you can find a lot of consolation in a book on math, or machine learning, or compiler construction, or on just about anything that promises to help in some small way.

You do have to learn to put up with the pain of learning, though, and make it a habit to push yourself a little each day.

As far as the actuals go, well, you'll just have to find an approach that works for you personally. You might only be able to devote one quiet hour a week to studying, but like unit testing, or working out, or brushing your teeth, it's better than not doing it at all.

Just try to have fun, and hopefully the rest will fall into place.

Author’s Note: Transformation

This is an article I wrote for the O’Reilly Ruby blog, where I was a guest blogger until I pooped out and got tired of it. I did manage to write three or four posts, though.

Actually, thinking back on it, the reason I stopped posting is that I largely stopped using Ruby shortly after this. Not because I don’t like Ruby—far from it; I think it’s great—but because it is unpopular inside Google. So there isn’t much opportunity to use it in my day-to-day work, except for little personal scripts. Anything bigger would be viewed with suspicion.

(Despite what I said in the Tour de Babel intro about Google being better at solving problems with languages than Amazon, by virtue of writing powerful DSLs when appropriate, Google is still largely biased against introducing new general-purpose languages. So it goes. Gotta pick your battles.)

This post is a direct comparison of Java, which I started out loving and over a decade grew to detest, and Ruby, which I feel is one of the finest language-design efforts of modern times. Ruby has its flaws, like all languages, but it’s still a breath of fresh air. Of all the non-Lisp languages in the world, I think Ruby may still be my favorite.

Anyway, I thought this article was a lovely little piece of writing, if I do say so myself. I think it has a structural elegance that I achieve only rarely. Most of my writing is unstructured rambling, but this post actually has plot development (or something approximating it)—even though I try to sneak it up on you unnoticed.

I was at least partly successful: nobody in Java-land noticed the post. Oh well.

Transformation

I keep hearing people say they couldn’t possibly use Ruby because it lacks automatic refactoring tools. And although it will eventually have some of them, there’s a class of automated refactorings that are automatable in Java but not in Ruby. This, people say, is a show-stopper.

I wonder.

What exactly is refactoring? I mean, it’s not a word in the dictionary.

Fowler tells us that it’s the art and science of turning smelly code into good code, in small, incremental steps. Provably correct, by construction. Algorithms for giving your code a makeover without breaking it in the process.

He gives us a nice taxonomy. He presents some good techniques, especially geared for Java programmers. Some are things we had already figured out and are habitual, and some of them are new.

Some of these “refactoring” techniques are automatable. And many of them are useful in languages other than Java. It seems Fowler and friends have stumbled on something real, something as big as OOP, almost. Or at least they were the first to market it and package it properly. Either way, we know about it now. Thank you, Fowler and friends!

Refactoring is one of the first programming books I’ve seen that talks about the almost mystical act of writing code. It takes the process, exposes all the insides, revels in it, walks you line by line through oh so many little decisions that affect code quality. These are things most people never talk about. They take them for granted. Most people talk about “architecture." Refactoring talks about the idioms in the code we write every day. Real now-code, not planned someday-code.

It’s remarkable, really, that nobody talks about this. They leave all the so-called style choices to the programmer. Refactoring rubs our noses in the implications of our line-by-line style choices. Beautiful.

Discovering Refactoring

Refactoring caught my eye in the bookstore one day in 2002, years after it was published. I hadn’t read it because it was published by those UML weenies. I’ve just never been a fan. It has its uses in database modeling (maybe), but I’ve never found it useful in class modeling. And I’ve never cared for the Booch/Jacobsen/etc. crowd’s books.

Refactoring is right there, smack in the middle of that weenie series. Every time I see it, my eyes sweep across the cover without a second glance. You know the old saying!

But one wintery day in 2002, I’m in the bookstore, and I pick it up. No real reason. I’m curious. Don’t know why. Maybe I’d finally heard the word “refactoring” somewhere. What did it mean? It’s not a word in the dictionary.

“Factoring," sure, that’s a dictionary word. You can factor numbers, or polynomials. Factoring I know. Don’t know why you would re-do it, though. What’s “re”-factoring?

I open the book. It says local variables are the root of all evil. Perhaps not exactly those words, but it’s the first discussion I stumble across. Local variables!? I plop down in a squashy armchair, outraged, to read more. I want to know if this guy is actually insane, or merely an idiot.

Horror sets in: he’s right. His explanation makes chilling sense. One of my cherished programming practices—caching intermediate values in local variables, as an inline performance optimization—is clearly demonstrated, before my very eyes in the squashy armchair, to be Evil. It explains why I have certain methods in my code base that keep growing and growing, and for reasons I’ve never been quite able to grok, the methods are unsplittable.

These big methods, they’re the Bad Places. The areas of the code base where I loathe to tread. Dark caves that grow more evil every time I visit them. Because add functionality I must, but the locals have threaded their way impenetrably through each function, spiderwebs that catch me and hold me.

The book shows me why they’re unsplittable, then gives me axes to split them. Sharp and precise tools. And the techniques make sense, right then and there. Some even appear to be automatable. Wow.

I move on. Turning pages faster, now. Interested.

The book next tells me: don’t comment my code. Insanity again! But once again, his explanation makes sense. I resolve to stop writing one-line comments, and to start making more descriptive function and parameter names.

I buy the book, take it home, read it over and over. I marvel. It appears to be pure genius. To this day, I still feel that way, although perhaps not so greatly as I did on that day. But the book is a landmark, and it made me a better programmer overnight. How often does that happen?

Sudden embarrassment. How could I not have read this back in 1998? I’m awash with a horrid cold feeling, as if I’ve just learned I’ve been coming to work for years with my pants down around my ankles. Has everyone else at work already read this book? Am I the only one who didn’t know?

I ask around the next day. Casual. Cool. Not tipping my hand. You’ve read Refactoring, right? Nope. Everyone I ask says No. Most haven’t heard of it. Out of 20 developers I survey, only one guy has read it. No surprise there, since he reads everything. His vote? “Yeah, that’s a great book!”

I feel a rush of relief. Most people don’t know about it, then. I’m safe. I can study it, use it, not worry that everyone will know how foolish my code has been. My code had only been bad along a few dimensions. Most of it was well-engineered. I used design patterns, unit testing, source control, all the usual software engineering discipline. It just smelled a little bad, and now I could fix it.

Refactoring Today

Everyone knows about Refactoring nowadays, because IDEs now have all of the automatable refactorings from the book, and a few extras to boot.

But despite its overnight popularity, I doubt most engineers have read Fowler’s book, not even a few chapters of it. I suspect most engineers today don’t realize there are still many refactorings that are not automatable, even in Java. Most of them, even. Although that’s a topic for another day.

Today, I still don’t know exactly why they called it “Refactoring.” Catchy, I guess. It feels distantly related to factoring, in the math sense. Reorganization? Too broad. Refactoring seems fine.

Sometimes a good name is all that lies between a great idea and mass acceptance.

Refactoring today is an entire industry. It’s a banner. It’s the battle cry of Java-IDE lovers everywhere. Refactoring tools are Productivity in a Bottle. You browse a menu of refactorings, choose one, and the earth moves.

Why is automated refactoring so popular in the Java camp, and nowhere near as popular in other languages?

Java people say it’s because only Java allows you to accomplish this level of automation of code transformations.

Reprise

I keep hearing people say they couldn’t possibly use Ruby because it lacks automatic refactoring tools. And although it will eventually have some of them, there’s a class of automated refactorings that are automatable in Java but not in Ruby. This, people say, is a show-stopper.

I wonder.

I read Fowler. I absorbed it. It’s the art and science of taking smelly code and turning it into better code, in small provable steps.

But he taught us something else, didn’t he?

Oh, but you wouldn’t know what that thing is, if you haven’t read his book. Have you? All of it? No skimming? C’mon now. Admit it. You skimmed.

Here’s the deal: to show us the paths from bad code to good, Fowler had to show us bad code. He showed us examples of what it looks like, and explained why it’s bad. He gave us a set of warning indicators and even called them “Code Smells." More clever marketing? Perhaps. But they’re right on the mark.

How did that code get smelly in the first place?

Well, we optimized prematurely. We stored too many intermediate values, for fear of recomputing them. We didn’t write small functions, for fear of virtual method-call overhead. We made bloated class hierarchies for the imagined benefits of reuse. We made huge parameter lists to avoid allocating a container object. We used null everywhere as a semantic token. We allowed boolean-logic expressions to grow into unreadable thickets. We failed to encapsulate data and data structures with accessor methods. And many more bad things besides.

We were making dozens, hundreds of little mistakes that added up to some pretty smelly code. The book catalogued our mistakes, gave them names, elevated them to First-Class Mistakes.

Then, presumably, we stopped making them? Well, maybe those of us who read the book. Even if we read it late. Better late than never. After reading it, you know what bad code smells like, and you know how it got that way. You’ve learned how to avoid writing it.

At what point did automated refactoring tools become the focus? The book’s original focus was about design, with tools for recovery. Now the focus is all on recovery, and specifically on the automatable subset of recovery techniques.

The implicit assumption here is that bad code just happens, inevitably. Even though we know all about its characteristics, and we know how to spot it instantly. Heck, if you read the book, you know it wasn’t just a catalog of 100-odd specific refactorings. It also presented themes. Once you get the core ideas, you can invent your own refactorings and identify new code smells.

And now you know better how to write the code correctly the first time around.

Oh, you disagree? Because code is a living thing, and requirements change constantly? Yes. Code needs to change. But Refactoring isn’t the whole story on code change; it’s a relatively small part. There’s data modeling, and architecture, and design patterns, all the high-level astronautics. And of course the custom code-pattern transformations you apply almost daily that aren’t general enough to give first-class names. Changing any of these things requires techniques that I am sure you do not have entries for in the Refactoring menu of your IDE.

Refactoring is zoomed way in. It’s focused on how you personally wrote this or that class or method, down at the level where you were making choices about local variables, control-flow constructs, and other micro-design decisions.

You now know how to avoid doing the wrong things, at that level.

Well—you know if you read the book, that is. Without skimming. And also, I suppose, only if you were already experienced enough for the book to stab at you like a blade, mocking you for not noticing every one of these things yourself, so that you will remember its lessons in your bones for the rest of your days.

Did Refactoring make us lazy? Maybe so. Especially if we skimmed it, or just read the tools-half without reading the explanation-half. Then maybe we think the whole story is about recovery from code that inevitably goes sour. And even if you read the book, maybe those siren-song automatic refactorings have made you forget what the book was really about. Namely, fixing your code and then never again writing so amateurishly.

Is this the whole story? No. Do I sometimes still need to refactor my code? Yes. Are there subtleties I’m punting on for now? Yes. Refactoring can’t really be discussed in a vacuum; it’s interdependent with other modern development ideas, including “don’t repeat yourself," “once and only once," unit testing, and others. I may revisit refactoring again in that context.

Today, though I’m just interested in why Java programmers are saying that the ability to “program” by pushing buttons is so critically important to them that they’re unwilling to consider using another language. Even a language that’s gaining rapid worldwide recognition as a step-function in productivity at least as great as Java was over C++, with almost none of the downsides or friction of similar-looking predecessors like Perl and Python.

I mean, they won’t even consider trying Ruby? Gosh. Pushing buttons must be… wonderful.

I close my eyes, envisioning that kind of power…

Pushbutton Productivity

Ah, those automated refactorings. Such programming power—instant productivity with the click of a button. Programming by Menu Selection. Choose your automated attack, and the very earth moves. Mechanical muscles moving mountains of code. It’s almost as if you’re superhuman. Programming never felt so much like a video game.

It must feel like piloting an earth-mover: a John Deere, a Komatsu, a Caterpillar—one of those huge yellow mechanical dinosaurs with the world’s largest tires, the ones that held us in awe when we were children. The driver pulls a lever, a hill of dirt moves aside. The work of a hundred men in a day, accomplished with the bored flick of a wrist. A lifeless yellow behemoth at your beck and call.

Now that’s productivity. If my job were moving mountains of dirt around, then I agree: I would not possibly be able to work effectively without an earth-mover, some tractors, a dump truck, maybe a crane or a backhoe. My personal tools for refactoring the surface of the earth. You couldn’t pry my fingers from their hundred-ton hulks.

Caterpillar. Such an odd name for a motorized Colossus. Or is it? They named it after a little segmented bug. All the segments look alike, repeating themselves. Each segment has two identical tiny legs. The legs have to move in coordinated waves to propel the bug forward. So much computational processing devoted entirely to crawling around in the mud!

Yes, I see it now. Caterpillars are long, machine-like insects. Earth-movers are huge, insect-like machines. It begins to make sense.

Automated code-refactoring tools work on caterpillar-like code. You have some big set of entities—objects, methods, names, anything patterned. All nearly identical. You have to change them all in a coordinated way, like a caterpillar’s crawl, moving all the legs or lines this way or that.

How did our code get that way to begin with? We wrote it badly. Refactoring to the rescue. Good design may be a lost cause, but we can recover, because we have automated servants to go fix all those little segments for us. They never get tired, and all we have to do is push buttons.

Well then. How could you possibly live without automated refactoring tools? How else could you coordinate the caterpillar-like motions of all Java’s identical tiny legs, its thousands of similar parts?

I’ll tell you how: Ruby is a butterfly.

Author’s Note: Is Weak Typing Strong Enough?

This is the second-oldest post in this book. Out of my nearly fifty Amazon-era blog posts, they only included two in this book, the other being Tour de Babel.

I wrote this post about a month before I left Amazon. This was my best effort at writing an unbiased, fair, reasonably accurate summary of the major philosophical differences I saw at Amazon. I saw these philosophical differences colliding in the Perl/Java camps in my organization, in the data modeling camps across the org, and in other places as well.

At the time, I focused mostly on the related issues of type-safety (for programming languages) and schema-safety (for relational database design). I have since then generalized the problem to include many other polarizing design issues—my recent “Notes from the Mystery Machine Bus” is a good summary of this dichotomy that I’ve observed over the years, over diverse projects, over different orgs and entire companies.

Back at Amazon I wasn’t sure which way was “better.” I now believe that neither way is intrinsically better, but that everyone has certain built-in preferences for one or the other.

Even though this post is slightly redundant with “Notes from the Mystery Machine Bus,” it contains a detailed description of a project I observed firsthand, a project on which Perl and Java developers worked side by side for over a year on the exact same problem. And the Perl folks, many of whom were self-taught programmers, absolutely kicked the Java guys’ asses.

Go figure.

Is Weak Typing Strong Enough?

So… how big can dynamically-typed systems get? Do static type systems really matter? I'd love to know the answer to this.

We have some big systems at Amazon, and most of them seem to use strong static typing; at least the ones I know about do. Is that a requirement, or could we just as well have done them in Perl, Ruby, Lisp, Smalltalk?

I'm actually interested in the question in a broader sense than just for programming languages. For instance, the idea of strong typing applies just as much to relational data modeling as it does to programming languages. You can choose to model the living hell out of every possible entity, or you can throw together a quick-and-dirty schema of name/value pairs. Same goes for XML data modeling.

Pros of static typing

Here's my stab at a list of the main advantages of static typing:

1.) Static types provide constraints that can detect some type errors early, before the program runs (or at the time a row is being inserted/updated, or an XML doc parsed, etc.)

2.) Static typing offers more (or perhaps just easier) opportunities for performance enhancements.

For instance, it's easier to create intelligent database indexes if you have a rich, thorough data model. And compilers can make better decisions when they have more precise information available about variable and expression types.

3.) With verbose type systems like those of C++ and Java, you can look at the code and see the static types of your variables, expressions, operators, and functions.

This advantage isn't as true in languages with type-inference like ML and Haskell; they evidently feel that requiring the type tags everywhere is a negative. But you can still specify type tags in cases where it clearly aids readability—something most dynamic languages don't allow.

4.) Static type annotations make it easier to do certain kinds of automated processing of your source code.

This includes automated doc gen, syntax highlighting and indenting, dependency analysis, style checking, and other code-looking-at-code kinds of stuff. Static type tags simply provide more "purchase" for compiler-like tools: more distinct syntactic elements available for lex-only tools, and less guesswork needed for semantic analysis.

5.) People can look at an API or schema (as opposed to the implementation code or the database tables) and get a feel for its overall structure and usage patterns.

Am I overlooking any other advantages?

Cons of static typing

Here are the disadvantages of static types, as I see it:

1.) They artificially limit your expressiveness.

In Java, for instance, the type system doesn't let you do operator overloading, multiple inheritance, mix-ins, reference parameters, or first-class functions. Any time the most natural design involves these things, you're stuck trying to mutate it into a design that fits Java's type system.

You can come up with similar complaints about every static type system out there, from Ada to C++ to OCaml to, er, Z-language. About half of all design patterns out there (not just the GoF patterns) appear to be ways to take perfectly natural design ideas and twist them to fit into someone's static type system: recipes for pounding square pegs into round holes.

2.) They make development go more slowly.

You spend time up front creating your static models (top-down design) and more time updating them as requirements change. Type annotations also take up real estate in your source code, so they're a little slower to enter and maintain. (This is only a serious problem in Java, which has no type-aliasing facilities.) And as I mentioned above, you also have to spend more time forcing your designs to fit the static type system.

3.) They take longer to learn.

It's much easier to get started with a dynamically-typed programming language. Static type systems are rigidly picky, and you have to spend a bunch of time learning their way of modeling the world, plus additional syntax rules for specifying the static types.

Also, static type errors (aka compiler errors) are especially hard for language learners to figure out, because the poor program hasn't had a chance to run yet. You can't even use printf-debugging to see what's going wrong. You're stuck rearranging your code randomly until it gets past the compiler.

This means it's harder to learn C++ than C or Smalltalk. It's harder to learn OCaml than Lisp, and harder to learn the Nice language than Java. And Perl has a whole bunch of static complexity—labyrinthine rules about what you're allowed to say, and how, and when—which makes harder to learn than Ruby or Python. I've never seen an example in which static typing made a language easier to learn.

4.) They can lead to a false sense of security.

Static type systems reduce runtime errors and increase data integrity, so it's easy to be lulled into thinking that if your system passes compilation and runs, then it's basically bug-free. People who use languages with strong static type systems seem to do a lot less unit testing. Maybe it's my imagination, though.

5.) They can lead to sloppy documentation practices.

It's easy to throw out the javadoc for your system with practically no comments, and think that it's sufficient for figuring out how to use it. I see this all the time in SourceForge projects, and even the Sun JDK packages often do this. (For instance, Sun frequently fails to provide any javadoc comments whatsoever on groups of static final constants.)

6.) They rarely yield highly dynamic/reflective systems.

Most likely for performance reasons, most statically typed languages throw away most or all of the compiler-generated metadata at runtime. As a result, the systems are usually very hard to modify (or even introspect on) at runtime. E.g., to add a new function to a module, or method to a class, you usually have to recompile, shutdown, and restart.

This doesn't just impact the development cycle; it can adversely affect the entire design, if you have a system that needs to be modified (or reflected on) while it runs. You wind up needing to build elaborate architectures to support dynamic facilities, and it's inescapably mixed in with your domain logic.

Have I overlooked any other disadvantages of static type systems?

You can pretty much invert all the points above to obtain the pros and cons of dynamically-typed languages. Dynamic languages give you more expressive power and more design options; they're easier to learn; they make development go faster; and they tend to provide more run-time flexibility. And in general, you lose on early warnings of type errors (at least from a compiler), they're harder to optimize for performance, it's harder to do automated static analysis on them, and it can be harder to look at the code and figure out the type of a variable or expression.

Just as static languages tend eventually to bend and start adding dynamic features, dynamic languages often try to add in some sort of optional static type system (and/or static analysis tools), as they try to improve performance or increase early error detection. But it's usually pretty ugly, and works best if the language designed optional static types in from the start.

What's the right approach?

The strong vs. weak typing issue really gets people worked up. The choice has a major impact on virtually all aspects of your project lifecycle, architecture, and development practices.

Which of the following positions would you say is most true at your company, assuming (for the moment) that you can only choose one of them:

1. Above all, we need stability. We have enormous scale and massive business complexity. To create order out of inevitable chaos, we need rigorous modeling for both our code and our data. If we don't get the model and architecture mostly correct in the beginning, it will hurt us later, so we'd better invest a lot of effort in up-front design. We need hardened interfaces—which means static typing by definition, or users won't be able to see how to use the interfaces. We need to maximize performance, and this requires static types and meticulous data models. Our most important business advantages are the stability, reliability, predictability, and performance of our systems and interfaces. Viva SOAP (or CORBA), UML and rigorous ERDs, DTDs or schemas for all XML, and {C++|Java|C#|OCaml|Haskell|Ada}.

2. Above all, we need flexibility. Our business requirements are constantly changing in unpredictable ways, and rigid data models rarely anticipate these changes adequately. Small teams need to be able to deliver quickly on their own goals, yet simultaneously keep up with rapid changes in the rest of the business. Hence we should use flexible, expressive languages and data models, even if it increases the cost of achieving the performance we need. We can achieve sufficient reliability through a combination of rigorous unit testing and agile development practices. Our most important business advantage is our ability to deliver on new initiatives quickly. Viva XML/RPC and HTTP, mandatory agile programming, loose name/value pair modeling for both XML and relational data, and {Python|Ruby|Lisp|Smalltalk|Erlang}.

I'll (slightly) caricature the workflows of these two philosophies, to make the essential differences as clear as possible.

The strong-typing camp basically works as follows: start by designing for your current requirements. Make a spec, even if it's a very lightweight spec. Define your interfaces and data models up front. Assume you're going to suffer from massive loads, so write everything with a keen eye towards performance. Avoid using abstractions like garbage collection and regular expressions. (Note: even Java programmers generally work extra hard to avoid garbage collection, always talking about object pooling before they've even started coding yet.)

The first camp only resorts to dynamic typing when they're backed into a corner. For instance, under extreme duress, a team using CORBA might finally add an XML string parameter to each interface call, effectively giving them an "escape" from the rigid type system that they adopted in the first place.

The second camp basically works as follows: start by making a prototype. Assume you can code it faster than a person could write a spec with the same level of detail, which means you can get customer feedback on it faster. Define whatever interfaces and data models make sense at the moment, but don't waste a bunch of time on it. Write everything with an eye towards doing the simplest thing that could possibly work. Assume you're going to suffer from massive requirements change, and write everything with a keen eye towards getting something working fast. Use abstractions (e.g., collections rather than buffers, or regexps rather than string compares), even when you know they may be overkill, because they buy you more flexibility, save you some typing, and generally have fewer bugs.

The second camp only resorts to performance optimizations and interface/schema lockdowns when backed into a corner. For example, under extreme duress, a team working in Perl might rewrite some heavily-used core modules in C and create XS bindings. And over time, as abstractions become standardized through usage, they gradually get locked down by wrapping them with schemas or finer-grained OO interfaces. (Even Perl programmers often bite the bullet and write OO interfaces for commonly-used abstractions.)

How do you suppose these strategies pan out over the long term?

Big Case Study

I watched the strong/weak battle play out (in various ways) in Amazon's Customer Service Applications group for years. I was initially aligned as follows:

- I was in the "strong" camp for languages, personally favoring development in Java.

- I liked the "weak" camp for protocols (e.g., favoring xml/rpc over SOAP) and XML modeling (favoring no DTD or schema at all).

- I was in the "clueless" camp for relational modeling (favoring keeping my mouth shut and learning from the experts).

One thing I observed was that the folks who favored Perl always seemed to be able to get stuff done really, really fast, even compared to experienced Java folks. And they had their act together; it wasn't just crude hackery, as many Java programmers would like to believe. Their code was generally very well organized, and when it wasn't, they'd go in periodically and fix it. Sometimes they did quick, hacky scripts, and in fact the ability to do this proved to be mission-critical time and time again. But generally the Perl stuff worked just as well as the Java stuff. Whenever performance became an issue, they'd find clever ways to make it perform well enough.

CS Apps had a relational data model for customer contacts, but it had a gaping hole, by way of an untyped attribute system (name/value pairs, basically). The relational model didn't change very often, in part because that was back in the days of centralized control over the databases, so it was pretty hard to get schema changes through, even though we had several talented data modelers among the software engineers. But the schema also didn't change much because even if they'd let us make the changes, our requirements were changing so fast that we might never have been able to keep up. Those flexible contact attributes were a lifesaver. Even the static-language camp had to agree.

Yes, we had some data integrity issues. Name/value pair models are somewhat more prone to wrong names (via typos), bad values, or bad dependencies, because you can't rely on the database to do constraint checking for you. If anything, this made us more careful. When we noticed data integrity problems, we'd write a backfill and fix them. Sometimes it was hard. But we realized that even with the strongly modeled tables, we still had occasional data integrity errors. You always need to be able to recover from errors, strong typing or no.

And yes, we had performance issues from time to time—with the Java stuff too, though, and also with the C++ code (CS Apps was a mixed bag because we had to interoperate with virtually every other system in the company). It doesn't matter what language you use—performance issues will always come up. You just have to find them and fix them.

So for years, we had Java and Perl development going on side by side. This was a decision made in old days, purely for expedience reasons. When we were deciding how to implement Arizona (our internal web-based application suite for CS), we had about a 50/50 split between Perl and Java programmers on the team.

During the initial development, the Perl use-cases got finished astonishingly fast. For a while, Arizona had more Perl than Java, because our Perl programmers started grabbing tasks assigned to the Java folks. Over time, the "climate" across the company pushed us to migrate everything towards Java. It was a pretty complicated situation not worth recounting here, but over a period of years, most of the Perl stuff in Arizona was gradually rewritten. (I heard after I left, the climate changed again, with Java being phased out in favor of C++ and Perl, but for reasons mostly unrelated to the languages themselves.)

In any case, for several years I got to watch Perl and Java folks working side by side doing pretty much the same tasks. In some cases, they even had to implement the same logic in both languages. Yes, there were inefficiencies with our Perl-and-Java approach. However, it was the right decision at the time, and as a result, I was personally able to witness a more or less apples-to-apples, multi-year comparison of the strong-typing and weak-typing camps at work in the same production environment.

In nutshell, I was pretty impressed. I was a diehard Java guy at the time, and even then, I could see that the Perl code was far smaller and simpler than the Java code. It didn't feel "cleaner," since Perl itself is a bit challenged in that department, but it seemed modular enough. It had a well-defined architecture, and it got the job done, year in and year out.

Our Java code (to me) seemed far more complex, even though I could read Java more easily. I think Java programmers have a tendency to over-engineer things, myself included. I suppose many Java folks would have thought of our Perl code base as grossly under-engineered. But if it was really under-engineered, it seems like it should have caused more problems than it did. In practice, most of the problems in the Perl code were interoperability issues with external services (or databases, in cases where there were no services yet). Most service owners didn't include Perl bindings for their interfaces, so our Perl folks had to do a lot of head-scratching to find workarounds.

On the data-modeling side, the team taught me how flexible attribute systems can be created in relational schemas. DBAs and (especially) data modelers tend to hate them, but SDEs like them just fine, since it beats trying to do "real" O/R mapping, and you don't have to make schema changes to accommodate certain large classes of new requirements. I doubt we'd have been able to keep up without that flexible system in place.

Why I'm Weak

These days, I believe approach number two (weak/latent typing with selective lockdowns) seems preferable to approach number one (strong/static typing with selective loosening) for most problem domains at Amazon. Our business is always changing rapidly, so we're always doing major overhauls of our interfaces and data models. We always have more work than we know what to do with. And we have Moooooooooore's Law to lean on… well, except that it's stalled out for the past three years. But still…

The CS Apps example isn't the only basis for my opinion, although I do feel it's pretty compelling evidence. (How often do you really get to see large-scale, long-term, mostly apples-to-apples comparisons of two different languages with different type-system philosophies?)

Another big reason I prefer weak typing is that I've seen plenty of examples where a team using strong-typing has thrown in the towel. I really have seen a team give up on "hardened" interfaces and add in an XML string parameter to a CORBA interface, and I thought it was absolutely the right decision. They were getting slain by CORBA's type system: every teenie weenie little change for one customer would affect all their other customers as well. Their "tunnel" gave them some breathing room.

I've seen this kind of thing elsewhere as well. Heck, even Sun's JMX interface looks strongly typed, but it has a weakly typed query language embedded as a String parameter ("ObjectID"), one that's totally opaque to the compiler, interface generators, etc. So much for strong typing saving the day.

And once we did a site for a major sports franchise, and they wanted customers to be able to put their initials (or a custom number, or whatever) on jerseys and such. Our rigid model for orders (and the associated strongly-typed interfaces wrapping that model) had no provision for passing a custom initials field through to the backend, to be sent off as part of the drop-ship request. (Dunno if that technically counts as drop-ship, but you know… ) I remember there being weeks of angst about how we were going to solve that problem.

The customer-initials thing was trivially tiny compared to the schema impact that Wireless (cell phones) had. Months and months of angst there. And Wireless seems to be turning out to be tiny compared to some of our more recent initiatives (from a data-modeling and interface-design perspective, at least).

I could go on with other firsthand examples. Generally speaking, strong static typing has gotten in our way, time and again, and weak typing has never resulted in more "horribly bad" things happening than the equivalent strong-typing approaches. Horribly bad stuff just happens sometimes, no matter what approach you use.

Purely from my own personal-productivity standpoint, I've found that well-designed weakly-typed systems allow me to be much more productive (and usually much happier) than equally well-designed strongly-typed systems. Emacs is one example. Even now, when I'm still much better at Java than Lisp, I'd rather write an Emacs plug-in than an Eclipse plug-in any day of the week. It's an order of magnitude simpler. Well, once you learn Lisp, which was pretty hard. But in the long run, I'd rather spend the extra study time up front, and wind up being more productive forever.

And Emacs isn't especially well-designed, at least by modern standards. It could benefit from OO interfaces, namespaces, multithreading, less reliance on dynamic scoping, etc., all of which it would have if it were written in a more modern Lisp. But even in a dinosaur like Emacs, it's still an order of magnitude simpler to write plug-ins. If you focus strictly on the user-extensibility mechanisms of Emacs and Eclipse, you have yet another compelling, essentially apples-to-apples comparison of strong-typed vs. weak-typed language approaches, and Weak wins again.

And look at Java: even Java proponents would still rank Java's reflection, dynamic proxies, dynamic class loading, varargs, and other non-statically-checkable features as being critically important. Sure, they'll caution you about potential performance problems, and they'll tell you not to rely too much on dynamic features—but I doubt the Java community would be willing to throw all that stuff out. Those features are considered major advantages to using Java.

I used to be a big proponent of strong typing, but over time, my own experience has led me to feel it's not the right thing, at least not for the kinds of systems we build. If you're at a bank, sure. If your business rarely changes, you can afford to have a rigid, well-specified data model. If you're in some industry that has much harder constraints on performance, or safety, then sure. Use a strongly-typed system.

But after watching the new Hitchhiker's Guide movie this weekend, and seeing the hilarious caricature of British governmental bureaucracy in the Vogons, I thought: hey, I hate bureaucracy. And static type systems are basically just bureaucracy. I want them to get out of my way and let me get stuff done without needing to fill out a bunch of forms. If the price of getting static type-safety is working with a brain-dead compiler and a restrictive type system, well, then I can handle my own type errors, thanks very much.

Are the Forces of Weakness strong enough?

I still have some doubts. Do weakly-typed systems have inherently lower scalability? Do they tend to dissolve into vast typeless traps at a certain size, as the static camp would have you believe? Do the runtime type-error rates get out of hand, even with rigorous unit testing and software-engineering discipline?

And is the performance prohibitively expensive? For instance, do you know of any large, weakly-typed systems that had to be thrown out and rewritten in a statically-typed language (by the same team, so we know it wasn't just language preference) in order to meet performance goals? I'm specifically interested in n-box distributed systems that failed, not embedded systems or programs for end-user desktops.

I don't think it should be a problem, and in theory, I feel it should be possible to use Ruby or Lisp (or Smalltalk, Python, or any other dynamic language with powerful mechanisms for code modularity and object-oriented abstraction) to build a large, customer-facing, production service at Amazon. I'd like to see some evidence of that before actually trying, though.

At this juncture, I think enforced static typing (e.g., what you find in Java, C++, OCaml, Ada, etc.) is detrimental to progress and flexibility. I also think that a complete lack of support for it (e.g., what you find in Ruby and today's Python) is problematic for being able to selectively tighten up systems as their usage patterns become established. I think Lisp's solution, where you can add in static types as needed, is close to ideal.

But I'm still a bit timid about trying to write something really significant in Ruby (my weakly-typed language of choice), on account of its performance and lack of native threading. I'm equally timid about trying Common Lisp, mostly because the package contributions on Cliki seem fairly paltry; the language doesn't appear to have enough momentum for me to commit to it. I have similar reservations about all the other viable options (e.g., Python, Erlang, Scheme, Lua).

And yet I'd prefer not to work in Java or C++ anymore.

This is a hard problem.

Note: I've deliberately misused the terms "strong" and "weak" in this article, for two reasons. First, to help make the distinction that I'm talking about more than just programming languages; the issue applies to data and interface modeling as well. Second, for poetic effect.

I'm obviously talking about static versus dynamic typing in the programming-language contexts. I'm well aware that the situation is better described as a 2 x 2 grid consisting of the combinations of static/dynamic and strong/weak; e.g., see Chapter 1 of Benjamin Pierce's Types and Programming Languages textbook.

I only bring this up because of some almost unbelievable whining coming from a few Python folks who've chosen to overlook the largely pro-Python(/Smalltalk/etc.) viewpoint of my article here, because they objected to having their dynamic typing referred to as "weak." Python and poetry evidently don't mix well.

Become a Hyperink reader. Get a special surprise.

Like the book? Support our author and leave a comment!

III.

Code Philosophies

Author’s Note: Software Needs Philosophers

I wrote this essay after losing a lot of sleep over a previous post—not included in this book, alas—called “Lisp is Not an Acceptable LISP.” It caused a tremendous uproar, one that was pretty unexpected at the time, since I wasn’t really a seasoned Lisp programmer. I didn’t have any street cred as a reliable critic nor pundit in the Lisp space. But it got people really upset.

I made various technical gaffes in the write-up, which the haters seized on and used to try to focus the discussion on the trees rather than the forest. They did so because I’d highlighted a dark scary forest that most Lisp aficionados did not want to think about: popularity.

Fortunately a few seasoned and well-respected Lisp folks actually understood what my real point was, and they argued that I was for all intents correct—that Lisp is way less popular than it should be, and less modern than it should be (as a spec, anyway), and that the community hasn’t been doing itself any favors in that space for twenty-plus years.

I guess what shocked me the most was the violence of the emotions that surfaced. I’d always assumed that Lisp people felt comfortable enough and self-confident enough in their choice of language to be able to weather some constructive criticism. I mean, the Perl crowd sure as hell can. You can criticize the hell out of Perl, and they just shrug and nod and smile and keep working.

But I discovered that Lisp community has a nontrivial percentage of members who are insecure, and who are easily pissed off if you criticize their language. I see that pattern in a lot of languages, but I wasn’t expecting it from the Lisp crowd.

So I lost sleep, and tossed and turned, and I wrote this post in record time. Couple of hours. And I got a lot of stuff off my chest. It’s probably still controversial, but I’ll stick by it.

Warning: I’m pretty harsh on organized religion in this post. I am not a big fan of organized religion. I realize that statistically speaking, you are probably a member of an organized religion, and I apologize for being harsh on them here (on Catholicism in particular).

But as it happens, I am an anti-fan of many kinds of organization. I think large organizations almost always begin to exhibit standard failure patterns that their founders never anticipated. They gradually acquire all sorts of properties of classic “bad” organization types—mafias, cults, old-boy networks, dictatorships, militant racist groups. I’m sure you can think of others.

Basically I think large organizations—especially very old ones—just kind of suck in general. I’ve even come to think this about code bases. We’re just not good at them. It’s too easy for large-scale organization to spoil.

So just to be clear: I am not against spiritualism, nor any particular set of beliefs, so long as they don’t get too big for their own good. I think beliefs and zealotry are sort of the yin and yang of spiritualism. Beliefs are inevitable, even in coding and design, but whenever zealotry rears its head, bad stuff happens.

I hope you understand what I mean.

Software Needs Philosophers

Software needs philosophers.

This thought has been nagging at me for a year now, and recently it's been growing like a tumor. One that plenty of folks on the 'net would love to see kill me.

People don't put much stock in philosophers these days. The popular impression of philosophy is that it's just rhetoric, just frivolous debating about stuff that can never properly be answered. "Spare me the philosophy; let's stick to the facts!"

The funny thing is, it's philosophers who gave us the ability to think rationally, to stick to the facts. If it weren't for the work of countless philosophers, facts would still be getting people tortured and killed for discovering and sharing them.

Does it ever strike you as just a teeny bit odd that after a brief period where philosophy flourished, from maybe 400 B.C.E. to roughly 100 C.E., we went through a follow-on period of well over one thousand five hundred years during which the Roman Catholic Church enslaved everyone's minds and killed anyone who dared think differently?

What's weirder is that we tend to pretend it didn't really happen. We like to just skip right over the dominance of religion over our minds for a hundred generations, and think of religion today as a kindly old grandpa who's just looking out for us kids. No harm, no foul. Let bygones be bygones. Sure, there were massacres and crusades and genocides and torture chambers with teeth grinding and eyes bleeding and intestines torn out in the name of God. But we were all just kids then, right? Nobody does that kind of thing today, at least not in civilized countries.

We try not to think about the uncivilized ones.

It was philosophers that got us out of that Dark Ages mess, and no small number of them lost their lives in doing so. And today, the philosophy majors are the butts of the most jokes, because after the philosophers succeeded in opening our minds, we forgot why we needed them.

And if we stop to think about it at all, we think that it was other people, people who are very unlike us, who committed those atrocities in the name of Faith (regardless of whether it's faith in a god, or in a political party, or any other form of mind control carried out by force).

We like to think we live in an enlightened age, but we don't. Humans haven't changed significantly in 10,000 years. We're still killing and torturing each other. It's apparently incredibly easy to decide to kill someone and then do it. Happens every day, all around the world. Torture, too.

But those people are just people. If they had been born down the street from you, they'd have gone to school with you, been friends with you, learned to program with you, written blogs and comments, never tortured or killed anyone in the name of an idea. They'd have been you. Which means they are you; you just got lucky in where you were born.

One of the commenters on my last blog entry expressed the fervent wish that I drop dead. To be sure, they qualified it with "on the internet." But if they really feel that way, especially about something as hilariously and absurdly unimportant in the Grand Scheme as whether the Lisp programming language has any acceptable implementations, then what does it say about us?

Everyone who commented angrily on that blog entry was caught. I caught you, anonymous or not, being a religious fanatic. The only "negative" commenter who doesn't appear to be a religious zombie was Paul Costanza (ironic, since he claims to be the opinionated one), who relegated his comments to pedantic technical corrections. They're welcome, of course; I'm always looking to correct any technical misconceptions I harbor. But they're moot, since even if I was wrong about every single technical point I brought up in that entry, my overall point—Lisp is not an acceptable Lisp—remains largely uncontested by the commenters.

Some of them just don't get it, which is fine; no harm in that. If you've been using Lisp for years and years, and you've written books and articles and zillions of lines of Lisp code, then you're unlikely to remember anything about what it's like coming to Lisp for the first time. They're religious because they've forgotten what it's like to be a skeptic.

But make no mistake; a substantial percentage of people who take a side in any programming language discussion that devolves into a flamewar know exactly what the other side means, and they want to invoke the Ultimate Censorship: drop dead! Killing someone, after all, is one of the best ways to silence them. You also have to burn all their writings, which is getting harder these days; hence the increased vehemence on the 'net.

Those of you who've followed what I've written over the past year or so know where I'm going. I'm taking a stand, all right, and it's a very definite one. I'm finding myself drawn inexorably towards a single goal: stamping out technological religion, because I'm frigging tired of not being able to stick to the facts.

FACT: Java has no first-class functions and no macros. This results in warped code that hacks around the problem, and as the code base grows, it takes on a definite, ugly shape, one that's utterly unique to Java. Lisp people can see this clear as day. So can Python folks, so can Ruby folks. Java people flip out, and say "macros are too much power," or "what do u mean i dont understand u" or "fuck you, you jerk, Lisp will NEVER win."

You think I don't hear all that, and much more, in the hate mail I get every day?

I sure wouldn't want to be alone with a Java fanatic in a medieval torture chamber, because God only knows what they're capable of.

Turn the mirror towards Python, and what happens? Funny, but the Java folks will mail me saying: "yeah, I've always known I detested Python, and you really nailed exactly why. Thanks!" Meanwhile, Python folks are literally frothing at the mouth, looking for the "Kill That Bastard" key on their 101-key keyboards.

I turned the mirror towards Lisp yesterday. Had to go to the bathroom like nobody's business, and my wife was expecting me home any minute, so I rushed it out: just a few thoughts here and there. So the Gorgon only caught the tiniest glimpse of itself, but hell evidently hath no fury like that of a Lisper scorned, and all that.

It doesn't matter that I rushed it out. I'm glad I did; spending any more time on it, trying to get it "right" by looking up useless factoids like how you can override length's non-polymorphicness with some weird setting (when it plainly should just be the default), would have had the exact same net effect: Lisp zealots would have found some way to turn it into a flamewar. And I'd have been out two or three more hours.

Let's call it a troll, then, because it was poorly researched; it was just some months-old recollections of pain I'd gone through last year trying to commit to Common Lisp, after another year of trying the same with various flavors of Scheme and finding them all wanting. As far as I'm concerned, Lisp is unacceptable today; it's my opinion and just that, but I'll stick with it.

I still need Lisp; after you learn enough of it, it becomes part of your soul. I get my fix hacking elisp, and I do a lot of it. The commenters are quite right; I've never written anything substantial in Common Lisp, because in each of my serious attempts, there was too much friction. Risk/reward wasn't high enough, and believe me, I wanted it.

But after many attempts, I've given up on Common Lisp. They won't let me use it where I work, and there are probably more Lispers per capita where I work, including some famous ones, than at any other big company in the world. If we can't use it where I work, then it's frigging unacceptable; that's the shortest proof I can offer.

What I'm far more interested today is the situation that arises if you consider my post a troll. I'm far more interested in the social consequences of working in a world filled with religious fanatics of different religious persuasions. Especially given that it's a world in which "natural religion" has, by and large, been marginalized through the work of philosophers.

Let's look at this world in a little more detail, starting with Peter Siebel's comment, which I believe is the most interesting. Peter said:

I was trying to figure out why on earth you spent so much time writing about something that you apparently don't like. Then it hit me: HCGS. So thanks for your help.

His first sentence speaks volumes about the sociology. His viewpoint is exactly what they teach us all as kids: If you don't have anything nice to say, don't say anything at all. We like to think people have a right to believe whatever they want, and that it's not nice to say mean things about other people's beliefs, especially when their livelihoods are at stake.

That's where philosophers come in, folks. They pick your beliefs apart and show you in unforgettable ways the consequences of what you believe in. I'm no philosopher; I know basically nothing about it, but I can tell you I wish fervently that some great philosophers would come along and effect change in our technical society.

Because if nothing else, I can see the consequences of the way we're thinking about things. One of many such consequences is that languages aren't getting any better, and the worst offenders are Lisp and Scheme, which by rights should be racing along the innovation curve faster than their supposedly less capable peers. But they've stagnated worse than any other non-dead language I can think of [1].

Programming languages are religions. For a long while now I've been mildly uncomfortable calling it "religion," but I don't feel bad about it anymore. They're similar enough. At the top of the language religion is the language itself; it serves as the deity and the object of worship.

Like any other organized religion, there's always a Pope (or a politburo chairman, in countries where the government has brutally set itself up as what is for all intents the religion of choice): a spiritual leader that gives the religion the human touch. This person is almost always the language designer, of course. In Lisp's case it's complicated, because McCarthy, Sussman, and Steele aren't very active as spiritual leaders for their languages anymore.

Every major organized religion is a hierarchical government, and programming languages are no exception. You'll find equivalents of cardinals, bishops, priests, and laity in programming language camps: the closer you are to the fire, to the spiritual center, the higher your rank. It's a great way to quantify your perceived self-importance: a high-score list, in effect. Great for the ego, but it makes you a piss-poor debater, because you're so emotionally invested in your status.

You'd think your rank would be accrued by virtue of your technical and/or documentation contributions, but in practice it's usually more of a function of how many converts you've gained, how many followers you have, how much you've been spreading the Word.

That's why Paul Graham isn't the Pope of Lisp. He's eminently qualified, but unfortunately he's a heretic. Notice that almost none of the commenters on my last blog mentioned the PG argument I made. The only one who did (as of this writing) tried to make it an argument for Common Lisp.

Let's face it: you can't give those heretics too much press; people might start listening to them!Peter, are you beginning to understand why I write so much about something I apparently don't like? It's because I wanted to like it but found it fatally flawed, technically and culturally. It's as if I were a would-be convert to Roman Catholicism, but I can't bring myself to commit because I've seen too much of their role in creating a history that ironically we all wish we could rewrite.

I was born and raised a Roman Catholic, and I renounced it when I was 13 years old, after my Uncle Frank (a devout terrorist Catholic if there ever was one) told me to stop reading the Bible, that it would "really screw a person up" to do that, that you needed someone to interpret it for you. That wasn't the only reason I renounced it, but it'll suffice for our purposes.

Technologically I was born and raised an assembly-language programmer; at least that's what my first real job was, for five years after I got my CS degree. Assembly is just flagellation, though, and damned uncomfortable at that, so I joined the Church of Java for fully seven years. And practically at the very moment I'd finally tired of chafing at Java's limitations, Paul Graham came along and through his early essays, showed me Lisp. What a great new religion!

Problem is, each time you switch religions, the next one has less impact on you. Once a Catholic, always a Catholic, they say. I don't know what that means for me, since I was raised by the assembly-language wolf, but it appears to mean that I'm never going to be enthralled with another programming language. And now that I've swallowed the red pill, what choice do I have? I need to try to show people what's out there.

Interestingly, it was Peter Siebel's most excellent book, Practical Common Lisp, that played the role of Uncle Frank and killed my desired to continue with Common Lisp. Peter was the first person to show me the beast's underbelly. Every other Lisp book had pretended it was pure and beautiful and uncorrupted, because they left all the nastiness out as "implementation-defined." Once I saw what you really need to do in order to build something resembling a portable Lisp code base, and then had a few runs at it myself, I threw in the towel.

I much prefer LISP the idea to Lisp the implementation [2].

I can tell you this: I've tried writing this essay for a year. I've tried fully a dozen times. I've tackled it from a dozen angles. I've wanted to say it—software needs philosophers!—so many times, in so many ways. We need great thinkers—the Fyodor Dostoyevskys and David Humes and Aristotles and Jean-Paul Sartres and Ben Franklins and Galileo Galileis and Bertrand Russells and Albert Einsteins to show us the way through the Software Dark Ages we're in today: a time that will doubtless be remembered as every bit as mired in darkness and ignorance as the Dark Ages themselves.

But I've failed. This isn't the essay I wanted to write, because I'm neither a great thinker nor a great writer. However, you might be: if not now, then perhaps someday. So I think it's better to get the idea out now than to hoard it in the hopes of someday writing a world-changing essay.

For those of you who were surprised at the suddenness and vehemence of the Lisp community's backlash to my little rant, I hope I've helped shed a little light, helped you see its inevitability. Basically they've had a lot of practice. Lisp is one of the oldest technology religions, and they've both experienced and doled out their share of religious persecution.

But that's not the lesson you should take away. The lesson is that they are you. Whenever you hear someone ranting about something you take for granted as wonderful and praiseworthy, and you're wondering why they don't leave well enough alone so we can all get back to our incestuous cheerleading, just remember: we went from the Dark Ages to our reasonably enlightened society today by questioning our most cherished beliefs.

So keep questioning them.

Notes

1. Yes, I've read all of R6RS. It's a lukewarm compromise that punts on most of the important issues. It's not going to make Scheme any more successful than it is today, which to me feels practically criminal; it was their one big chance to break out of the rut they're in. But it doesn't matter. Let's pretend this footnote is just a troll. If your hackles went up, then you're a techno-religious zombie, and I hope in my lifetime to find you a cure. Try your best to think about that long and hard before responding.

2. For the record, the commenter I agree the most with is the one who said the problem basically boils down to an IDE issue. SLIME doesn't cut it, either, as beautiful as SLIME is. Can't use it on Windows to save your life, for instance. But that's one of a thousand problems with the Lisp IDE situation; it's pointless to try to discuss them all in blogger. It's probably pointless to discuss them at all, because it's just going to make me more miserable that no decent IDE exists for Lisp, except for Emacs-as-Elisp-IDE. Which is why I get my Lisp fix by hacking elisp these days.

Author’s Note: Code's Worst Enemy

This is another rant against Java. However, this time it has little or nothing to do with the type system. It’s more bemoaning Java’s fundamental non-composability and non-compressibility. And, for those of you squirming over the attack on Java—it’s the last one in this book. Whew!

But the core issue goes deeper than just Java. I really do believe—even more so now, five years later—that all code is in a sense bad. It’s a liability. It’s a scaling problem, a maintenance problem, a tooling problem, an analysis problem, a build problem, an everything-problem. The more you have of it, the bigger the problem is.

I know we all love code. I love to write code just as much as the next person. It’s fun, and it’s important. But code is data. We all know this. And we know that large data sets are a problem. Even Google, who may be the best in the world at handling the world’s largest datasets, can trivially generate data sets that are way too big to be practical.

So the logic is inescapable: there is such a thing as too much code. It’s undeniable. Code feels light and fluffy, but it is highly structured text data that has unusually complex processing requirements. And in sufficient quantities, it becomes a pain in every company’s ass.

The question then, boils down to how much compression is tolerable, and how much you are willing to sacrifice in order to keep your code base small.

In this essay I make what I hope is a compelling (or at least thought-provoking) argument that if you are a Java programmer, you need to make the “ultimate sacrifice” and ditch your language. Java programmers will of course not listen to me. But it’s their burden to bear.

Code's Worst Enemy

I'm a programmer, and I'm on vacation today. Guess what I'm doing? As much as I'd love to tell you I'm sipping Mai Tais in the Bahamas, what I'm actually doing on my vacation is programming.

So it's a "vacation" only in the HR sense—I'm taking official time off work, to give myself some free time to get my computer game back online. It's a game I started writing about 10 years ago and spent about 7 years developing. It's been offline for a while, and I need to bring it back up, in part so the players will stop stalking me. It's going to take me at least a week of all-day days, so I had to take a vacation from work to make it happen.

Why did my game go offline? Not for want of popularity. It's a pretty successful game for a mostly part-time effort from one person. I've had over a quarter million individuals try it out (at least getting as far as creating a character), and tens of thousands of people who've spent countless hours playing it over the years. It's won awards and been featured in magazines; it's attracted the attention of game portals, potential investors, and whole schools full of kids.

Yup, kids. It was supposed to be a game for college students, but it's been surprisingly popular with teenagers and even pre-teens, who you'd think would be off playing some 3D console game or other. But I wrote it for myself, and apparently there are sufficient people who like the same kinds of games I do to create a sustainable community.

I took the game down for all sorts of mundane reasons—it needed some upgrades, work got busy, I didn't have lots of time at nights, etc. But the mundane reasons all really boil down to just one rather deeper problem: the code base is too big for one person to manage.

I've spent nearly 10 years of my life building something that's too big.

I've done a lot of thinking about this—more than you would probably guess. It's occupied a large part of my technical thinking for the past four or five years, and has helped shaped everything I've written in that time, both in blogs and in code.

For the rest of this little rant, I'm going to assume that you're a young, intelligent, college age or even high school age student interested in becoming a better programmer, perhaps even a great programmer.

(Please—don't think I'm implying that I'm a great programmer. Far from it. I'm a programmer who's committed decades of terrible coding atrocities, and in the process I've learned some lessons that I'm passing along to you in the hope that it'll help you in your quest to become a great programmer.)

I have to make the assumption that you're young in order to make my point, because if I assume I'm talking to "experienced" programmers, my blood pressure will rise, and I will not be able to focus for long enough to finish my rant. You'll see why in a bit.

Fortunately for me, you're young and eager to learn, so I can tell you how things really are. Just keep your eyes open for the next few years, and watch to see if I'm right.

Minority View

I happen to hold a hard-won minority opinion about code bases. In particular I believe, quite staunchly I might add, that the worst thing that can happen to a code base is size.

I say "size" as a placeholder for a reasonably well-formed thought for which I seem to have no better word in my vocabulary. I'll have to talk around it until you can see what I mean, and perhaps provide me with a better word for it. The word "bloat" might be more accurate, since everyone knows that "bloat" is bad, but unfortunately most so-called experienced programmers do not know how to detect bloat, and they'll point at severely bloated code bases and claim they're skinny as a rail.

Good thing we're not talking to them, eh?

I say my opinion is hard-won because people don't really talk much about code base size; it's not widely recognized as a problem. In fact it's widely recognized as a non-problem. This means that anyone sharing my minority opinion is considered a borderline lunatic, since what rational person would rant against a non-problem?

People in the industry are very excited about various ideas that nominally help you deal with large code bases, such as IDEs that can manipulate code as "algebraic structures," and search indexes, and so on. These people tend to view code bases much the way construction workers view dirt: they want great big machines that can move the dirt this way and that. There's conservation of dirt at work: you can't compress dirt, not much, so their solution set consists of various ways of shoveling the dirt around. There are even programming interview questions, surely metaphorical, about how you might go about moving an entire mountain of dirt, one truck at a time.

Industry programmers are excited about solutions to a big non-problem. It's just a mountain of dirt, and you just need big tools to move it around. The tools are exciting but the dirt is not.

My minority opinion is that a mountain of code is the worst thing that can befall a person, a team, a company. I believe that code weight wrecks projects and companies, that it forces rewrites after a certain size, and that smart teams will do everything in their power to keep their code base from becoming a mountain. Tools or no tools. That's what I believe.

It turns out you have to have something bad happen to you before you can hold my minority opinion. The bad thing that happened to me is that I wrote a beautiful game in an ugly language, and the result was lovely on the outside and quite horrific internally. The average industry programmer today would not find much wrong with my code base, aside from the missing unit tests (which I now regret) which would, alas, double the size of my game's already massive 500,000-line code base. So the main thing they would find wrong with it is, viewed in a certain way, that it's not big enough. If I'd done things perfectly, according to today's fashions, I'd be even worse off than I am now.

Some people will surely miss my point, so I'll clarify: I think unit testing is great. In fact I think it's critical, and I vastly regret not having unit tests for my game. My point is that I wrote the game the way most experienced programmers would tell you to write that kind of system, and it's now an appallingly unmanageable code base. If I'd done the "right thing" with unit tests, it would be twice as appalling! The apparent paradox here is crucial to understanding why I hold my minority belief about code base size.

Most programmers never have anything truly bad happen to them. In the rare cases when something bad happens, they usually don't notice it as a problem, any more than a construction worker notices dirt as a problem. There's just a certain amount of dirt at every site, and you have to deal with it: it's not "bad"—it's just a tactical challenge.

Many companies are faced with multiple million lines of code, and they view it as a simple tools issue, nothing more: lots of dirt that needs to be moved around occasionally.

Most people have never had to maintain a half-million line code base single-handedly, so their view of things will probably be different from mine. Hopefully you, being the young, eager-to-learn individual that you are, will realize that the only people truly qualified to express opinions on this matter are those who have lived in and helped create truly massive code bases.

You may hear some howling in response to my little rant today, and a lot of hand-wavy "he just doesn't understand" dismissiveness. But I posit that the folks making these assertions have simply never been held accountable for the messes they've made.

When you write your own half-million-line code base, you can't dodge accountability. I have nobody to blame but myself, and it's given me a perspective that puts me in the minority.

It's not just from my game, either. That alone might not have taught me the lesson. In my twenty years in the industry, I have hurled myself forcibly against some of the biggest code bases you've ever imagined, and in doing so I've learned a few things that most people never learn, not in their whole career. I'm not asking you to make up your mind on the matter today. I just hope you'll keep your eyes and ears open as you code for the next few years.

Invisible Bloat

I'm going to try to define bloat here. I know in advance that I'll fail, but hopefully just sketching out the problem will etch out some patterns for you.

There are some things that can go wrong with code bases that have a nice intuitive appeal to them, inasmuch as it's not difficult for most people to agree that they're "bad."

One such thing is complexity. Nobody likes a complex code base. One measure of complexity that people sometimes use is "cyclomatic complexity," which estimates the possible runtime paths through a given function using a simple static analysis of the code structure.

I'm pretty sure that I don't like complex code bases, but I'm not convinced that cyclomatic complexity measurements have helped. To get a good cyclomatic complexity score, you just need to break your code up into smaller functions. Breaking your code into smaller functions has been a staple of "good design" for at least 10 years now, in no small part due to the book Refactoring by Martin Fowler.

The problem with Refactoring as applied to languages like Java, and this is really quite central to my thesis today, is that Refactoring makes the code base larger. I'd estimate that fewer than 5 percent of the standard refactorings supported by IDEs today make the code smaller. Refactoring is like cleaning your closet without being allowed to throw anything away. If you get a bigger closet, and put everything into nice labeled boxes, then your closet will unquestionably be more organized. But programmers tend to overlook the fact that spring cleaning works best when you're willing to throw away stuff you don't need.

This brings us to the second obviously-bad thing that can go wrong with code bases: copy and paste. It doesn't take very long for programmers to learn this lesson the hard way. It's not so much a rule you have to memorize as a scar you're going to get whether you like it or not. Computers make copy-and-paste really easy, so every programmer falls into the trap once in a while. The lesson you eventually learn is that code always changes, always always always, and as soon as you have to change the same thing in N places, where N is more than 1, you'll have earned your scar.

However, copy-and-paste is far more insidious than most scarred industry programmers ever suspect. The core problem is duplication, and unfortunately there are patterns of duplication that cannot be eradicated from Java code. These duplication patterns are everywhere in Java; they're ubiquitous, but Java programmers quickly lose the ability to see them at all.

Java programmers often wonder why Martin Fowler "left" Java to go to Ruby. Although I don't know Martin, I think it's safe to speculate that "something bad" happened to him while using Java. Amusingly (for everyone except perhaps Martin himself), I think that his "something bad" may well have been the act of creating the book Refactoring, which showed Java programmers how to make their closets bigger and more organized, while showing Martin that he really wanted more stuff in a nice, comfortable, closet-sized closet.

Martin, am I wrong?

As I predicted would happen, I haven't yet defined bloat except in the vaguest terms. Why is my game code base half a million lines of code? What is all that code doing?

Design Patterns Are Not Features

The other seminal industry book in software design was Design Patterns, which left a mark the width of a two-by-four on the face of every programmer in the world—assuming the world contains only Java and C++ programmers.

Design Patterns was a mid-1990s book that provided twenty-three fancy new boxes for organizing your closet, plus an extensibility mechanism for defining new types of boxes. It was really great for those of us who were trying to organize jam-packed closets with almost no boxes, bags, shelves or drawers. All we had to do was remodel our houses to make the closets four times bigger, and suddenly we could make them as clean as a Nordstrom merchandise rack.

Interestingly, salespeople didn't get excited about Design Patterns. Nor did PMs, nor marketing folks, nor even engineering managers. The only people who routinely get excited about Design Patterns are programmers, and only programmers who use certain languages. Perl programmers were, by and large, not very impressed with Design Patterns. However, Java programmers misattributed this; they concluded that Perl programmers must be slovenly, no good bachelors who pile laundry in their closets up to the ceiling.

It's obvious now, though, isn't it? A design pattern isn't a feature. A Factory isn't a feature, nor is a Delegate nor a Proxy nor a Bridge. They "enable" features in a very loose sense, by providing nice boxes to hold the features in. But boxes and bags and shelves take space. And design patterns—at least most of the patterns in the "Gang of Four" book—make code bases get bigger. Tragically, the only GoF pattern that can help code get smaller (Interpreter) is utterly ignored by programmers who otherwise have the names of Design Patterns tattooed on their various body parts.

Dependency Injection is an example of a popular new Java design pattern that programmers using Ruby, Python, Perl, and JavaScript have probably never heard of. And if they've heard of it, they've probably (correctly) concluded that they don't need it. Dependency Injection is an amazingly elaborate infrastructure for making Java more dynamic in certain ways that are intrinsic to higher-level languages. And—you guessed it—DI makes your Java code base bigger.

Bigger is just something you have to live with in Java. Growth is a fact of life. Java is like a variant of the game of Tetris in which none of the pieces can fill gaps created by the other pieces, so all you can do is pile them up endlessly.

Millions of Lines of Code

I recently had the opportunity to watch a self-professed Java programmer give a presentation in which one slide listed Problems (with his current Java system) and the next slide listed Requirements (for the wonderful new vaporware system). The number one problem he listed was code size: his system has millions of lines of code.

Wow! I've sure seen that before, and I could really empathize with him. Geoworks had well over ten million lines of assembly code, and I'm of the opinion that this helped bankrupt them (although that also appears to be a minority opinion—those industry programmers just never learn!) And I worked at Amazon for seven years; they have well over a hundred million lines of code in various languages, and "complexity" is frequently cited internally as their worst technical problem.

So I was really glad to see that this guy had listed code size as his number one problem.

Then I got my surprise. He went on to his Requirements slide, on which he listed "must scale to millions of lines of code" as a requirement. Everyone in the room except me just nodded and accepted this requirement. I was floored.

Why on earth would you list your number one problem as a requirement for the new system? I mean, when you're spelling out requirements, generally you try to solve problems rather than assume they're going to be created again. So I stopped the speaker and asked him what the heck he was thinking.

His answer was: well, his system has lots of features, and more features means more code, so millions of lines are Simply Inevitable. "It's not that Java is verbose!" he added—which is pretty funny, all things considered, since I hadn't said anything about Java or verbosity in my question.

The thing is, if you're just staring in shock at this story and thinking "how could that Java guy be so blind," you are officially a minority in the programming world. An unwelcome one, at that.

Most programmers have successfully compartmentalized their beliefs about code base size. Java programmers are unusually severe offenders but are by no means the only ones. In one compartment, they know big code bases are bad. It only takes grade-school arithmetic to appreciate just how bad they can be. If you have a million lines of code, at 50 lines per "page," that's 20,000 pages of code. How long would it take you to read a 20,000-page instruction manual? The effort to simply browse the code base and try to discern its overall structure could take weeks or even months, depending on its density. Significant architectural changes could take months or even years.

In the other compartment, they think their IDE makes the code size a non-issue. We'll get to that shortly.

And a million lines is nothing, really. Most companies would love to have merely a million lines of code. Often a single team can wind up with that much after a couple years of hacking. Big companies these days are pushing tens to hundreds of millions of lines around.

I'll give you the capsule synopsis, the one-sentence summary of the learnings I had from the Bad Thing that happened to me while writing my game in Java: if you begin with the assumption that you need to shrink your code base, you will eventually be forced to conclude that you cannot continue to use Java. Conversely, if you begin with the assumption that you must use Java, then you will eventually be forced to conclude that you will have millions of lines of code.

Is it worth the trade-off? Java programmers will tell you Yes, it's worth it. By doing so they're tacitly nodding to their little compartment that realizes big code bases are bad, so you've at least won that battle.

But you should take anything a "Java programmer" tells you with a hefty grain of salt, because an "X programmer," for any value of X, is a weak player. You have to cross-train to be a decent athlete these days. Programmers need to be fluent in multiple languages with fundamentally different "character" before they can make truly informed design decisions.

Recently I've been finding that Java is an especially bad value for X. If you absolutely must hire an X programmer, make sure it's Y.

I didn't really set out to focus this rant on Java (and Java clones like C#, which despite now being a "better" language still has Java's fundamental character, making it only a marginal win at best). To be sure, my minority opinion applies to any code base in any language. Bloat is bad.

But I find myself focusing on Java because I have this enormous elephant of a code base that I'm trying to revive this week. Can you blame me? Hopefully someone with a pet C++ elephant can come along and jump on the minority bandwagon with me. For now, though, I'll try to finish my explanation of bloat as a bona-fide problem using Java for context.

Can IDEs Save You?

The Java community believes, with near 100 percent belief compliance, that modern IDEs make code base size a non-issue. End of story.

There are several problems with this perspective. One is simple arithmetic again: given enough code, you eventually run out of machine resources for managing the code. Imagine a project with a billion lines of code, and then imagine trying to use Eclipse or IntelliJ on that project. The machines—CPU, memory, disk, network—would simply give up. We know this because twenty-million line code bases are already moving beyond the grasp of modern IDEs on modern machines.

Heck, I've never managed to get Eclipse to pull in and index even my 500,000-line code base, and I've spent weeks trying. It just falls over, paralyzed. It literally hangs forever (I can leave it overnight, and it makes no progress). Twenty million lines? Forget about it.

It may be possible to mitigate the problem by moving the code base management off the local machine and onto server clusters. But the core problem is really more cultural than technical: as long as IDE users refuse to admit there is a problem, it's not going to get solved.

Going back to our crazed Tetris game, imagine that you have a tool that lets you manage huge Tetris screens that are hundreds of stories high. In this scenario, stacking the pieces isn't a problem, so there's no need to be able to eliminate pieces. This is the cultural problem: they don't realize they're not actually playing the right game anymore.

The second difficulty with the IDE perspective is that Java-style IDEs intrinsically create a circular problem. The circularity stems from the nature of programming languages: the "game piece" shapes are determined by the language's static type system. Java's game pieces don't permit code elimination because Java's static type system doesn't have any compression facilities—no macros, no eval, no declarative data structures, no templates, nothing that would permit the removal of the copy-and-paste duplication patterns that Java programmers think of as "inevitable boilerplate," but which are in fact easily factored out in dynamic languages.

Completing the circle, dynamic features make it more difficult for IDEs to work their static code-base-management magic. IDEs don't work as well with dynamic code features, so IDEs are responsible for encouraging the use of languages that require… IDEs. Ouch.

Java programmers understand this at some level; for instance, Java's popular reflection facility, which allows you to construct method names on the fly and invoke those methods by name, defeats an IDE's ability to perform basic refactorings such as Rename Method. But because of successful compartmentalization, Java folks point at dynamic languages and howl that (some) automated refactorings aren't possible, when in fact they're just as possible in these languages as they are in Java—which is to say, they're partly possible. The refactorings will "miss" to the extent that you're using dynamic facilities, whether you're writing in Java or any other language. Refactorings are essentially never 100 percent effective, especially as the code base is shipped off-site with public APIs: this is precisely why Java has a deprecation facility. You can't rename a method on everyone's machine in the world. But Java folks continue spouting the provably false belief that automated refactorings work on "all" their code.

I'll bet that by now you're just as glad as I am that we're not talking to Java programmers right now! Now that I've demonstrated one of the many ways in which they're utterly irrational, it should be clear that their response isn't likely to be a rational one.

Rational Code Size

The rational response would be to take a very big step back, put all development on hold, and ask a difficult question: "What should I be using instead of Java?"

I did that about four years ago. That's when I stopped working on my game, putting it into maintenance mode. I wanted to rewrite it down to 100,000 to 150,000 lines, somewhere in that vicinity, with the exact same functionality.

It took me six months to realize it can't be done with Java, not even with the stuff they added to Java 5, and not even with the stuff they're planning for Java 7 (even if they add the cool stuff, like non-broken closures, that the Java community is resisting tooth and nail).

It can't be done with Java. But I do have a big investment in the Java virtual machine, for basically the same reason that Microsoft now has a big investment in .NET. Virtual machines make sense to me now. I mean, they "made sense" at some superficial level when I read the marketing brochures, but now that I've written a few interpreters and have dug into native-code compilers, they make a lot more sense. It's another rant as to why, unfortunately.

So taking for granted today that VMs are "good," and acknowledging that my game is pretty heavily tied to the JVM—not just for the extensive libraries and monitoring tools, but also for more subtle architectural decisions like the threading and memory models—the rational answer to code bloat is to use another JVM language.

One nice thing about JVM languages is that Java programmers can learn them pretty fast, because you get all the libraries, monitoring tools, and architectural decisions for free. The downside is that most Java programmers are X programmers, and, as I said, you don't want X programmers on your team.

But since you're not one of those people who've decided to wear bell-bottom polyester pants until the day you die, even should you live unto five hundred years, you're open to language suggestions. Good for you!

Three years ago, I set out to figure out which JVM language would be the best code-compressing successor to Java. That took a lot longer than I expected, and the answer was far less satisfactory than I'd anticipated. Even now, three years later, the answer is still a year or two away from being really compelling. (Author’s note, six years later: still way out!)

I'm patient now, though, so after all the dust settles, I know I've got approximately a two-year window during which today's diehard Java programmers are writing their next multi-million line disaster. Right about the time they're putting together their next Problems/Requirements slide, I think I'll actually have an answer for them.

In the meantime, I'm hoping that I'll have found time to rewrite my game in this language, down from 500,000 lines to 150,000 lines with the exact same functionality (plus at least another 50,000+ for unit tests).

The Next Java

So what JVM language is going to be the Next Java?

Well, if you're going for pure code compression, you really want a Lisp dialect: Common Lisp or Scheme. And there are some very good JVM implementations out there. I've used them. Unfortunately, a JVM language has to be a drop-in replacement for Java (otherwise a port is going to be a real logistics problem), and none of the Lisp/Scheme implementors seems to have that very high on their priority list.

Plus everyone will spit on you. People who don't habitually spit will expectorate up to thirty feet, like zoo camels, in order to bespitle you if you even suggest the possibility of using a Lisp or Scheme dialect at your company.

So it's not gonna be Lisp or Scheme. We'll have to sacrifice some compression for something a bit more syntactically mainstream.

It could theoretically be Perl 6, provided the Parrot folks ever actually get their stuff working, but they're even more patient than I am, if you get my drift. Perl 6 really is a pretty nice language design, for the record—I was really infatuated with it back in 2001. The love affair died about five years ago, though. And Perl 6 probably won't ever run on the JVM. It's too dependent on powerful Parrot features that the JVM will never support. (I'd venture that Parrot probably won't ever support them either, but that would be mean.)

Most likely New Java is going to be an already reasonably popular language with a very good port to the JVM. It'll be a language with a dedicated development team and a great marketing department.

That narrows the field from 200+ languages down to maybe three or four: JRuby, Groovy, Rhino (JavaScript), and maybe Jython if it comes out of its coma.

Each of these languages (as does Perl 6) provides mechanisms that would permit compression of a well-engineered 500,000-line Java code base by 50 to 75 percent. Exactly where the dart lands remains to be seen, but I'm going to try it myself.

I personally tried Groovy and found it to be an ugly language with a couple of decent ideas. It wants to be Ruby but lacks Ruby's elegance (or Python's for that matter). It's been around a long time and does not seem to be gaining any momentum, so I've ruled it out for my own work. (And I mean permanently—I will not look at it again. Groovy's implementation bugs have really burned me.)

I like Ruby and Python a lot, but neither JVM version was up to snuff when I did my evaluation three years ago. JRuby has had a lot of work done to it in the meantime. If the people I work with weren't so dead-set against Ruby, I'd probably go with that, and hope like hell that the implementation is eventually "fast enough" relative to Java.

As it happens, though, I've settled on Rhino. I'll be working with the Rhino dev team to help bring it up to spec with EcmaScript Edition 4. I believe that ES4 brings JavaScript to rough parity with Ruby and Python in terms of (a) expressiveness and (b) the ability to structure and manage larger code bases. Anything it lacks in sugar, it more than makes up for with its optional type annotations. And I think JavaScript (especially on ES4 steroids) is an easier sell than Ruby or Python to people who like curly braces, which is anyone currently using C++, Java, C#, JavaScript, or Perl. That's a whole lot of curly brace lovers. I'm nothing if not practical these days. (Author’s note: ES4 died a horrible death in standards committee. Alas.)

I don't expect today's little rant to convince anyone to share my minority opinion about code base size. I know a that few key folks (Bill Gates, for instance, as well as Dave Thomas, Martin Fowler, and James Duncan Davidson) have independently reached the same conclusion: namely, that bloat is the worst thing that can happen to code. But they all got there via painful things happening to them.

I can't exactly wish for something painful to happen to Java developers, since hey, it's already happening; they've already taught themselves to pretend it's not hurting them.

But as for you, the eager young high school or college student who wants to become a great programmer someday, hopefully I've given you an extra dimension to observe as your tend your code gardens for the next few years.

When you're ready to make the switch, well, Mozilla Rhino will be ready for you. It works great today and will be absolutely outstanding a year from now. And I sincerely hope that JRuby, Jython, and friends will also be viable Java alternatives for you as well. You might even try them out now and see how it goes.

Your code base will thank you for it.

Author’s Note: A Little Anti-Anti-Hype

Continuing in my tradition of making fun of nearly all programming languages, in this one I make fun of Python. I don’t make much of a fuss about the language itself, because even though internally it’s somewhat horrible, on the outside it’s pleasant enough.

But I really ribbed the community hard here, because I think that at the time, they were digging themselves a hole that would soon be too deep for them to escape.

They’ve gotten a lot better since then. I’m not claiming credit… but there was a discussion of the Python marketing problem on the Python discussion boards a few weeks after I posted this, where at least one poster cited my article. And their ancient, crufty website got a huge overhaul soon after (which broke most of the links in my article, but no biggie).

Even a small catalyst like my rant here can have a surprising ripple effect, if it happens at just the right time, and if it reaches people in just the right way. Which, in this case, was to irritate them. But hey, whatever gets their attention.

Today, Python is worlds different. Some Pythonistas may want to deny it, and try to claim that they’ve been super nice to newbies all along. But it’s hard to rewrite history in the presence of web caches; anyone sufficiently determined could debunk their claim. And in any case I think it would be more appropriate for them to give themselves a pat on the back for having turned their culture around. I wish the Lisp folks could pull that off.

A Little Anti-Anti-Hype

Note: if this article enrages you, please read the next article “Bambi Meets Godzilla” that explains some of my outrageous comments a bit more gently and thoroughly. Hopefully it shows that I'm not Python-bashing here.

Everyone's buzzing about Bruce Eckel's “anti-hype” article. I hope the irony isn't lost on him.

The thrust of Eckel's article appears to be that hyper-enthusiasm is diminishing the Ruby camp's message, and it's spoiling a good gentleman's argument. Those darn hyper-enthusiasts are focusing relentlessly on how cool Ruby is and how much they like it, when what's really needed here is a balanced, objective, neutral, moderated, standards-based, point-by-point, academic discussion of Python vs. Ruby, in which we can all make well-informed decisions, and may the best language win, as long as it's Python.

Python folks never really did understand marketing.

I'm surprised we need a history lesson here; we've all been through this so many times before. But let's look once again at the basics of language adoption.

First, inferior languages and technologies are just as likely to win. Maybe even more likely, since it takes less time to get them right. Java beat Smalltalk; C++ beat Objective-C; Perl beat Python; VHS beat Beta; the list goes on. Technologies, especially programming languages, do not win on merit. They win on marketing. Begging for fair, unbiased debate is going to get your language left in the dust.

You can market a language by pumping money into a hype machine, the way Sun and IBM did with Java, or Borland did back with Turbo Pascal. It's pretty effective, but prohibitively expensive for most. More commonly, languages are marketed by a small group of influential writers, and the word-of-mouth hyping extends hierarchically down into the workplace, where a bunch of downtrodden programmers wishing they were having more fun stage a coup and start using a new "forbidden" language on the job. Before long, hiring managers start looking for this new language on resumes, which drives book sales, and the reactor suddenly goes supercritical.

Perl's a good example—how did it beat Python? They were around at more or less the same time. Perl might predate Python by a few years, but not enough for it to matter much. Perl captured roughly ten times as many users as Python, and has kept that lead for a decade. How? Perl's success is the result of Larry Wall's brilliant marketing, combined with the backing of a strong publisher in O'Reilly.

Programming Perl was a landmark language book: it was chatty, it made you feel welcome, it was funny, and you felt as if Perl had been around forever when you read it; you were just looking at the latest incarnation. Double marketing points there: Perl was hyped as a trustworthy, mature brand name (like Barnes and Noble showing up overnight and claiming they'd been around since 1897 or whatever), combined with that feeling of being new and special. Larry continued his campaigning for years. Perl's ugly deficiencies and confusing complexities were marketed as charming quirks. Perl surrounded you with slogans, jargon, hip stories, big personalities, and most of all, fun. Perl was marketed as fun.

What about Python? Is Python hip, funny, and fun? Not really. The community is serious, earnest, mature, and professional, but they're not much focused on fun, which is an important part of marketing languages. (Author’s note: This originally read “…but they’re about as fun as a bunch of tax collectors.” That made them Mad. Very Mad. Everyone else thought it was funny, though. And accurate.)

One could write a fat book about this, but just to give you the flavor, consider what happens when you type "python" at a command prompt. It fires up a little interactive interpreter. At the prompt, if you type "quit," it responds with “Use Ctrl-D (i.e., EOF) to exit.”

Well that's not very nice, is it? It knows you want to quit, even going so far as to call you an EOF, whatever that means. (Yes, you and I both know, but is it really the right thing to show to a beginner? Hardly.) Why didn't it just quit, then?

If you were to bring this issue up on a Python newsgroup at any time in the past 10 years, someone would tersely have instructed you to go look at the FAQ. Or they'd have explained that having “quit” quit would be a strict violation of the semantics of the REPL, which has no a priori knowledge of English, and Ctrl-D is universally recognized as the EOF char on most terminal emulators, excepting of course broken ones on win32 and VAX platforms, and the interactive shell's clean design allows the interpreter to treat the input as if it were coming from a file or similar stream, blah Blah BLAH, ergo, the current behavior is correct, quod erat demonstrandum.

Never mind that it's patently obvious that "quit" should just quit the frigging shell, semantics be damned. They don't care a whit, because they're focused on the "right thing" at the expense of the user experience. There's an old adage for this; it's called "missing the forest for the trees."

Of course it's just as difficult to figure out how to exit the Perl shell, if not more so. But if you were to bring it up on a mailing list or newsgroup, some nice Perl person would come along, eager to show you how to add one more snippet of job security to your lineup of Perl folklore, and would spend an hour explaining how cool it is that you can quit the shell with a single keystroke, one that works in other Unix commands as well, and then maybe show you how to hack the Perl binary so that "quit" also exits the shell for you. The difference is huge: both shells have that crappy misfeature, but Python folks will bore you with justifications while the Perl folks excite you with marketing.

Pedantry: it’s just how things work in the Python world. The status quo is always correct by definition. If you don’t like something, you are incorrect. If you want to suggest a change, put in a PEP, Python’s equivalent of Java’s equally glacial JSR process. The Python FAQ goes to great lengths to rationalize a bunch of broken language features. They’re obviously broken if they’re frequently asked questions, but rather than ‘fessing up and saying “we’re planning on fixing this," they rationalize that the rest of the world just isn’t thinking about the problem correctly. Every once in a while some broken feature is actually fixed (e.g., lexical scoping), and they say they changed it because people were “confused." Note that Python is never to blame.

In contrast, Matz is possibly Ruby's harshest critic; his presentation “How Ruby Sucks” exposes so many problems with his language that it made my blood run a bit cold. But let's face it: all languages have problems. I much prefer the Ruby crowd's honesty to Python's blaming, hedging, and overt rationalization. (Author’s note: funny how all those links are borked now. Coincidence?)

As for features, Perl had a very different philosophy from Python: Larry would add in just about any feature anyone asked for. Over time, the Perl language has evolved from a mere kitchen sink into a vast landfill of flotsam and jetsam from other languages. But they never told anyone: "Sorry, you can't do that in Perl." That would have been bad for marketing.

Today, sure, Perl's ugly; it's got generations of cruft, and they've admitted defeat by turning their focus to Perl 6, a complete rewrite. If Perl had started off with a foundation as clean as Ruby's, it wouldn't have had to mutate so horribly to accommodate all its marketing promises, and it'd still be a strong contender today. But now it's finally running out of steam. Larry's magical marketing vapor is wearing off, and people are realizing that Perl's useless toys (references, contexts, typeglobs, ties, etc.) were only fun back when Perl was the fastest way to get things done. In retrospect, the fun part was getting the job done and showing your friends your cool software; only half of Perl's wacky features were helping with that.

So now we have a void. Perl's running out of steam for having too many unnecessary features; Java's running out of steam for being too bureaucratic. Both are widely beginning to be perceived as offering too much resistance to getting cool software built. This void will be filled by… you guessed it: marketing. Pretty soon everyone (including hiring managers) will see which way the wind is blowing, and one of Malcolm Gladwell's tipping points will happen.

We're in the middle of this tipping-point situation right now. In fact it may have already tipped, with Ruby headed to become the winner, a programming-language force as prominent on resumes and bookshelves as Java is today. This was the entire point of Bruce Tate's book. You can choose to quibble over the details, as Eckel has done, or you can go figure out which language you think is going to be the winner, and get behind marketing it, rather than complaining that other language enthusiasts aren't being fair.

Could Python be the next mega-language? Maybe. It's a pretty good language (not that this really matters much). To succeed, they'd have to get their act together today. Not in a year, or a few months, but today—and they'd have to realize they're behind already. Ruby's a fine language, sure, but now it has a killer app. Rails has been a huge driving and rallying force behind Ruby adoption. The battleground is the web framework space, and Python's screwing it up badly. There are at least five major Python frameworks that claim to be competing with Rails: Pylons, Django, TurboGears, Zope, and Subway. That's at least three (maybe four) too many. From a marketing perspective, it doesn't actually matter which one is the best, as long as the Python community gets behind one of them and starts hyping it exclusively. If they don't, each one will get 20 percent of the developers, and none will be able to keep pace with the innovation in Rails.

The current battle may be over web frameworks, but the war is broader than that. Python will have to get serious about marketing, which means finding some influential writers to crank out some hype books in a hurry. Needless to say, they also have to abandon their anti-hype position, or it's a lost cause. Sorry, Bruce. Academic discussions won't get you a million new users. You need faith-based arguments. People have to watch you having fun, and envy you.

My guess is that the Python and Java loyalists will once again miss the forest for the trees. They'll debate my points one by one, and declare victory when they've proven beyond a doubt that I'm mistaken: that marketing doesn't really matter. Or they'll say, "gosh, it's not really a war; there's room for all of us," and they'll continue to wonder why the bookshelves at Barnes and Noble are filling up with Ruby books.

I won't be paying much attention though, 'cuz Ruby is soooo cool. Did I mention that "quit" exits the shell in Ruby? It does, and so does Ctrl-D. Ruby's da bomb. And Rails? Seriously, you don't know what you're missing. It's awesome. Ruby's dad could totally beat up Python's dad. Check out Why's Poignant Guide if you don't b'lieve me. Ruby's WAY fun—it's like the only language I want to use these days. It's so easy to learn, too. Not that I'm hyping it or anything. You just can't fake being cool.

Author’s Note: Bambi Meets Godzilla

This was a follow-up, a few days later, after the torrential outpouring of alarm and pedantic disagreement from the Python community in response to my “A Little Anti-Anti Hype” post.

In this follow-up, I take a much nicer tone, and I provide some context that I think is pretty sobering. Watching a language die is a serious business. There are lessons here for everyone, I think.

This post mostly got the Python folks off my back, for which I remain eternally thankful.

Bambi Meets Godzilla

Gregory wrote:

“Steve, I'm just suggesting not putting fuel on a fire most Rubyists never intended to start in the first place.

“There is no need for a crusade or jihad here, on either side of the fence.

“This post just seems to be painfully biased with the expressed intent of bashing Python. That's just not cool.

“Pythonistas are not all tax collectors.”

* * * * *

Yes, yes, I know how my "anti-anti-hype" post must have felt. Given that I'm like 600 dog-years old, I sometimes forget the context is missing when I post. I'm going to try once more to explain where I was coming from in that post. If this attempt fails, it's really no big deal—I can certainly stick to technical blogs.

Incidentally, I'm going to talk about several languages, and eventually make my way back to Ruby at the end. Hang in there.

I think there are some issues we don't often talk about that have a direct impact on our lives as programmers. Like politics, they're tricky to talk about without arousing great, fiery passions. I'll be talking about some of them today. Why would I do that?

Well, first of all, let's get my agenda out in the open. I'm a programmer, and like you, I love building things. And ideally I'd like to build things in my favorite programming language, which happens to be Ruby—but only by a slim margin, because I love several other languages, too, including Python, Scheme, Lisp, the "D" language, C (but not C++), and various others. I like many languages a little bit, and there are only a few languages that I don't like very much.

As it happens, Python is my second-favorite language (well, tied for second with Scheme).

Yup, that's right. You heard me properly. I love Python and I think Guido is brilliant. Matz, too. Those guys are just amazing.

My agenda is really simple: I would like to write the majority of my code, at home and at work, in a great programming language.

Well, it's simple to state. It's not simple to do, for some complex and rather painful reasons that should be non-issues, but they're not. Let's look at them a little, and hopefully it will shed a little light on my "uncool" post.

Death of a beautiful language

I watched Smalltalk die.

I wasn't particularly invested in Smalltalk at the time, but I had done some programming in it. Smalltalk was (and still is) a superb programming language. And it died after I learned it.

There are some Smalltalk enthusiasts out there who will point to Squeak and other Smalltalk enclaves, and claim it's not dead. This is an important point. Chances are, you don't take Smalltalk very seriously. It's not on your radar. You don't think you'll ever need to learn it. So when I say "I watched Smalltalk die," to you it sounds like I'm talking about ancient history.

To many people, though, especially the ones who loved Smalltalk and whose very livelihoods depended on its commercial success, the failure of Smalltalk is a very painful subject. It's not boring ancient history. It still hurts them, deeply, and they even maintain hope that it may someday experience a glorious return to popularity.

This pain they feel, which you probably do not, is really close to the heart of our discussion. Hurt feelings are why these issues are so hard to talk about. It's very easy for you to say something insensitive about Smalltalk, especially if you don't know the language. You can take one look at it and say: "looks dumb," and you've just made someone mad.

In fact discussing it is extra hard because some people are just mad about Smalltalk in general. You can say anything at all about it, and simply bringing the subject up will have made them angry.

Regardless of whether Smalltalk is really dead, or merely a wounded bear in deep hibernation, I think it's clear that Smalltalk is not having a direct impact on the programming world today.

Smalltalk has plenty of indirect impact, of course—for instance, Ruby inherits a great deal from Smalltalk; all OO languages do to some extent, but Ruby more than many. But you can't walk into an ordinary computer bookseller and expect to find more than a couple of Smalltalk books. And if you want to get a job as a Smalltalk programmer, you will probably have to travel far, and you likely won't have much say in the kind of work you get to do. Smalltalk has retreated into a relatively small set of domains, at least for now.

What killed Smalltalk, anyway? I've read analyses and talked at length with some of the key players. The consensus seems to be that Java killed Smalltalk. And it did so rather decisively. Have you ever watched the short animated film Bambi Meets Godzilla? That's pretty much what it feels like now, although it actually happened over a period of several years.

Smalltalk wasn't terribly different from Java, really. It had an unusual all-in-one image model, where it acted like your OS, hosting the language, IDE, and application environment all in one binary. That is widely considered unpopular in retrospect, but the irony is that it's not really much different from the JVM. Smalltalk was commercial, and required user licenses; Java was commercial, but gave end-user licenses away for free. They were really pretty similar, and yet one wound up being far and away more popular than the other.

You can argue that Smalltalk would have failed fair and square, without Java, but I think most people agree that Java was a key contributor to Smalltalk's failure. And it wasn't a quiet thing, either. Millions of dollars were at stake. There were two large commercial Smalltalk vendors, and a bunch of unhappy about-to-be-ex-Smalltalk programmers, and hallways echoed with roars of protest at how Java, an "obviously" inferior language, had unfairly stolen a market that rightfully belonged to Smalltalk.

It all quieted down eventually, and to most of you, Smalltalk probably feels like a niche academic language that never had any real popularity.

I think Java coming along and smooshing Smalltalk was largely due to marketing. It's not the only factor, of course. Timing was a factor in various ways. Syntax and static typing were both factors, because Java deliberately went after disenchanted C++ programmers, which wasn't a bad strategy at all. And Java had some genuine innovations that helped, too.

But it was marketing that tied all those things together and helped Sun build a worldwide community of millions of Java programmers.

Java wasn't really offering anything that Smalltalk hadn't already been doing for years. (Where have we heard that argument before?)

Love and Money

It seems to me that there are two major contributors to language flame wars.

The first is that most programmers don't like to learn new languages. I don't know why, but true polyglots like me seem to be comparatively rare, maybe 5 to 10 percent of the programmer population. Most folks apparently prefer to master one language and stick with it for life.

The second is economics. Money motivates most decisions in the end. Companies need to make money, programmers need to get paid. You know all the old sayings: time is money, business is war, money (or love) makes the world go 'round, all's fair in love and war.

Programmers fall in love with their languages, so you've got two of the biggest forces in the world at play here: love and money, mixed with either fear or laziness. Is it any wonder people fight over languages?

Well… sort of. It seems like people should be able to use whatever language makes them most happy. But in practice, you can only use your favorite language if it happens to be C++, Java, Perl, or whatever language(s) your employer has decided are the "official" languages.

Most technical employers have a relatively small set of official languages, and you're forbidden from using any others. There are a few odd things about this situation.

One is that most companies couldn't care less about programming languages—all they want is to get their products built. It's always the engineers who impose the language restrictions. They're not restricting themselves, of course; it's a situation where engineers are governing other engineers by decree, within the same company. I've heard all the reasoning, and it still seems a little odd to me.

Another other odd thing is that most companies are using anywhere from 15 to 40 programming languages, but they only officially recognize two or three of them. They'll claim they're a Java/C++ shop, but have huge gobs of shell-script, awk, PHP, Perl, JavaScript, Tcl, emacs-lisp, vim-script, excel macros, pl*sql, and other languages threaded through their tools, databases, build systems, and so on. Maybe this is less true at Windows-based development shops.

And one last odd thing is that programmers often have to learn at least one new language when they arrive at a new job, but they never have any trouble. Programmers usually think learning a new language will be hard. When it's a job requirement, though, it happens amazingly fast. Programmers are generally pretty smart people.

You'd be amazed at how much resistance the "old guard" of a company will offer if you try to use your favorite language, and it's not on the approved-list. The "old guard" could even be 23-year-old CS grads that have just made a successful startup. "Old" here just means "first."

I've heard their arguments for 20 years. Don't use C++, it's slow (my first company). Don't use Java, it's slow (my second and third). Don't use Python, it's slow and has that whitespace thing. (All but my most recent.) Don't use Ruby, it's weird (90 percent of all companies). Language diversity is bad. What if someone has to debug your code in the middle of the night and they don't know that language? This fearmongering happens at every company, even those that don't work in the middle of the night. Don't use other languages. We don't hire for those skills. We don't trust those languages. We've invested in Fortran or Cobol or C++ or Java or whatever. No, no, no.

"No" always comes from engineers. You build something cool and popular, and your CEO will love you for it. She won't care if it's written in Intercal, as long as it works and your team can keep it working. So why do the engineers care so much? Who knows. I think it's often ego—they think of themselves as a great Java programmer, or an important Perl luminary, or a famous Python person, and they let their perceived self-image influence other peoples' technical decisions. Whatever the reason, it's a very real force in the workplace, one that plays a large role in the language wars we see on the internet.

Because, hey, if enough companies are already using your favorite language, then the problem still exists—but not for you!

Return of Godzilla vs. Bambi

I programmed in assembly language for five years at Geoworks; maybe that's why I love all languages a little. Then C/C++ for a while, and then a long seven-year stint with Java.

After two or three years with Java, I stumbled on Jython, a pretty nifty port of Python to the JVM. I'd been doing my scripting and auxiliary work in Perl. This was before it had ever occurred to me, still being pretty new, that one language could actually be suitable for most programming tasks. Java's not very good at many things Perl is good at, and vice-versa, so I had a big mixture of Java and Perl.

I talked in my “Anti-Anti-Hype” rant a little about Perl's marketing. It was world-class, and for a while I even thought I liked Perl. The marketing was so powerful that I simply took it for granted that I liked Perl.

Jython was a breath of fresh air, and I started wishing I could replace all my Java and Perl with it. Development had stopped on it, though, so it naturally led me to Python. For at least three years, Python was my favorite language, but I was heavily invested in the JVM, so I had to settle for Jython most of the time. It sure was fun, even though it was an old, relatively unsupported version of Python.

During those years, I wondered why Python wasn't as popular as Perl. It seemed like a much stronger language than Perl. That's just my opinion, of course, and there were certainly things I missed from Perl, so I'm not claiming that Python is the be-all, end-all of language design. But it seemed like the best thing out there.

Why wasn't it more popular? It seemed to be getting crushed by marketing forces—by fiery-eyed Perl zealots who went around and gained converts, one at a time. Perl was acting like a virus, and spreading rapidly, while Python sort of limped along, growing much more slowly. Richard Gabriel, of course, had already pointed out that C and Unix were virus-like in his famous short essay, The Rise of “Worse is Better.''

Let's be careful here: I believe Python has failed so far, and lots of people have jumped to say that Python "beat" Perl. Sure it did, in a number of quality-related ways. But the most immediately relevant metric to me is popular success in the commercial marketplace, because (remember my agenda?) I want to write my day-to-day code in a great language. I can't just tromp into most companies and announce I'm going to be writing in Python; they'd lynch me. So to this extent, Python has failed. And I really, really wish it hadn't. Because unlike when it happened with Smalltalk, I was invested this time around.

Because Python was my favorite language, I read all the Python books, and wrote a ton of Python code, and lurked on Python newsgroups, and basically soaked up the culture. And over a few years, I developed my own pet theory as to why Python has (so far) failed commercially.

Culture

I know you're gonna hate this part, but I'm going to talk a little about culture. Culture is very real. It matters every bit as much as love or money.

French waiters in Paris, on average, behave very, very differently from Japanese waiters in Tokyo. It is absolutely undeniable, and the difference is striking. I've spent plenty of time and eaten at plenty of restaurants in both places.

Once I was dining with a friend, and he whispered across the table: "I'd ask for some salt, but I think our waiter would kill us." Which country do you think was I in?

French waiters are not good or bad people, nor are Japanese waiters. They're just doing what's acceptable in their culture. But their cultures are very, very different. Waiting tables usually has a distinct subculture in any country, so I'm really just comparing the subcultures of French waiters in Paris and Japanese waiters in Tokyo.

I'm going to go out on a limb here, and say that I found French waiters in Paris almost terrifying. They huffed and puffed and stomped and glared and slammed the food down and were so comically over-the-top rude that it had to be an act, since my friends and I never did anything but politely sit down and point tentatively at menu items. (Author’s note: they were all pretty nice on my most recent visit.)

In contrast, I've seen Japanese waiters go to almost comical lengths to try to accommodate the requests of drunken people on business trips, to the point where I started feeling really un-proud to be an American. Japanese customer service practically defines world-class.

Okay, I hope we've established that cultures differ, and they have an enormous impact on your experience with people in that culture.

I think it should be obvious to you that programming languages have subcultures, too. The Perl culture is very different from the Python culture, and both are very different from Ruby culture.

The Python culture has a lot going for it. I was pretty immersed in it. But over time, as I wondered why Python wasn't becoming an overnight phenomenon, I started noticing some cultural behaviors in the Python community that I feel may be partly responsible. This is, of course, just my own opinion, endorsed by nobody.

I pointed out some of these behaviors in my “Anti-Anti-Hype” blog, and of course some people (Rubyists, Pythonistas, innocent bystanders) assumed I was Python-bashing, because they didn't watch Smalltalk die, and they didn't have the context I'm giving you now.

In reality, I'm actually just flat-out disappointed that Python never captured Perl-like marketplace success—and if you've been with me so far, you'll know this has real economic ramifications in terms of ability to write Python code in the workplace. And worse, it appears to be an avoidable problem: I think there are certain accepted practices in the Python world that are materially harming Python's adoption in the commercial marketplace.

I could spend a long time justifying each of my claims from “Anti-Anti-Hype,” but let's just focus on one of them: the tendency to label people as "incorrect." It's just an annoying habit, but one that can easily drive a potential new user away. It's a cultural habit, just like stomping and glaring and saying "psh!" loudly is a cultural habit among waiters in small restaurants in the Quartier Latin district of Paris.

The fact is, it doesn't take very much searching to find examples of this labeling. For instance, Recipe 1.7 of the Python Cookbook ends with a discussion around attributes versus items, and claims that many newcomers to Python "desire confusion" (by asking for uniform access to both), especially if they've come from a JavaScript background.

That seems kinda mean to me. If a programmer is genuinely confused, then fine, they're confused, although there's no need to harp on it. But if programmers are asking for a way to solve problems in a way they're used to, then labeling them as "confused" (a word that dictionaries varyingly define as baffled, perplexed, or unable to think with clarity or act intelligently) seems kinda harsh. Doesn't it?

Similarly, in Chapter 5 of the Jython Essentials book, during a discussion of Python's class system, it says: "Sometimes people erroneously see the need to explicitly specify the instance in the method argument list as evidence that object-oriented programming is somehow 'tacked on' to Python."

"Erroneously"? Gosh, this issue seems like a matter of pure opinion, not a fact that one can be either correct or incorrect about. How can an opinion be erroneous? Well, it's a cultural thing. If you have a culture of labeling differing opinions as incorrect, then an opinion can easily be considered erroneous.

There used to be an entry in the Python FAQ, which they removed a year or two ago, that said something along the lines of "Am I allowed to suggest changes to the language?" and the answer was a terse: "No." I can't remember the exact wording, but I found it pretty jarring, and it was there for years before getting cleaned up.

These little things add up, and they're ubiquitous. You may not notice them when reading Python discussions or documentation. I noticed because I was actively reading the write-ups and opinions of people who had tried Python and decided not to use it. Often as not, they said they felt rebuffed, or felt the community wasn't welcoming them, or they cited some other touchy-feely issue that didn't seem like it should have mattered at all. But there it was: a pattern had emerged.

Are all Python folks to blame for this? Of course not. Most Pythonistas people are really nice, warm, genuine, honest, smart people.

But a culture is a culture, and it has a big impact, like it or not. If the initial experience results in frequent enough culture-shock, it'll drive potential new users away.

Feel free to draw your own conclusions about why you use can Perl at most companies, but Python at relatively few. I've given you my take on it, and even if it's not the whole story, I honestly think it's a factor. There's more to marketing than glossy banners and shapeless cartoon mascots. Marketing can work all the way down to the level of individuals in one-on-one interactions.

In 10 years, I really don't want to be able to say: "I watched Python die." There's plenty of room for maybe five to eight really huge languages in the marketplace. I think Ruby's going to be up there soon, and frankly I'd more than welcome Python up there too.

In the meantime, though, I've been half-assuming that you can't change a culture—that once it's set, it's set. I hope I'm wrong. But my assumption is one of the biggest reasons that I finally switched to Ruby, just recently, over the summer, and committed to it for the foreseeable future. (I'm guessing 5 to 10 years.)

Ruby

The worldwide Ruby culture is the warmest and friendliest I've seen in my long history with programming languages. And Ruby is a sweet language. Other people seem to agree, and are taking steps to market it, which is getting them labeled as "hyper-enthusiasts" by the Sour Grapes camp. It appears to me that Ruby is doing what I wanted Python to do a few years ago, so I've finally learned Ruby and have switched most of my development over to it.

After all, both languages have a long way to go before they catch up with Java in terms of tools, IDEs, books, performance, threading stability, and tons of other stuff. I wanted to make a reasonably educated bet, and choose the language I think is going to be bigger, so it'll work well for me, and so I won't have to fight so hard to use it in my job.

It wasn't hard to learn Ruby. In fact after a few days with it, Ruby felt as comfortable as languages I'd been using for years. I've really been enjoying it. It has a few warts; all languages do. No biggie. It looks as if Matz is intent on fixing them in Rite.

I don't know if I like it more than Python and Scheme. I like it at least as much as those languages, certainly. But Ruby's my preferred language now because I can see the trajectory it's on, especially now with Rails, and I believe it's going to be the Next Big Thing—a Java-like phenomenon. So did Bruce Tate when he wrote "Beyond Java." So do James Duncan Davidson, Dave Thomas, Martin Fowler, and many other people who are a heck of a lot smarter than me. You'd think they're onto something big, wouldn't you? I do.

Java-like worldwide adoption really matters. Without that level of mass-market adoption, Ruby won't get the tools, stability, and CPAN-like library selection that it needs in order to compete with Java and Perl. It's a chicken-and-egg problem that all languages face, and Ruby stands a chance of succeeding where Smalltalk, Python, and other great languages have (to date) failed.

I see Rubyists worrying that Rails is stealing the show. Geez, folks, let it steal the show. Talk about a free ticket for Ruby success. Java Applets were a way to get Java in front of a million or so programmers, ultimately allowing the Java platform to succeed in all sorts of domains that it might never have seen without the initial "killer app" of Applets.

We live in a world where culture matters, economics matter, and marketing hype matters. They are very real forces that directly affect our quality of life as programmers. You ignore them at your peril, a lesson learned by so many almost-forgotten languages that were stomped by marketing hurricanes like Java and Perl.

I really wanted Python to succeed, and I still wish them the best, but I think they're ignoring marketing. I really want Ruby to succeed, so I get a bit miffed when I hear famous people like Bruce Eckel making uninformed generalizations about both Ruby and the folks who are working hard to make it successful. I think Pythonistas should be focusing on doing the same—working to make Python successful. I do think it will take a minor cultural adjustment on their part. And they need to start accepting hype as a natural part of the world we live in, a requirement for cutting through the noise. But I think they can do it.

With this context, does my “Anti-Anti-Hype” post start to make a bit more sense? Try re-reading it and see.

If not, well, you can't please everyone. I'm old enough not to mind.

Author’s Note: Math For Programmers

This is another post that makes the rounds on Hacker News a couple times a year. It’s an exciting post, in that it has a positive message for everyone—well, for everyone except educators, who whined a little about it.

It’s been a few years since I posted this essay, but I still think they should reform K-12 math education. The writing’s on the wall. Discrete math is where it’s at.

Incidentally, one of the best-ever comments on anything I’ve ever written came from Jamie Zawinski in response to this post. He apparently liked it and linked to it on his world-famous cat-pictures blog at www.jwz.org/blog. Shortly afterwards, he wrote:

Update: You know, I posted this link because I thought he had some interesting comments on math education and on how it relates to programming, and you all took it and turned it into some kind of dick-waving referendum on "well of course I do long division every day!" Good for you. Thank you for sharing. Thank you for like, not getting bogged down in the minutiae or anything.

I hate each and every one of you. STFU.

Today, over four years later, Jamie Zawinski still has the top search result for “dick-waving referendum.” I’ve sifted through my life’s accomplishments and can find nothing even approaching that level of awesomeness.

Math For Programmers

I've been working for the past 15 months on repairing my rusty math skills, ever since I read a biography of Johnny von Neumann. I've read a huge stack of math books, and I have an even bigger stack of unread math books. And it's starting to come together.

Let me tell you about it.

Conventional Wisdom Doesn't Add Up

First: programmers don't think they need to know math. I hear that so often; I hardly know anyone who disagrees. Even programmers who were math majors tell me they don't really use math all that much! They say it's better to know about design patterns, object-oriented methodologies, software tools, interface design, stuff like that.

And you know what? They're absolutely right. You can be a good, solid, professional programmer without knowing much math.

But hey, you don't really need to know how to program, either. Let's face it: there are a lot of professional programmers out there who realize they're not very good at it, and they still find ways to contribute.

If you're suddenly feeling out of your depth, and everyone appears to be running circles around you, what are your options? Well, you might discover you're good at project management, or people management, or UI design, or technical writing, or system administration, any number of other important things that "programmers" aren't necessarily any good at. You'll start filling those niches (because there's always more work to do), and as soon as you find something you're good at, you'll probably migrate towards doing it full-time.

In fact, I don't think you need to know anything, as long as you can stay alive somehow.

So they're right: you don't need to know math, and you can get by for your entire life just fine without it.

But a few things I've learned recently might surprise you:

1. Math is a lot easier to pick up after you know how to program. In fact, if you're a halfway decent programmer, you'll find it's almost a snap.

2. They teach math all wrong in school. Way, WAY wrong. If you teach yourself math the right way, you'll learn faster, remember it longer, and it'll be much more valuable to you as a programmer.

3. Knowing even a little of the right kinds of math can enable you to do write some pretty interesting programs that would otherwise be too hard. In other words, math is something you can pick up a little at a time, whenever you have free time.

4. Nobody knows all of math, not even the best mathematicians. The field is constantly expanding, as people invent new formalisms to solve their own problems. And with any given math problem, just like in programming, there's more than one way to do it. You can pick the one you like best.

5. Math is… ummm, please don't tell anyone I said this; I'll never get invited to another party as long as I live. But math, well… I'd better whisper this, so listen up: it's actually kinda fun.

The Math You Learned (And Forgot)

Here's the math I learned in school, as far as I can remember:

- Grade School: Numbers, Counting, Arithmetic, Pre-Algebra ("story problems")

- High School: Algebra, Geometry, Advanced Algebra, Trigonometry, Pre-Calculus (conics and limits)

- College: Differential and Integral Calculus, Differential Equations, Linear Algebra, Probability and Statistics, Discrete Math

How'd they come up with that particular list for high school, anyway? It's more or less the same courses in most U.S. high schools. I think it's very similar in other countries, too, except that their students have finished the list by the time they're nine years old. (Americans really kick butt at monster-truck competitions, though, so it's not a total loss.)

Algebra? Sure. No question. You need that. And a basic understanding of Cartesian geometry, too. Those are useful, and you can learn everything you need to know in a few months, give or take. But the rest of them? I think an introduction to the basics might be useful, but spending a whole semester or year on them seems ridiculous.

I'm guessing the list was designed to prepare students for science and engineering professions. The math courses they teach in high school don't help ready you for a career in programming, and the simple fact is that the number of programming jobs is rapidly outpacing the demand for all other engineering roles.

And even if you're planning on being a scientist or an engineer, I've found it's much easier to learn and appreciate geometry and trig after you understand what exactly math is—where it came from, where it's going, what it's for. No need to dive right into memorizing geometric proofs and trigonometric identities. But that's exactly what high schools have you do.

So the list is no good anymore. Schools are teaching us the wrong math, and they're teaching it the wrong way. It's no wonder programmers think they don't need any math: most of the math we learned isn't helping us.

The Math They Didn't Teach You

The math computer scientists use regularly, in real life, has very little overlap with the list above. For one thing, most of the math you learn in grade school and high school is continuous: that is, math on the real numbers. For computer scientists, 95 percent or more of the interesting math is discrete: i.e., math on the integers.

I'm going to talk in a future blog about some key differences between computer science, software engineering, programming, hacking, and other oft-confused disciplines. I got the basic framework for these (upcoming) insights in no small part from Richard Gabriel's Patterns Of Software, so if you absolutely can't wait, go read that. It's a good book. (Author’s note: I never did write that post. It turned out to be unnecessary; everyone has figured out that “Computer Science” is a misnomer.)

For now, though, don't let the term "computer scientist" worry you. It sounds intimidating, but math isn't the exclusive purview of computer scientists. You can learn it all by yourself as a closet hacker, and be just as good (or better) at it than they are. Your background as a programmer will help keep you focused on the practical side of things.

The math we use for modeling computational problems is, by and large, math on discrete integers. This is a generalization. If you're with me on today's blog, you'll be studying a little more math from now on than you were planning to before today, and you'll discover places where the generalization isn't true. But by then, a short time from now, you'll be confident enough to ignore all this and teach yourself math the way you want to learn it.

For programmers, the most useful branch of discrete math is probability theory. It's the first thing they should teach you after arithmetic, in grade school. What's probability theory, you ask? Why, it's counting. How many ways are there to make a Full House in poker? Or a Royal Flush? Whenever you think of a question that starts with "how many ways…" or "what are the odds…," it's a probability question. And as it happens (what are the odds?), it all just turns out to be "simple" counting. It starts with flipping a coin and goes from there. It's definitely the first thing they should teach you in grade school after you learn basic calculator usage. (Author’s note: I was informed by many people that I’ve conflated Combinatorics and Probability Theory, which to me are just two sides of the same coin toss, as it were. Goddamn pedants.)

I still have my discrete math textbook from college. It's a bit heavyweight for a third-grader (maybe), but it does cover a lot of the math we use in "everyday" computer science and computer engineering.

Oddly enough, my professor didn't tell me what it was for. Or I didn't hear. Or something. So I didn't pay very close attention: just enough to pass the course and forget this hateful topic forever, because I didn't think it had anything to do with programming. That happened in quite a few of my comp-sci courses in college, maybe as many as 25 percent of them. Poor me! I had to figure out what was important on my own, later, the hard way.

I think it would be nice if every math course spent a full week just introducing you to the subject, in the most fun way possible, so you know why the heck you're learning it. Heck, that's probably true for every course.

Aside from probability and discrete math, there are a few other branches of mathematics that are potentially quite useful to programmers, and they usually don't teach them in school, unless you're a math minor. This list includes:

- Statistics, some of which is covered in my discrete math book, but it's really a discipline of its own. A pretty important one, too, but hopefully it needs no introduction.

- Algebra and Linear Algebra (i.e., matrices). They should teach Linear Algebra immediately after algebra. It's pretty easy, and it's amazingly useful in all sorts of domains, especially machine learning.

- Mathematical Logic. I have a really cool totally unreadable book on the subject by Stephen Kleene, the inventor of the Kleene closure and, as far as I know, Kleenex. Don't read that one. I swear I've tried 20 times, and never made it past chapter 2. It's obviously important stuff, though. (Author’s note: some people apparently thought I was serious, and informed me that he did not, in fact, invent the Kleenex. Mathematicians, ho!)

- Information Theory and Kolmogorov Complexity. Weird, eh? I bet none of your high schools taught either of those. They're both pretty new. Information theory is (very roughly) about data compression, and Kolmogorov Complexity is (also roughly) about algorithmic complexity (i.e., how small you can you make it, how long will it take, how elegant can the program or data structure be, things like that). They're both fun, interesting, and useful.

There are others, of course, and some of the fields overlap. But it just goes to show: the math that you'll find useful is pretty different from the math your school thought would be useful.

What about calculus? Everyone teaches it, so it must be important, right?

Well, calculus is actually pretty easy. Before I learned it, it sounded like one of the hardest things in the universe, right up there with quantum mechanics. Quantum mechanics is still beyond me, but calculus is nothing. After I realized programmers can learn math quickly, I picked up my Calculus textbook and got through the entire thing in about a month, reading for an hour an evening.

Calculus is all about continuums—rates of change, areas under curves, volumes of solids. Useful stuff, but the exact details involve a lot of memorization and a lot of tedium that you don't normally need as a programmer. It's better to know the overall concepts and techniques, and go look up the details when you need them.

Geometry, trigonometry, differentiation, integration, conic sections, differential equations, and their multidimensional and multivariate versions—these all have important applications. It's just that you don't need to know them right this second. So it probably wasn't a great idea to make you spend years and years doing proofs and exercises with them, was it? If you're going to spend that much time studying math, it ought to be on topics that will remain relevant to you for life.

The Right Way To Learn Math

The right way to learn math is breadth-first, not depth-first. You need to survey the space, learn the names of things, figure out what's what.

To put this in perspective, think about long division. Raise your hand if you can do long division on paper, right now. Hands? Anyone? I didn't think so.

I went back and looked at the long-division algorithm they teach in grade school, and damn if it isn't annoyingly complicated. It's deterministic, sure, but you never have to do it by hand, because it's easier to find a calculator, even if you're stuck on a desert island without electricity. You'll still have a calculator in your watch, or your dental filling, or something.

Why do they even teach it to you? Why do we feel vaguely guilty if we can't remember how to do it? It's not as if we need to know it anymore. And besides, if your life were on the line, you know you could perform long division of any arbitrarily large numbers. Imagine you're imprisoned in some slimy third-world dungeon, and the dictator there won't let you out until you've computed 219308862/103503391. How would you do it? Well, easy. You'd start subtracting the denominator from the numerator, keeping a counter, until you couldn't subtract it anymore, and that'd be the remainder. If pressed, you could figure out a way to continue using repeated subtraction to estimate the remainder as decimal number (in this case, 0.1185678219, or so my Emacs M-x calculator tells me. Close enough!)

You could figure it out because you know that division is just repeated subtraction. The intuitive notion of division is deeply ingrained now.

The right way to learn math is to ignore the actual algorithms and proofs, for the most part, and to start by learning a little bit about all the techniques: their names, what they're useful for, approximately how they're computed, how long they've been around, (sometimes) who invented them, what their limitations are, and what they're related to. Think of it as a Liberal Arts degree in mathematics.

Why? Because the first step to applying mathematics is problem identification. If you have a problem to solve, and you have no idea where to start, it could take you a long time to figure it out. But if you know it's a differentiation problem, or a convex optimization problem, or a boolean logic problem, then you at least know where to start looking for the solution.

There are lots and lots of mathematical techniques and entire sub-disciplines out there now. If you don't know what combinatorics is, not even the first clue, then you're not very likely to be able to recognize problems for which the solution is found in combinatorics, are you?

But that's actually great news, because it's easier to read about the field and learn the names of everything than it is to learn the actual algorithms and methods for modeling and computing the results. In school they teach you the Chain Rule, and you can memorize the formula and apply it on exams, but how many students really know what it "means"? So they're not going to be able to know to apply the formula when they run across a chain-rule problem in the wild. Ironically, it's easier to know what it is than to memorize and apply the formula, even though they don’t teach it that way. The chain rule is just how to take the derivative of "chained" functions—meaning, function x() calls function g(), and you want the derivative of x(g()). Well, programmers know all about functions; we use them every day, so it's much easier to imagine the problem now than it was back in school.

Which is why I think they're teaching math wrong. They're doing it wrong in several ways. They're focusing on specializations that aren't proving empirically to be useful to most high-school graduates, and they're teaching those specializations backwards. You should learn how to count, and how to program, before you learn how to take derivatives and perform integration.

I think the best way to start learning math is to spend 15 to 30 minutes a day surfing in Wikipedia. It's filled with articles about thousands of little branches of mathematics. You start with pretty much any article that seems interesting (e.g., String theory, say, or the Fourier transform, or Tensors, anything that strikes your fancy.) Start reading. If there's something you don't understand, click the link and read about it. Do this recursively until you get bored or tired.

Doing this will give you amazing perspective on mathematics, after a few months. You'll start seeing patterns—for instance, it seems that just about every branch of mathematics that involves a single variable has a more complicated multivariate version, and the multivariate version is almost always represented by matrices of linear equations. At least for applied math. So Linear Algebra will gradually bump its way up your list, until you feel compelled to learn how it actually works, and you'll download a PDF or buy a book, and you'll figure out enough to make you happy for a while.

With the Wikipedia approach, you'll also quickly find your way to the Foundations of Mathematics, the Rome to which all math roads lead. Math is almost always about formalizing our "common sense" about some domain, so that we can deduce and/or prove new things about that domain. Metamathematics is the fascinating study of what the limits are on math itself: the intrinsic capabilities of our formal models, proofs, axiomatic systems, and representations of rules, information, and computation.

One great thing that soon falls by the wayside is notation. Mathematical notation is the biggest turnoff to outsiders. Even if you're familiar with summations, integrals, polynomials, exponents, etc., if you see a thick nest of them your inclination is probably to skip right over that sucker as one atomic operation.

However, by surveying math, trying to figure out what problems people have been trying to solve (and which of these might actually prove useful to you someday), you'll start seeing patterns in the notation, and it'll stop being so alien-looking. For instance, a summation sign (capital-sigma) or product sign (capital-pi) will look scary at first, even if you know the basics. But if you're a programmer, you'll soon realize it's just a loop: one that sums values, one that multiplies them. Integration is just a summation over a continuous section of a curve, so that won't stay scary for very long, either.

Once you're comfortable with the many branches of math, and the many different forms of notation, you're well on your way to knowing a lot of useful math. Because it won't be scary anymore, and next time you see a math problem, it'll jump right out at you. "Hey," you'll think, "I recognize that. That's a multiplication sign!"

And then you should pull out the calculator. It might be a very fancy calculator such as R, Matlab, Mathematica, or a even C library for support vector machines. But almost all useful math is heavily automatable, so you might as well get some automated servants to help you with it.

When Are Exercises Useful?

After a year of doing part-time hobbyist catch-up math, you're going to be able to do a lot more math in your head, even if you never touch a pencil to a paper. For instance, you'll see polynomials all the time, so eventually you'll pick up on the arithmetic of polynomials by osmosis. Same with logarithms, roots, transcendentals, and other fundamental mathematical representations that appear nearly everywhere.

I'm still getting a feel for how many exercises I want to work through by hand. I'm finding that I like to be able to follow explanations (proofs) using a kind of "plausibility test"—for instance, if I see someone dividing two polynomials, I kinda know what form the result should take, and if their result looks more or less right, then I'll take their word for it. But if I see the explanation doing something that I've never heard of, or that seems wrong or impossible, then I'll dig in some more.

That's a lot like reading programming-language source code, isn't it? You don't need to hand-simulate the entire program state as you read someone's code; if you know what approximate shape the computation will take, you can simply check that their result makes sense. E.g., if the result should be a list, and they're returning a scalar, maybe you should dig in a little more. But normally you can scan source code almost at the speed you'd read English text (sometimes just as fast), and you'll feel confident that you understand the overall shape and that you'll probably spot any truly egregious errors.

I think that's how mathematically-inclined people (mathematicians and hobbyists) read math papers, or any old papers containing a lot of math. They do the same sort of sanity checks you'd do when reading code, but no more, unless they're intent on shooting the author down.

With that said, I still occasionally do math exercises. If something comes up again and again (like algebra and linear algebra), then I'll start doing some exercises to make sure I really understand it.

But I'd stress this: don't let exercises put you off the math. If an exercise (or even a particular article or chapter) is starting to bore you, move on. Jump around as much as you need to. Let your intuition guide you. You'll learn much, much faster doing it that way, and your confidence will grow almost every day.

How Will This Help Me?

Well, it might not—not right away. Certainly it will improve your logical reasoning ability; it's a bit like doing exercise at the gym, and your overall mental fitness will get better if you're pushing yourself a little every day.

For me, I've noticed that a few domains I've always been interested in (including artificial intelligence, machine learning, natural language processing, and pattern recognition) use a lot of math. And as I've dug in more deeply, I've found that the math they use is no more difficult than the sum total of the math I learned in high school; it's just different math, for the most part. It's not harder. And learning it is enabling me to code (or use in my own code) neural networks, genetic algorithms, bayesian classifiers, clustering algorithms, image matching, and other nifty things that will result in cool applications I can show off to my friends.

And I've gradually gotten to the point where I no longer break out in a cold sweat when someone presents me with an article containing math notation: n-choose-k, differentials, matrices, determinants, infinite series, etc. The notation is actually there to make it easier, but (like programming-language syntax) notation is always a bit tricky and daunting on first contact. Nowadays I can follow it better, and it no longer makes me feel like a plebeian when I don't know it. Because I know I can figure it out.

And that's a good thing.

And I'll keep getting better at this. I have lots of years left, and lots of books, and articles. Sometimes I'll spend a whole weekend reading a math book, and sometimes I'll go for weeks without thinking about it even once. But like any hobby, if you simply trust that it will be interesting, and that it'll get easier with time, you can apply it as often or as little as you like and still get value out of it.

Math every day. What a great idea that turned out to be!

Author’s Note: Rich Programmer Food

I have a secret recruiting weapon. When I’m fishing for strong “generalist” software engineer candidates, I just look for “compilers.” There are lots of red-flag keywords and phrases out there which—on average—indicate weakness. But there are only a handful of keywords you can look for on resumes that on average indicate strength, by which I mean “probability of success in traditional interviews.” And “compilers” is one of the hottest of the bunch.

No, I’m not going to tell you all of them. I can’t give all my secrets away! But this is a big one. Compiler people tend to do unusually well in regular interviews, even when they’re not asked about compilers. And, conversely, people who know nothing about compilers often have gaping holes in their computer science education and in their overall understanding of system architecture. Such people may pass their interviews, but they’re just lucky.

I think I first noticed this phenomenon—the more or less linear correspondence between knowledge of compiler construction and interview performance—back at Amazon. But looking back, I can see that even at prior companies, the most badass engineers were the ones who worked on the compilers and interpreters.

So by the time I got to Google, it was just my secret weapon. I’d bring in amazing candidates, and people would wonder how I found them, and, well, now you know.

If you’re a recruiter or a hiring manager, you just got your money’s worth from this book.

If you’re a programmer, and you don’t know this stuff yet, I hope this convinces you to learn.

Rich Programmer Food

“The Olive Garden: it's where poor people go to eat rich people food.” -Dave Yegge

This is another one of those blog topics I've been sitting on for way too long, trying to find a polite way of saying something fundamentally impolite. I don't see a way to do it. So: you stand a good chance of being offended by this blog entry. (Hey, just don't say I didn't warn ya.)

I've turned off blog comments, incidentally, because clever evil people have figured out how to beat captchas using non-algorithmic approaches, and I don't have the bandwidth to police spam myself. Sorry.

I don't want to give you a heart attack, so I'm going to give you the gentle-yet-insistent executive summary right up front. If you can make it through my executive summary without a significant increase in heart rate, then you're probably okay. Otherwise, you might consider drinking heavily before reading this, just like people did in the old movies when they needed their leg sawed off. That's what I'm doing, in any case (drinking, that is, not sawing my leg off).

Gentle, yet insistent executive summary: If you don't know how compilers work, then you don't know how computers work. If you're not 100 percent sure whether you know how compilers work, then you don't know how they work.

You have to know you know, you know?

In fact, Compiler Construction is, in my own humble and probably embarrassingly wrong opinion, the second most important CS class you can take in an undergraduate computer science program.

Because every deep-dive I've attempted on this topic over the past year or so has failed utterly at convincing me after I sobered up, I'm going to stage this production as a, erm, stage production, with N glorious, er, parts, separated by intermissions. So without further ado…

Actually, that sounds like way too much work. So I'll just rant. That's what you paid good money to hear anyway, right? I promise to make so much fun of other people that when I make fun of you, you'll hardly notice.

Cots and Beards

I took compilers in school. Yup. Sure did. From Professor David Notkin at the University of Washington, circa late 1991 or thereabouts.

Guess what grade I got? I got a zero. As in, 0.0. That was my final grade, on my transcript. That's what happens at the University of Washington when you get an “Incomplete” and don't take the necessary corrective actions (which I've never figured out, by the way.) After some time elapses, it turns into a zero.

You can get an “Incomplete” in various different legitimate ways, including my way, which was to be an ill-considered beef-witted mooncalf who takes the course past the drop-date and then decides not to finish it because he doesn't feel like it. I earned that “Incomplete,” I tell you.

I took Compilers again a few years later. I was in college for a long time, because I got hired on as a full-time employee by Geoworks about a year before I graduated (among other reasons), and it wound up extending my graduation for several years.

Don't do that, by the way. It's really hard to finish when you're working full-time. Get your degree, then go to work. All the more so if you're a Ph.D. candidate within any reach of finishing. You don't want to be just another ABD for the rest of your life. Even if you're not sad, per se, we'll be sad for you.

I got a decent grade in Compilers the second time around. I actually understood compilers at a reasonably superficial level the first time, and not too badly the second time. What I failed to grasp for many more years, and I'm telling you this to save you that pain, is why compilers actually matter in the first place.

Here's what I thought when I took it back in 1991. See if it sounds familiar. I thought: a compiler is a tool that takes my program, after whining about it a lot, and turns it into computer-speak. If you want to write programs, then a compiler is just one of those things you need. You need a computer, a keyboard, an account maybe, a compiler, an editor, optionally a debugger, and you're good to go. You know how to Program. You're a Programmer. Now you just need to learn APIs and stuff.

Whenever I gave even a moment's thought to whether I needed to learn compilers, I'd think: I would need to know how compilers work in one of two scenarios. The first scenario is that I go work at Microsoft and somehow wind up in the Visual C++ group. Then I'd need to know how compilers work. The second scenario is that the urge suddenly comes upon me to grow a long beard and stop showering and make a pilgrimage to MIT where I beg Richard Stallman to let me live in a cot in some hallway and work on GCC with him like some sort of Jesuit vagabond.

Both scenarios seemed pretty unlikely to me at the time, although if push came to shove, a cot and beard didn't seem all that bad compared to working at Microsoft.

By the way, my brother Dave was at a party once, long ago, that had more than its fair share of Microsoft people, and apparently there was some windbag there bragging loudly (this is a party, mind you) that he had 15 of the world's best compiler writers working for him in the Visual C++ group. I told Dave: "Wow, I didn't realize Richard Stallman worked at Microsoft," and Dave was bummed that he hadn't thought of that particular riposte at the time. So it goes.

The sad part about that story is that I've found myself occasionally wanting to brag that I work with some of the best compiler writers in the world at Google. Please, I beg you: if you ever find me at a party bragging about the compiler writers I work with, have pity on us all and shoot me dead on the spot. Hell, bash me over the head with a lamp if you have to.

Anyway, now you know what I thought of compilers in 1991. Why I even took the class is beyond me. But I didn't finish. And the second time around—which I only did because I felt bad about the first time around: not from the zero, but from having let David Notkin down—I only took the time to understand the material well enough to finish the course with a decent grade.

I was by no means atypical. If you're a CS student and you love compilers (which, anecdotally, often means you're in the top 5 percent of computer science students in your class worldwide), then I salute you. I bet I'm way better at Nethack than you are! The reality is that most programmers are just like I was, and I can't really fault 'em for that.

Before I leave this sordid story forever, I feel obliged to point out that it's partly academia's fault. Except for type systems research, which is being treated with approximately the same scholarly caution and restraint as Arthur's Grail Quest, compilers have been out of favor in academia for a long time. So schools don't do a good job at marketing compilers, and giving them due credit as a critical topic in their own right. It's a sad fact that most schools don't require you to take compilers in order to graduate with a Computer Science degree.

Sigh.

How Would You Solve…

You're a programmer, right? Okay, I'll propose some programming situations for you, and you tell me how you'd solve them.

Situation 1: You're doing a bunch of Java programming, and your company has explicit and non-negotiable guidelines as to how to format your Java code, down to every last imaginable detail. How do you configure your editor to auto-format your code according to the style guide?

Situation 2: Your company does a lot of Ajax stuff, and your JavaScript code base is growing almost as fast as your other code. You decide to start using jsdoc, a javadoc pseudo-clone for JavaScript, to document your functions in a way that permits automated doc extraction. You discover that jsdoc is a miserable sod of a Perl script that seg faults on about 50 percent of your code base, and—bear with me here—you've vowed never to write another line of Perl, because, well, it's Perl. Pick your favorite reason. How do you write your own jsdoc extractor, bearing in mind that it will need to do at least a cursory parse of the JavaScript code itself?

Situation 3: Your company has a massive C++ code base, the result of many years of hard work by dozens, if not hundreds, of engineers. You discover that the code needs to be refactored in a nontrivial way, e.g., to upgrade from 32-bit to 64-bit, or to change the way you do your database transactions, or (God help you) because you're upgrading your C++ compiler, and the syntax and semantics have all changed again. You're tasked with fixing it. What do you do?

Situation 4: Someone at your company writes a bitchin' new web-based code review tool. Everyone switches to it. You realize, after using it for a while, that you miss having it syntax-highlight the source code for you. You don't have much time, but you might be able to afford a week or so, part-time, to make it happen. How do you do it? (Let's say your company uses five to eight languages for 99 percent of their code.)

Situation 5: An unexpected and slightly bizarre new requirement arises on your current project: you need to be able to use a new kind of hardware router. Maybe all your Web 2.0 stuff is screwing up your border routers or network bandwidth monitors, who knows. All you know is the sysops and network engineers are telling you that you need to talk to these new routers directly. The routers have IP addresses, a telnet interface, and a proprietary command language. You send commands, and they send responses. Each command has its own syntax for its arguments, and you need to parse the responses (which have no documented format, but you can reverse-engineer it) to look for certain patterns, in order to set them in the right state for your wacky uploads or downloads. What tool do you use?

Situation 6: Your company's projects are starting to slip. The engineers are all smart, and they are all using the latest and greatest state-of-the-art Agile Object-Oriented Software Engineering Principles and programming languages. They are utterly blameless. However, for some reason your code base is getting so complex that project estimates are going wildly awry. Simple tasks seem to take forever. The engineers begin talking about a redesign. This is the Nth such redesign they have gone through in the past five years, but this is going to be the big one that fixes everything. What color slips of paper do you give them? Woah, ahem, sorry, I mean how do you ensure their success this time around?

Situation 7: You have a small, lightweight startup company filled with cool young people with long blue-tinted hair and nose rings and tongue rivets and hip black clothes and iPhones and whatever the hell else young people have these days. You use Ruby on Rails for your site, and it scales just fine for your number of visitors. (You've never bothered to measure whether your number of visitors is a function of your site's latency, because it's never occurred to you to wonder.) You read about the latest horrible godawful Rails security vulnerability, under which users can make arbitrary SEC filings on behalf of your company by sending properly formatted GET requests to your public site. You download the new version and read the unit test code to figure out what the actual vulnerability is, since they didn't say, and you determine that you need to make a set of nontrivial code changes to remove a particular (and mysteriously non-greppable) idiom from your code base, replacing it by mechanical transformation to a different idiom. How do you do it?

Situation 8: Some drunken blogger presents you with seven weird situations and asks you to speculate about what they have in common. Do you already know the answer?

Here are the answers. What, you thought these were rhetorical?

Situation 1: You lobby your company to change the style guide to match whatever Eclipse does by default.

Situation 2: You post to the jsdoc mailing list and ask if anyone else has had this problem. Several people say they have, and the issue pretty much dies right then and there.

Situation 3: You quit. Duh. You knew that was the answer before you reached the first comma.

Situation 4: Tough it out. Colors are for weenies. Or maybe you wire up GNU Source Highlight, which covers languages all the way from Fortran to Ada, and you live with the broken highlighting it provides.

Situation 5: Perl. It's a Swiss Army Knife. You can use it to sidestep this problem with honor, by disemboweling yourself.

Situation 6: Pink.

Situation 7: Fix it by hand. Hell, you only have about 10,000 lines of code for your whole site. It's Rails, fer cryin' out loud. This was a trick question.

Situation 8: Yes. You skim until the end of the blog, just to find out what the first-most-important CS class is. Stevey's well known for shaggy-dog jokes like this.

And there you have it. You're now equipped to deal with just about every programming situation you could come across. So you obviously don't need to know compilers.

How Compilers Work

Here are some real-life answers from real-life candidates, with real-life Ph.D.s in Computer Science, when asked how compilers work.

Real Candidate #1: "Oh! They, ah, um, scan your program one line at a time and convert each line to assembly language."

Real Candidate #2: "Compilers check errors in your program and, ah, tell you if you had bad syntax. That's all I remember."

Real Candidate #3: "I… (three minute pause)… I don't know."

Real Candidate #4: "They preprocess your program and convert #DEFINE statements into code, and then, um, emit machine code."

That's pretty much all the detail you'll ever get out of 75 percent of all interview candidates, because, hey, they don't want to work in a hallway at MIT. Can you blame them?

Only about 3 to 5 percent of all interview candidates (and that's being optimistic) can tell you any details about how a compiler works. The rest will do some handwaving about lex and yacc and code generation, maybe.

I told you your heart rate would go up. Didn't I?

Take a deep breath.

Why Compilers Matter, Part 1

The first reason Compiler Construction is such an important CS course is that it brings together, in a very concrete way, almost everything you learned before you took the course.

You can't fully understand how compilers work without knowing machine architecture, because compilers emit machine code. It's more than just instructions; compilers need to understand how the underlying machine actually operates in order to translate your source code efficiently.

Incidentally, "machines" are just about anything that can do computations. Perl is a machine. Your OS is a machine. Emacs is a machine. If you could prove your washing machine is Turing complete, then you could write a compiler that executes C code on it.

But you knew that already.

You can't understand how modern compilers work without knowing how Operating Systems work, because no self-respecting machine these days runs without an operating system. The OS interface forms part of the target machine. Sure, you can find people working on five- to ten-year mainframe projects that ultimately run no faster than a PC from Costco, and they may dispense with the operating system due to time constraints, plus the fact that they have a worldwide market of one customer. But for most of us, the OS is part of the machine.

You won't understand how compilers work unless you've taken a theory of computation course. The theory of computation reads like part one of chapter 1 of a compilers book. You need all of it.

You'll have difficulty keeping the phases (and even the inputs and outputs) of a compiler straight in your head unless you've taken a programming languages course. You have to know what the capabilities of programming languages are, or at least have an inkling, before you can write a program that implements them. And unless you know more than one language well, it won't make much sense to write a program in language A that converts language B to language C.

You're actually surrounded by compilation problems. You run into them almost every day. The seven scenarios I outlined above are the tip of the iceberg. (The eighth one is the rest of the iceberg, but no skimming!)

Compilers take a stream of symbols, figure out their structure according to some domain-specific predefined rules, and transform them into another symbol stream.

Sounds pretty general, doesn't it? Well, yeah.

Could an image be considered a symbol stream? Sure. Stream each row of pixels. Each pixel is a number. A number is a symbol. You can transform images with compilers.

Could English be considered a symbol stream? Sure. The rules are pretty damn complex, but yes, natural language processing (at least the deterministic kind that doesn't work and has been supplanted by stochastic methods) can be considered a fancy kind of compilation.

What about ordinary code? I mean, we don't all deal with image processing, or natural language processing. What about the rest of us? We just write code, so do compilers really matter?

Well, do you ever, ever need to write code that deals with your own code base? What if you need to write a syntax highlighter? What if your programming language adds some new features and your editor doesn't support them yet? Do you just sit around and wait for "someone" to fix your editor? What if it takes years? Doesn't it seem like you, as the perfectly good programmer that you are, ought to be able to fix it faster than that?

Do you ever need to process your code base looking for certain idioms? Do you ever need to write your own doc extractor?

Have you ever worked on code bases that have grown inexplicably huge, despite all your best efforts to make them modular and object-oriented? Of course you have. What's the solution?

You either learn compilers and start writing your own DSLs, or your get yourself a better language.

I recommend NBL, by the way. It's my personal favorite: the local maximum in a tensor-field of evil, the highest ground in the vicinity of Hell itself. I'm not going to tell you what NBL is, yet, though. Patience! I'm only half done with my Emacs-mode for it.

If you don't take compilers…

One reason many programmers don't take compilers is that they've heard it's really, really hard. It's often the "capstone" course of a CS program (OS often being the other one), which means it's a sort of "optional rite of passage" that makes you a Real Programmer and puts hair on your chest, regardless of gender or chest-hair preference.

If you're trying to plan out a schedule that gets you to graduation before the money runs out, and hopefully with a GPA that doesn't cause prospective employers to summon the guard dogs on you, then when you hear the phrase "optional rite of passage," who can blame you if you look for alternatives?

I'm not saying other CS courses aren't important, incidentally. Operating Systems, Machine Learning, Distributed Computing, and Algorithm Design are all arguably just as important as Compiler Construction. Except that you can take them all and still not know how computers work, which to me means that Compilers really needs to be a mandatory 300-level course. But it has so many prerequisites that you can't realistically make that happen at most schools.

Designing an effective undergrad CS degree is hard. It's no wonder so many ivy-league schools have more or less given up and turned into Java Certification shops.

If you're a conscientious CS student, you'll at least take OS and AI. You may come out without knowing exactly how compilers work, which is unfortunate, but there will be many problem domains in which you can deliver at least as much value as all the other people just like you. That's something to feel good about, or at least as good as everyone else feels at any rate.

Go team.

Most programmers these days, sadly, just want the degree. They don't care what they learn. They want a degree so they can get a job so they can pay the bills.

Most programmers gravitate towards a set of courses that can best be described as the Olive Garden of computer science: the places where dumb programmers go to learn smart programmer stuff.

I hesitate to name these courses explicitly. I wouldn't be agile enough to dodge the game of graphic bloodshed aimed at me by animated, project-managing, object-oriented engineers using Java and Web 2.0 technologies to roast me via user interfaces designed rationally through teamwork and modern software methodologies. I'd become a case study in the ethics of software and its impact on our culture.

But you can probably imagine what some of the courses are.

If you don't take compilers then you run the risk of forever being on the programmer B-list: the kind of eager young architect who becomes a saturnine old architect who spends a career building large systems and being damned proud of it.

Large Systems Suck

This rule is 100 percent transitive. If you build one, you suck.

Compiler Camps

It turns out that many compiler "experts" don't know compilers all that well, because compilers can logically be thought of as three separate phases—so separate, in fact, that they constitute entirely different and mostly non-overlapping research domains.

The first big phase of the compilation pipeline is parsing. You need to take your input and turn it into a tree. So you go through preprocessing, lexical analysis (aka tokenization), and then syntax analysis and IR generation. Lexical analysis is usually done with regexps. Syntax analysis is usually done with grammars. You can use recursive descent (most common), or a parser generator (common for smaller languages), or with fancier algorithms that are correspondingly slower to execute. But the output of this pipeline stage is usually a parse tree of some sort.

You can get a hell of a lot farther as a professional programmer just by knowing that much. Even if you have no idea how the rest of the compilation works, you can make practical use of tools or algorithms that produce a parse tree. In fact, parsing alone can help you solve situations one through four above.

If you don't know how parsing works, you'll do it badly with regular expressions, or if you don't know those, then with hand-rolled state machines that are thousands of lines of incomprehensible code that doesn't actually work.

Really.

In fact I used to ask candidates, as a standard interview question, how they'd find phone numbers in a tree of HTML files, and many of them (up to 30 percent) chose to write 2,500-line C++ programs as their answer.

At some point, candidates started telling me they'd read that one in my blog, which was pretty weird, all things considered. Now I don't ask it anymore. (Author’s note: I tried it in a phone screen five years later, and the guy immediately said: “Saw that one in your blog!” It was such a great question, too. Sigh.)

I ask variants of it occasionally, and it still gets them: you either recognize it as an easy problem, or you get out the swiss army knife and start looking for a second to behead yourself before the pain causes you to dishonor your family.

C++ does that surprisingly often.

The next big phase is Type Checking. This is a group of zealous academics (and their groupies and/or grad students) who believe that they can write programs that are smart enough to figure out what your program is trying to do, and tell you when you're wrong. They don't think of themselves as AI people, oddly enough, because AI has (wisely) moved beyond deterministic approaches.

The Typies have figured out more or less the practical limit of what they can check deterministically, and they have declared that this is the boundary of computation itself, beyond the borders of which you are crossing the outskirts of civilization into kill-or-be-killed territory, also occasionally known as The Awful Place Where People Make Money With Software.

You should hear them when they're drunk at raves.

A good friend of mine with a Ph.D. in languages told me recently that it's "very painful" to experience the realization that all those years of slaving over beautiful mathematical purity have more or less zero bearing on the real world.

The problem—well, one problem—is the underlying premise, which is apparently that without the Hindley-Milner type system, or failing that, some crap-ass type system like Java's, you will never be able to write working code; it'll collapse under its own weight: a vast, typeless trap for the unwary adventurer.

They don't get out much, apparently.

Another problem is that they believe any type "error," no matter how insignificant it might be to the operation of your personal program at this particular moment, should be treated as a news item worthy of the Wall Street Journal front page. Everyone should throw down their ploughshares and stop working until it's fixed. The concept of a type "warning" never enters the discussion.

Remember when fuzzy logic came along? Oh, oh, wait—remember when von Neumann and Stan Ulam introduced the Monte Carlo method? Oh, right, I keep forgetting: you were born in nineteen-ninety something, and you're nineteen, and I'm ninety-something.

Well, someday they will realize that strict determinism has always, always failed, in every dimensionality-cursed domain to which it's ever been applied, and it's always replaced by probabilistic methods.

Call it "optional static types," as an embryonic version of the glorious future. NBL, anyone?

The third camp, who tends to be the most isolated, is the code generation camp. Code generation is pretty straightforward, assuming you know enough recursion to realize your grandparents weren't Adam and Eve. So I'm really talking about Optimization, which is the art of generating code that is just barely correct enough that most of your customers won't notice problems. Wait, sorry, that's Amazonization. Optimization is the art of producing correct code that is equivalent to the naive, expensive code written by your presumably naive, expensive programmers.

I'd call compiler optimization an endless chasm of eternal darkness, except that it's pretty fun. So it's an endless chasm of fun eternal darkness, I guess. But you can take it to extremes you'd never guess were possible, and it's a fertile, open research field, and when they "finish," they'll be in the same place the Type Checking camp wants to be, namely AI experts.

By which I mean Machine Learning, since the term "AI" smacks of not just determinism, but also a distinct lack of VC funding.

In any case, the three camps don't really mingle much, and all of them have a valid claim at calling themselves "compiler experts" at parties.

The Dark Side of Compilers

One of the reasons it took me so long to write this ridiculous blog entry is that I wanted to go write a compiler for myself before I spouted off about them.

Done!

Well, sort of. Actually, "not done" would be more accurate, since that, as I've found, is the steady state for compilers everywhere.

Without giving any details away, as that would be premature, I took a stab at writing an interpreter for a useful language, using another useful language, with the output being useful bytecode for a useful platform.

It was fun. It went pretty fast. I learned a lot, even though I'd taken compilers twice in school 15 years ago, and even though I've gradually taught myself about compilers and programming languages over the past five years or so.

I still learned a lot just by doing it.

Unfortunately, writing a compiler creates a living thing. I didn't realize this going into it. I wasn't asking for a baby. It was a complete surprise to me, after 20-odd years of industry experience, that even writing a simple interpreter would produce a lifetime of work.

Go figure.

I credit the phrase "a lifetime of work" to Bob Jervis, a friend of mine who happens to be the original author of Turbo C (with which I myself learned to program), and a damn good, even world-class compiler writer.

He gave a tech talk recently (Google does that a LOT) in which he pointed out that even just the set of features the audience had asked for was a lifetime of work.

This phrasing resonated deeply with me. It was similar to my realization about 18 months back that I only have a small finite number of 5-year projects left, and I have to start choosing them very carefully. After writing my own "production interpreter," I realized that the work remaining was unbounded.

I mean it. Unbounded.

So from one perspective, I suppose I should just release what I've got and start marketing it, so other people will jump on board and start helping out. On the other hand, I started this particular side-project not to create a lifetime of work for myself (far from it), but to make sure I knew enough about compilers to be able to rant semi-intelligently about them, after a few glasses of wine, to a quarter million readers.

So I'd at least better finish the byte compiler first.

I'll get there. It'll be neat. I've only described this crazy little side project to a handful of people, and they reacted pretty uniformly by yelling "WTF????" You know, the kind of shout you'd yell out if you discovered the most sane person you knew in the entire world trying to stuff a lit stick of dynamite into their mouth.

That's compilers for ya. You can hardly attempt one without trying to change the world in the process.

That's why you need to learn how they work. That's why you, yes you personally, need to write one.

It's not as hard as you think, except for the fact that it will turn into a lifetime of work. It's OK. You can walk away from it, if you want to. You probably won't want to. You may be forced to, due to time constraints, but you'll still be a far better programmer for the effort.

You'll be able to fix that dang syntax highlighting.

You'll be able to write that doc extractor.

You'll be able to fix the broken indentation in Eclipse.

You won't have to wait for your tools to catch up.

You might even stop bragging about how smart your tools are, how amazing it is that they can understand your code—which, if I may say so, isn't something I'd go broadcasting quite so loudly, but maybe it's just me.

You'll be able to jump in and help fix all those problems with your favorite language. Don't even try to tell me your favorite language doesn't have problems.

You'll be able to vote with confidence against the tired majority when some of the smartest people in the world (like, oh, say, James Gosling and Guy Steele) try to introduce non-broken closures and real extensibility to the Java community. Those poor Java Community schmucks. I pity them all. Really I do.

Heck, you might even start eating rich programmer food. Writing compilers is only the beginning; I never claimed it was the end of the road. You'll finally be able to move past your little service APIs and JavaScript widgets, and start helping to write the program that cures cancer, or all viruses worldwide, or old age and dying. Or even (I'm really going out on a limb here) the delusion of Static Typing as a deterministic panacea.

If nothing else, you'll finally really learn whatever programming language you're writing a compiler for. There's no other way. Sorry!

And with that, I suppose I should wrap up. I'm heading to Foo Camp in the morning, and I have no idea what to expect, but I have a pretty good guess that there won't be much discussion of compilers, except hopefully from GVR vis a vis Python 3000. That might be cool.

If you don't know compilers, don't sweat it. I still think you're a good programmer. But it's good to have stretch goals!

But What's The Most Important CS Course?

Typing 101. Duh.

Hie thee hence.

Become a Hyperink reader. Get a special surprise.

Like the book? Support our author and leave a comment!

IV.

On Google

Author’s Note: Get that job at Google

This article—like most that I’ve written—was born of frustration. But this time it was frustration that we were turning great people away. We did it at Geoworks; we did it at Amazon; we do it at Google. Companies know they’re turning great people away. It’s a subject for another day—a long one, and I promise great interest and controversy when I finally tackle it—but we can focus here on one small aspect, which is that candidates often come unprepared.

This is pretty awful, if you think about it. A smart candidate comes in perfectly capable of getting hired and doing a great job, and flubs one or more interviews due to simple lack of preparation.

I’ve done a LOT of recruiting and hiring stuff over the years—well over a thousand interviews, maybe over fifteen hundred, plus equally many phone screens. We’d each sometimes do ten or fifteen a day on college trips at Amazon.

After that much interviewing over 20-odd years, you start noticing patterns. And a big one is that candidates walk through the door completely unprepared for the interview. They haven’t practiced; they haven’t warmed up; they haven’t refreshed their memory on stuff they learned in school; they are easily thrown by the strangeness of the interviewing environment. And they fail. And it sucks.

Interviewing is hard. It’s hard to be a good interviewer, and even harder to be a good interviewee.

We really want people to be prepared and functioning at their very best when they arrive onsite. Google Kirkland has taken to coaching people for an hour, post phone-screen, before their interviews. We teach group classes on interview tips.

To that end, even though this was a totally unsolicited and unapproved post when I first wrote it, Google recruiters around the world have been pointing thousands of candidates here as a preparation step for their phone screens and interviews. It’s that useful.

And I’ve collected feedback from hundreds of people about how useful they found it. Many of them didn’t even get job offers, but they still took the time to mail me and tell me how useful it was, and how much worse they’d have done without the tips here.

Google Recruiting initially anguished over whether to redact the section on “The Interview Anti-Loop.” As I’m sure you can imagine, they want to paint a pretty picture of Google, and this is calling out an obvious blemish in our process. But they ultimately decided that it’s in their best interest to include it as-is, because it’s useful. Many candidates have told us that the Anti-Loop section is what gave them the nerve to go through with the application and interviews.

This post was, unsurprisingly, somewhat controversial. A lot of programmers were butthurt over the fact that Google and similar companies ask questions to which they don’t know the answers. Many self-taught programmers who are no doubt doing outstanding jobs were dealt a serious blow to their self-esteem by this post. One commenter accused me of being “full of himself” for disclosing the fact that other Google interviewers ask questions about graph theory.

But I’ve learned that you can’t talk about recruiting without damaging some egos. It just comes with the territory. And my post tips are fairly accurate for most tech companies, including Microsoft, Google, Amazon, Apple, and Facebook. In fact it’s all pretty basic stuff—mostly junior-level and low senior-level undergrad computer science. It’s not fancy.

I’ve been planning a sequel to this post for a long time, and it’s almost ready. (Really almost ready—as in, weeks away.) But it’s mostly just testimonials. The bottom line is that a little preparation can make a staggering difference in your performance on the Big Day.

Get that job at Google

I've been meaning to write up some tips on interviewing at Google for a good long time now. I keep putting it off, though, because it's going to make you mad. Probably. For some statistical definition of "you," it's very likely to upset you.

Why? Because… well, here, I wrote a little ditty about it:

I didn't realize this was such a typical reaction back when I first started writing about interviewing, way back at other companies. Boy-o-howdy did I find out in a hurry.

See, it goes like this:

Me: blah blah blah, I like asking question X in interviews, blah blah blah…

You: Question X? Oh man, I haven't heard about X since college! I've never needed it for my job! He asks that in interviews? But that means someone out there thinks it's important to know, and, and… I don't know it! If they detect my ignorance, not only will I be summarily fired for incompetence without so much as a thank-you, I will also be unemployable by people who ask question X! If people listen to Stevey, that will be everyone! I will become homeless and destitute! For not knowing something I've never needed before! This is horrible! I would attack X itself, except that I do not want to pick up a book and figure enough out about it to discredit it. Clearly I must yell a lot about how stupid Stevey is so that nobody will listen to him!

Me: So in conclusion, blah blah… huh? Did you say "fired"? "Destitute?" What are you talking about?

You: Aaaaaaauuuggh!!! *stab* *stab* *stab*

Me: That's it. I'm never talking about interviewing again.

It doesn't matter what X is, either. It's arbitrary. I could say: "I really enjoy asking the candidate (their name) in interviews," and people would still freak out, on account of insecurity about either interviewing in general or their knowledge of their own name, hopefully the former.

But then, time passes, and interview candidates come and go, and we always wind up saying: "Gosh, we sure wish that obviously smart person had prepared a little better for his or her interviews. Is there any way we can help future candidates out with some tips?"

And then nobody actually does anything, because we're all afraid of getting stabbed violently by People Who Don't Know X.

I considered giving out a set of tips in which I actually use variable names like X, rather than real subjects, but decided that in the resultant vacuum, everyone would get upset. Otherwise that approach seemed pretty good, as long as I published under a pseudonym.

In the end, people really need the tips, regardless of how many feelings get hurt along the way. So rather than skirt around the issues, I'm going to give you a few mandatory substitutions for X along with a fair amount of general interview-prep information.

Caveats and Disclaimers

This blog is not endorsed by Google. Google doesn't know I'm publishing these tips. It's just between you and me, okay? Don't tell them I prepped you. Just go kick ass on your interviews, and we'll be square.

I'm only talking about general software engineering positions, and interviews for those positions.

These tips are actually generic; there's nothing specific to Google vs. any other software company. I could have been writing these tips about my first software job 20 years ago. That implies that these tips are also timeless, at least for the span of our careers.

These tips obviously won't get you a job on their own. My hope is that by following them you will perform your very best during the interviews.

Oh, and um, why Google?

Oho! Why Google, you ask? Well let's just have that dialog right up front, shall we?

You: Should I work at Google? Is it all they say it is, and more? Will I be serenely happy there? Should I apply immediately?

Me: Yes.

You: To which ques… wait, what do you mean by "Yes?" I didn't even say who I am!

Me: Dude, the answer is Yes. (You may be a woman, but I'm still calling you Dude.)

You: But… but… I am paralyzed by inertia! And I feel a certain comfort level at my current company, or at least I have become relatively inured to the discomfort. I know people here and nobody at Google! I would have to learn Google's build system and technology and stuff! I have no credibility, no reputation there—I would have to start over virtually from scratch! I waited too long, there's no upside! I'm afraid!

Me: DUDE. The answer is YES already, okay? It's an invariant. Everyone else who came to Google was in the exact same position as you are, modulo a handful of famous people with beards that put Gandalf's to shame, but they're a very tiny minority. Everyone who applied had the same reasons for not applying as you do. And everyone here says: "GOSH, I SURE AM HAPPY I CAME HERE!" So just apply already. But prep first.

You: But what if I get a mistrial? I might be smart and qualified, but for some random reason I may do poorly in the interviews and not get an offer! That would be a huge blow to my ego! I would rather pass up the opportunity altogether than have a chance of failure!

Me: Yeah, that's at least partly true. Heck, I kinda didn't make it in on my first attempt, but I begged like a street dog until they gave me a second round of interviews. I caught them in a weak moment. And the second time around, I prepared, and did much better.

The thing is, Google has a well-known false negative rate, which means we sometimes turn away qualified people, because that's considered better than sometimes hiring unqualified people. This is actually an industry-wide thing, but the dial gets turned differently at different companies. At Google the false-negative rate is pretty high. I don't know what it is, but I do know a lot of smart, qualified people who've not made it through our interviews. It's a bummer.

But the really important takeaway is this: if you don't get an offer, you may still be qualified to work here. So it needn't be a blow to your ego at all!

As far as anyone I know can tell, false negatives are completely random, and are unrelated to your skills or qualifications. They can happen from a variety of factors, including but not limited to:

1. You're having an off-day.

2. One or more of your interviewers is having an off-day.

3. There were communication issues invisible to you and/or one or more of the interviewers.

4. You got unlucky and got an Interview Anti-Loop.

Oh no, not the Interview Anti-Loop!

Yes, I'm afraid you have to worry about this.

What is it, you ask? Well, back when I was at Amazon, we did (and they undoubtedly still do) a lot of soul-searching about this exact problem. We eventually concluded that every single employee E at Amazon has at least one "Interview Anti-Loop": a set of other employees S who would not hire E. The root cause is important for you to understand when you're going into interviews, so I'll tell you a little about what I've found over the years.

First, you can't tell interviewers what's important. Not at any company. Not unless they're specifically asking you for advice. You have a very narrow window of perhaps one year after an engineer graduates from college to inculcate them in the art of interviewing, after which the window closes and they believe they are a "good interviewer" and they don't need to change their questions, their question styles, their interviewing style, or their feedback style, ever again.

It's a problem. But I've had my hand bitten enough times that I just don't try anymore.

Second problem: every "experienced" interviewer has a set of pet subjects and possibly specific questions that he or she feels is an accurate gauge of a candidate's abilities. The question sets for any two interviewers can be widely different and even entirely non-overlapping.

A classic example found everywhere is: Interviewer A always asks about C++ trivia, file systems, network protocols, and discrete math. Interviewer B always asks about Java trivia, design patterns, unit testing, web frameworks, and software project management. For any given candidate with both A and B on the interview loop, A and B are likely to give very different votes. A and B would probably not even hire each other, given a chance, but they both happened to go through interviewer C, who asked them both about data structures, unix utilities, and processes versus threads, and A and B both happened to squeak by.

That's almost always what happens when you get an offer from a tech company. You just happened to squeak by. Because of the inherently flawed nature of the interviewing process, it's highly likely that someone on the loop will be unimpressed with you, even if you are Alan Turing. Especially if you're Alan Turing, in fact, since it means you obviously don't know C++.

The bottom line is, if you go to an interview at any software company, you should plan for the contingency that you might get genuinely unlucky, and wind up with one or more people from your Interview Anti-Loop on your interview loop. If this happens, you will struggle, then be told that you were not a fit at this time, and then you will feel bad. Just as long as you don't feel meta-bad, everything is okay. You should feel good that you feel bad after this happens, because hey, it means you're human.

And then you should wait 6-12 months and re-apply. That's pretty much the best solution we (or anyone else I know of) could come up with for the false-negative problem. We wipe the slate clean and start over again. There are lots of people here who got in on their second or third attempt, and they're kicking butt.

You can too.

Okay, I feel better about potentially not getting hired

Good! So let's get on to those tips, then.

If you've been following along very closely, you'll have realized that I'm interviewer D. Meaning that my personal set of pet questions and topics is just my own, and it's no better or worse than anyone else's. So I can't tell you what it is, no matter how much I'd like to, because I'll offend interviewers A through X who have slightly different working sets.

Instead, I want to prep you for some general topics that I believe are shared by the majority of tech interviewers at Google-like companies. Roughly speaking, this means the company builds a lot of their own software and does a lot of distributed computing. There are other tech-company footprints, the opposite end of the spectrum being companies that outsource everything to consultants and try to use as much third-party software as possible. My tips will be useful only to the extent that the company resembles Google.

So you might as well make it Google, eh?

First, let's talk about non-technical prep.

The Warm-Up

Nobody goes into a boxing match cold. Lesson: you should bring your boxing gloves to the interview. No, wait, sorry, I mean: warm up beforehand!

How do you warm up? Basically there is short-term and long-term warming up, and you should do both.

Long-term warming up means: study and practice for a week or two before the interview. You want your mind to be in the general "mode" of problem solving on whiteboards. If you can do it on a whiteboard, every other medium (laptop, shared network document, whatever) is a cakewalk. So plan for the whiteboard.

Short-term warming up means: get lots of rest the night before, and then do intense, fast-paced warm-ups the morning of the interview.

The two best long-term warm-ups I know of are:

1) Study a data-structures and algorithms book. Why? Because it is the most likely to help you beef up on problem identification. Many interviewers are happy when you understand the broad class of question they're asking without explanation. For instance, if they ask you about coloring U.S. states in different colors, you get major bonus points if you recognize it as a graph-coloring problem, even if you don't actually remember exactly how graph-coloring works.

And if you do remember how it works, then you can probably whip through the answer pretty quickly. So your best bet, interview-prep wise, is to practice the art of recognizing that certain problem classes are best solved with certain algorithms and data structures.

My absolute favorite for this kind of interview preparation is Steven Skiena's The Algorithm Design Manual. More than any other book it helped me understand just how astonishingly commonplace (and important) graph problems are—they should be part of every working programmer's toolkit. The book also covers basic data structures and sorting algorithms, which is a nice bonus. But the gold mine is the second half of the book, which is a sort of encyclopedia of one-pagers on zillions of useful problems and various ways to solve them, without too much detail. Almost every one-pager has a simple picture, making it easy to remember. This is a great way to learn how to identify hundreds of problem types.

Other interviewers I know recommend Introduction to Algorithms. It's a true classic and an invaluable resource, but it will probably take you more than two weeks to get through it. But if you want to come into your interviews prepped, then consider deferring your application until you've made your way through that book.

2) Have a friend interview you. The friend should ask you a random interview question, and you should go write it on the board. You should keep going until it is complete, no matter how tired or lazy you feel. Do this as much as you can possibly tolerate.

I didn't do these two types of preparation before my first Google interview, and I was absolutely shocked at how bad at whiteboard coding I had become since I had last interviewed seven years prior. It's hard! And I also had forgotten a bunch of algorithms and data structures that I used to know, or at least had heard of.

Going through these exercises for a week prepped me mightily for my second round of Google interviews, and I did way, way better. It made all the difference.

As for short-term preparation, all you can really do is make sure you are as alert and warmed up as possible. Don't go in cold. Solve a few problems and read through your study books. Drink some coffee: it actually helps you think faster, believe it or not. Make sure you spend at least an hour practicing immediately before you walk into the interview. Treat it like a sports game or a music recital, or heck, an exam: if you go in warmed up you'll give your best performance.

Mental Prep

So! You're a hotshot programmer with a long list of accomplishments. Time to forget about all that and focus on interview survival.

You should go in humble, open-minded, and focused.

If you come across as arrogant, then people will question whether they want to work with you. The best way to appear arrogant is to question the validity of the interviewer's question—it really ticks them off, as I pointed out earlier on. Remember how I said you can't tell an interviewer how to interview? Well, that's especially true if you're a candidate.

So don't ask: "Gosh, are algorithms really all that important? Do you ever need to do that kind of thing in real life? I've never had to do that kind of stuff." You'll just get rejected, so don't say that kind of thing. Treat every question as legitimate, even if you are frustrated that you don't know the answer.

Feel free to ask for help or hints if you're stuck. Some interviewers take points off for that, but occasionally it will get you past some hurdle and give you a good performance on what would have otherwise been a horrible stony half-hour silence.

Don't say "choo choo choo" when you're "thinking."

Don't try to change the subject and answer a different question. Don't try to divert the interviewer from asking you a question by telling war stories. Don't try to bluff your interviewer. You should focus on each problem they're giving you and make your best effort to answer it fully.

Some interviewers will not ask you to write code, but they will expect you to start writing code on the whiteboard at some point during your answer. They will give you hints but won't necessarily come right out and say: "I want you to write some code on the board now." If in doubt, you should ask them if they would like to see code.

Interviewers have vastly different expectations about code. I personally don't care about syntax (unless you write something that could obviously never work in any programming language, at which point I will dive in and verify that you are not, in fact, a circus clown and that it was an honest mistake). But some interviewers are really picky about syntax, and some will even silently mark you down for missing a semicolon or a curly brace, without telling you. I think of these interviewers as—well, it's a technical term that rhymes with "bass soles," but they think of themselves as brilliant technical evaluators, and there's no way to tell them otherwise.

So ask. Ask if they care about syntax, and if they do, try to get it right. Look over your code carefully from different angles and distances. Pretend it's someone else's code and you're tasked with finding bugs in it. You'd be amazed at what you can miss when you're standing two feet from a whiteboard with an interviewer staring at your shoulder blades.

It's okay (and highly encouraged) to ask a few clarifying questions, and occasionally verify with the interviewer that you're on the track they want you to be on. Some interviewers will mark you down if you just jump up and start coding, even if you get the code right. They'll say you didn't think carefully first, and you're one of those "let's not do any design" type cowboys. So even if you think you know the answer to the problem, ask some questions and talk about the approach you'll take a little before diving in.

On the flip side, don't take too long before actually solving the problem, or some interviewers will give you a delay-of-game penalty. Try to move (and write) quickly, since often interviewers want to get through more than one question during the interview, and if you solve the first one too slowly then they'll be out of time. They'll mark you down because they couldn't get a full picture of your skills. The benefit of the doubt is rarely given in interviewing.

One last non-technical tip: bring your own whiteboard dry-erase markers. They sell pencil-thin ones at office supply stores, whereas most companies (including Google) tend to stock the fat kind. The thin ones turn your whiteboard from a 480i standard-definition tube into a 58-inch 1080p HD plasma screen. You need all the help you can get, and free whiteboard space is a real blessing.

You should also practice whiteboard space-management skills, such as not starting on the right and coding down into the lower-right corner in Teeny Unreadable Font. Your interviewer will not be impressed. Amusingly, although it always irks me when people do this, I did it during my interviews, too. Just be aware of it!

Oh, and don't let the marker dry out while you're standing there waving it. I'm tellin' ya: you want minimal distractions during the interview, and that one is surprisingly common.

Okay, that should be good for non-tech tips. On to X, for some value of X! Don't stab me!

Tech Prep Tips

The best tip is: go get a computer science degree. The more computer science you have, the better. You don't have to have a CS degree, but it helps. It doesn't have to be an advanced degree, but that helps too.

However, you're probably thinking of applying to Google a little sooner than two to eight years from now, so here are some shorter-term tips for you.

Algorithm Complexity: You need to know Big-O. It's a must. If you struggle with basic big-O complexity analysis, then you are almost guaranteed not to get hired. It's, like, one chapter in the beginning of one theory of computation book, so just go read it. You can do it.

Sorting: Know how to sort. Don't do bubble-sort. You should know the details of at least one n*log(n) sorting algorithm, preferably two (say, quicksort and merge sort). Merge sort can be highly useful in situations where quicksort is impractical, so take a look at it.

For God's sake, don't try sorting a linked list during the interview.

Hashtables: Hashtables are arguably the single most important data structure known to mankind. You absolutely have to know how they work. Again, it's like one chapter in one data structures book, so just go read about them. You should be able to implement one using only arrays in your favorite language, in about the space of one interview.

Trees: You should know about trees. I'm tellin' ya: this is basic stuff, and it's embarrassing to bring it up, but some of you out there don't know basic tree construction, traversal and manipulation algorithms. You should be familiar with binary trees, n-ary trees, and trie-trees at the very very least. Trees are probably the best source of practice problems for your long-term warmup exercises.

You should be familiar with at least one flavor of balanced binary tree, whether it's a red/black tree, a splay tree or an AVL tree. You should actually know how it's implemented.

You should know about tree traversal algorithms, BFS and DFS, and know the difference between inorder, postorder, and preorder.

You might not use trees much day-to-day, but if so, it's because you're avoiding tree problems. You won't need to do that anymore once you know how they work. Study up!

Graphs: Graphs are, like, really really important. More than you think. Even if you already think they're important, it's probably more than you think.

There are three basic ways to represent a graph in memory (objects and pointers, matrix, and adjacency list), and you should familiarize yourself with each representation and its pros and cons.

You should know the basic graph traversal algorithms: breadth-first search and depth-first search. You should know their computational complexity, their tradeoffs, and how to implement them in real code.

You should try to study up on fancier algorithms, such as Dijkstra and A*, if you get a chance. They're really great for just about anything, from game programming to distributed computing to you name it. You should know them.

Whenever someone gives you a problem, think graphs. They are the most fundamental and flexible way of representing any kind of a relationship, so it's about a 50-50 shot that any interesting design problem has a graph involved in it. Make absolutely sure you can't think of a way to solve it using graphs before moving on to other solution types. This tip is important!

Other data structures

You should study up on as many other data structures and algorithms as you can fit in that big noggin of yours. You should especially know about the most famous classes of NP-complete problems, such as traveling salesman and the knapsack problem, and be able to recognize them when an interviewer asks you them in disguise.

You should find out what NP-complete means.

Basically, hit that data structures book hard, and try to retain as much of it as you can, and you can't go wrong.

Math: Some interviewers ask basic discrete math questions. This is more prevalent at Google than at other places I've been, and I consider it a Good Thing, even though I'm not particularly good at discrete math. We're surrounded by counting problems, probability problems, and other Discrete Math 101 situations, and those innumerate among us blithely hack around them without knowing what we're doing.

Don't get mad if the interviewer asks math questions. Do your best. Your best will be a heck of a lot better if you spend some time before the interview refreshing your memory on (or teaching yourself) the essentials of combinatorics and probability. You should be familiar with n-choose-k problems and their ilk—the more the better.

I know, I know, you're short on time. But this tip can really help make the difference between a "we're not sure" and a "let's hire her." And it's actually not all that bad—discrete math doesn't use much of the high-school math you studied and forgot. It starts back with elementary-school math and builds up from there, so you can probably pick up what you need for interviews in a couple of days of intense study.

Operating Systems: This is just a plug, from me, for you to know about processes, threads, and concurrency issues. A lot of interviewers ask about that stuff, and it's pretty fundamental, so you should know it. Know about locks and mutexes and semaphores and monitors and how they work. Know about deadlock and livelock and how to avoid them. Know what resources a process needs, and a thread needs, and how context switching works, and how it's initiated by the operating system and underlying hardware. Know a little about scheduling. The world is rapidly moving towards multi-core, and you'll be a dinosaur in a real hurry if you don't understand the fundamentals of "modern" (which is to say, "kinda broken") concurrency constructs.

The best, most practical book I've ever personally read on the subject is Doug Lea's Concurrent Programming in Java. It got me the most bang per page. There are obviously lots of other books on concurrency. I'd avoid the academic ones and focus on the practical stuff, since it's most likely to get asked in interviews.

Coding: You should know at least one programming language really well, and it should preferably be C++ or Java. C# is okay too, since it's pretty similar to Java. You will be expected to write some code in at least some of your interviews. You will be expected to know a fair amount of detail about your favorite programming language.

Other Stuff

Because of the rules I outlined above, it's still possible that you'll get Interviewer A, and none of the stuff you've studied from these tips will be directly useful (except being warmed up). If so, just do your best. Worst case, you can always come back in 6-12 months, right? Might seem like a long time, but I assure you it will go by in a flash.

The stuff I've covered is actually mostly red flags: stuff that really worries people if you don't know it. The discrete math is potentially optional, but somewhat risky if you don't know the first thing about it. Everything else I've mentioned you should know cold, and then you'll at least be prepped for the baseline interview level. It could be a lot harder than that, depending on the interviewer, or it could be easy.

It just depends on how lucky you are. Are you feeling lucky? Then give it a try!

Send me your resume

I'll probably batch up any resume submissions people send me and submit them weekly. In the meantime, study up! You have a lot of warming up to do. Real-world work makes you rusty.

I hope this was helpful. Let the flames begin, etc. Yawn.

Author’s Note: Good Agile, Bad Agile

This was, at the time, the most viral any of my posts had ever gone. “Agile” was creeping into Google, and it wasn’t fucking welcome there. So I killed it, damn-near single-handedly.

I put weeks and weeks of thought into this post. I asked advice about wording and messaging from many people, including (at the time) Agile enthusiasts. And I put my heart into it.

Before my post, it was almost to the point where Agile had become “mainstream,” meaning you were in the minority if you disliked it. Which mean that it was in danger of becoming non-optional. That mess would have taken years to clean up.

I saw it coming and did a preemptive strike, right at its heart. I made some enemies. I don’t mind. Because I don’t give a shit if YOU want to use Agile, but you’d damned well better not try to make ME use it.

And I succeeded. *We* succeeded. There were some other bloggers who jumped on the bandwagon with me right around the same time—in many cases, arriving there via convergent evolution. The time was ripe for a counter-revolution. And we succeeded in getting one message through; namely: IT IS OKAY TO SAY NO TO AGILE. I wouldn’t object to that being on my headstone. In fact, I’d even leave off the “TO AGILE” part. It’s always okay to say no.

This was a big win for software engineering in our generation. You have no idea. Today, Agile has largely been debunked, and the consultants have more or less abandoned it in favor of “Lean” and other desperate attempts to revive the money machine. But Agile has been in decline since my post. I dealt it a mortal blow. It takes time for something with that much money and that much arbitrary bullshit fashion marketing sunk into it to really die. But it’s inevitable now. Just a matter of time.

The pen really is mightier, you know. If you are passionate about something—write!

Good Agile, Bad Agile

“Scrums are the most dangerous phase in rugby, since a collapse or improper engage can lead to a front row player damaging or even breaking his neck.” -Wikipedia

When I was growing up, cholesterol used to be bad for you. It was easy to remember. Fat, bad. Cholesterol, bad. Salt, bad. Everything, bad. Nowadays, though, they differentiate between "good" cholesterol and "bad" cholesterol, as if we're supposed to be able to distinguish them somehow. And it was weird when they switched it up on us, because it was as if the FDA had suddenly issued a press release announcing that there are, in fact, two kinds of rat poison: Good Rat Poison and Bad Rat Poison, and you should eat a lot of the Good kind, and none of the Bad kind, and definitely not mix them up or anything.

Up until maybe a year ago, I had a pretty one-dimensional view of so-called "Agile" programming, namely that it's an idiotic fad-diet of a marketing scam making the rounds as yet another technological virus implanting itself in naive programmers who've never read No Silver Bullet, the kinds of programmers who buy extended warranties and self-help books and believe their bosses genuinely care about them as people, the kinds of programmers who attend conferences to make friends and who don't know how to avoid eye contact with leaflet-waving fanatics in airports and who believe writing shit on index cards will suddenly make software development easier.

You know. Chumps. That's the word I'm looking for. My bad-cholesterol view was that Agile Methodologies are for chumps.

But I've had a lot of opportunity to observe various flavors of Agile-ism in action lately, and I now think I was only about 90 percent right. It turns out there's a good kind of Agile, although it's taken me a long time to be able to see it clearly amidst all the hype and kowtowing and moaning feverishly about scrums and whatnot. I have a pretty clear picture of it now.

And you can attend my seminar on it for the low, low price of $499.95! Hahaha, chump!

No, just kidding. You'll only find seminars about the Bad kind of Agile. And if in the future you ever find me touring around as an Agile Consultant, charging audiences to hear my deep wisdom and insight about Agile Development, you have my permission to cut my balls off. If I say I was just kidding, say I told you I'd say that. If I then say I'm Tyler Durden and I order you not to cut my balls off, say I definitely said I was going to say that, and then you cut 'em right off.

I'll just go right ahead and tell you about the Good Kind, free of charge.

It's kinda hard to talk about Good Agile and Bad Agile in isolation, so I might talk about them together. But I'll be sure to label the Good kind with a happy rat, and the Bad kind with a sad dead rat, so you'll always know the difference.

The Bad Kind

Back in Ye Olden Dayes, most companies approached software development as follows:

- Hire a bunch of engineers, then hire more.

- Dream up a project.

- Set a date for when they want it launched.

- Put some engineers on it.

- Whip them until they're either dead or it's launched. Or both.

- Throw a cheap-ass pathetic little party, maybe. This step is optional.

- Then start over.

Thank goodness that doesn't happen at your company, eh now? Whew!

Interestingly, this is also exactly how non-technical companies (like, say, Chrysler) handled software development. Except they didn't hire the engineers. Instead, they contracted with software consultants, and they'd hand the consultants two-year project specs, and demanded the consultants finish everything on time plus all the crap the customer threw in and/or changed after signing the contract. And then it'd all fall apart, and the contractors wouldn't get paid, and everyone would be really miffed.

So some of the consultants began to think: "Hey, if these companies insist on acting like infants, then we should treat them like infants!" And so they did. When a company said, "we want features A through Z," the consultants would get these big index cards and write "A" on the first one, "B" on the second one, etc., along with time estimates, and then post them on their wall. Then when the customer wanted to add something, the consultant could point at the wall and say: "Okay, boy. Which one of these cards do you want to replace, BOY?"

Is it any wonder Chrysler canceled the project?

So the consultants, now having lost their primary customer, were at a bar one day, and one of them (named L. Ron Hubbard) said: "This nickel-a-line-of-code gig is lame. You know where the real money is at? You start your own religion." And that's how both Extreme Programming and Scientology were born.

Well, people pretty quickly demonstrated that XP was a load of crap. Take Pair Programming, for instance. It's one of the more spectacular failures of XP. None of the Agileytes likes to talk about it much, but let's face it: nobody does it. The rationale was something like: "Well if ONE programmer sitting at a terminal is good, then TEN must be better, because MORE is ALWAYS better! But most terminals can only comfortably fit TWO programmers, so we'll call it PAIR programming!"

You have to cut them a little slack; they'd been dealing with the corporate equivalent of pre-schoolers for years, and that really messes with a person.

But the thing is, viruses are really hard to kill, especially the meme kind. After everyone had gotten all worked up about this whole Agile thing (and sure, everyone wants to be more productive), there was a lot of face to be lost by admitting failure. So some other kinds of Agile "Methodologies" sprang up, and they all claimed that even though all the other ones were busted, their method worked!

I mean, go look at some of their sites. Tell me that's not an infomercial. C'mon, just try. It's embarrassing even to look at the thing. (Author’s note: Six years later. *Still* embarrassing.)

Yeah. Well, they make money hand over fist, because of P.T. Barnum's Law, just like Scientology does. Can't really fault 'em. Some people are just dying to be parted with their cash. And their dignity.

The rest of us have all known that Agile Methodologies are stupid, by application of any of the following well-known laws of marketing:

- Anything that calls itself a "Methodology" is stupid, on general principle.

- Anything that requires "evangelists" and offers seminars, exists solely for the purpose of making money.

- Anything that never mentions any competition or alternatives is dubiously self-serving.

- Anything that does diagrams with hand-wavy math is stupid, on general principle.

And by "stupid," I mean it's "incredibly brilliant marketing targeted at stupid people."

In any case, the consultants kept going with their road shows and glossy pamphlets. Initially, I'm sure they went after corporations; they were looking to sign flexible contracts that allowed them to deliver "whatever" in "two weeks" on a recurring basis until the client went bankrupt. But I'm equally sure they couldn't find many clients dumb enough to sign such a contract.

That's when the consultants decided to take their road show to YOU. Why not take it inside the companies and sell it there, to the developers? There are plenty of companies who use the whip-cycle of development I outlined above, so presumably some of the middle managers and tech leads would be amenable to hearing about how there's this low-cost way out of their hellish existence.

And that, friends, was exactly, precisely the point at which they went from "harmless buffoons" to "potentially dangerous," because before they were just bilking fat companies too stupid to develop their own software, but now the manager down the hall from me might get infected. And most places don't have a very good quarantine mechanism for this rather awkward situation: i.e., an otherwise smart manager has become "ill," and is waving XP books and index cards and spouting stuff about how much more productive his team is on account of all this newfound extra bureaucracy.

How do we know it's not more productive? Well, it's a slippery problem. Observe that it must be a slippery problem, or it all would have been debunked fair and square by now. But it's exceptionally difficult to measure software developer productivity, for all sorts of famous reasons. And it's even harder to perform anything resembling a valid scientific experiment in software development. You can't have the same team do the same project twice; a bunch of stuff changes the second time around. You can't have two teams do the same project; it's too hard to control all the variables, and it's prohibitively expensive to try it in any case. The same team doing two different projects in a row isn't an experiment either.

About the best you can do is gather statistical data across a lot of teams doing a lot of projects, and try to identify similarities, and perform some regressions, and hope you find some meaningful correlations. But where does the data come from? Companies aren't going to give you their internal data, if they even keep that kind of thing around. Most don't; they cover up their schedule failures and they move on, ever optimistic. (Author’s note: A lot of Agiloons tried to tell me that someone proved Agile “works” with a real scientific experiment. Their eyeballs were staring in different directions as they told me this.)

Well if you can't do experiments and you can't do proofs, there isn't much science going on. That's why it's a slippery problem. It's why fad diets are still enormously popular. People want fad diets to work, oh boy you bet they do, even I want them to work. And you can point to all these statistically meaningless anecdotes about how Joe lost 35 pounds on this one diet, and all those people who desperately want to be thinner will think, "Hey, it can't hurt. I'll give it a try."

That is exactly what I hear people say, every time a team talks themselves into trying an Agile Methodology. It's not a coincidence.

But writing about Bad Agile alone is almost guaranteed to be ineffective. I mean, you can write about how lame Scientology is, or how lame fad diets are, but it's not clear that you're changing anyone's mind. Quitting a viral meme is harder than quitting smoking. I've done both. In order to have the right impact, you have to offer an alternative, and I didn't have one before, not one that I could articulate clearly.

One of the (many) problems with Bad Agile is that they condescendingly lump all non-Agile development practices together into two buckets: Waterfall and Cowboy. Waterfall is known to be bad; I hope we can just take that as an axiom today. But what about so-called Cowboy programming, which the Agileers define as "each member of the team does what he or she thinks is best"?

Is it true that this is the only other development process? And is Cowboy Programming actually bad? They say it as if it's obviously bad, but they're not super clear on how or why, other than to assert that it's, you know, "chaos."

Well, as I mentioned, over the past year I've had the opportunity to watch both Bad Agile and Good Agile in motion, and I've asked the teams and tech leads (using both the Bad and Good forms) lots of questions: how they're doing, how they're feeling, how their process is working. I was really curious, in part because I'd consented to try Agile last Christmas ("hey, it can't hurt"), and wound up arguing with a teammate over exactly what metadata is allowed on index cards before giving up in disgust. Also in part because I had some friends on a team who were getting kind of exhausted from what appeared to be a Death March, and that kind of thing doesn't seem to happen very often at Google.

So I dug in, and for a year, I watched and learned.

The Good Kind

(cue happy rat)

I'm going to talk a little about Google's software development process. It's not the whole picture, of course, but it should suffice for today. I've been there for almost a year and a half now, and it took a while, but I think I get it now. Mostly. I'm still learning. But I'll share what I've got so far.

From a high level, Google's process probably does look like chaos to someone from a more traditional software development company. As a newcomer, some of the things that leap out at you include:

- There are managers, sort of, but most of them code at least half-time, making them more like tech leads.

- Developers can switch teams and/or projects any time they want, no questions asked; just say the word and the movers will show up the next day to put you in your new office with your new team.

- Google has a philosophy of never telling developers what to work on, and they take it pretty seriously.

- Developers are strongly encouraged to spend 20 percent of their time (and I mean their M-F, 8-5 time, not weekends or personal time) working on whatever they want, as long as it's not their main project.

- There aren't very many meetings. I'd say an average developer attends perhaps three meetings a week, including their 1:1 with their lead.

- It's quiet. Engineers are quietly focused on their work, as individuals or sometimes in little groups of two to five.

- There aren't Gantt charts or date-task-owner spreadsheets or any other visible project-management artifacts in evidence, not that I've ever seen.

- Even during the relatively rare crunch periods, people still go get lunch and dinner, which are (famously) always free and tasty, and they don't work insane hours unless they want to.

These are generalizations, sure. Old-timers will no doubt have a slightly different view, just as my view of Amazon is slightly biased by having been there in 1998 when it was a pretty crazy place. But I think most Googlers would agree that my generalizations here are pretty accurate.

How could this ever work?

I get that question a lot. Heck, I asked it myself. What's to stop engineers from leaving all the trouble projects, leaving behind bug-ridden operational nightmares? What keeps engineers working towards the corporate goals if they can work on whatever they want? How do the most important projects get staffed appropriately? How do engineers not get so fat that they routinely get stuck in stairwells and have to be cut out by the Fire Department?

I'll answer the latter question briefly, then get to the others. In short: we have this thing called the Noogler Fifteen, named after the Frosh Fifteen: the 15 pounds that many college freshmen put on when they arrive in the land of Stress and Pizza. Google has solved the problem by lubricating the stairwells.

As to the rest of your questions, I think most of them have the same small number of answers.

First, and arguably most importantly, Google drives behavior through incentives. Engineers working on important projects are, on average, rewarded more than those on less-important projects. You can choose to work on a far-fetched research-y kind of project that may never be practical to anyone, but the work will have to be a reward unto itself. If it turns out you were right and everyone else was wrong (the startup's dream), and your little project turns out to be tremendously impactful, then you'll be rewarded for it. Guaranteed.

The rewards and incentives are too numerous to talk about here, but the financial incentives range from gift certificates and massage coupons up through giant bonuses and stock grants, where I won't define "giant" precisely, but think of Google's scale and let your imagination run a bit wild, and you probably won't miss the mark by much.

There are other incentives. One is that Google is a peer-review oriented culture, and earning the respect of your peers means a lot there. More than it does at other places, I think. This is in part because it's just the way the culture works; it's something that was put in place early on and has managed to become habitual. It's also true because your peers are so damn smart that earning their respect is a huge deal. And it's true because your actual performance review is almost entirely based on your peer reviews, so it has an indirect financial impact on you.

Another incentive is that every quarter, without fail, they have a long all-hands in which they show every single project that launched to everyone, and put up the names and faces of the teams (always small) who launched each one, and everyone applauds. Gives me a tingle just to think about it. Google takes launching very seriously, and I think that being recognized for launching something cool might be the strongest incentive across the company. At least it feels that way to me.

And there are still other incentives; the list goes on and on and on; the perks are over the top, and the rewards are over the top, and everything there is so comically over the top that you have no choice, as an outsider, but to assume that everything the recruiter is telling you is a bald-faced lie, because there's no possible way a company could be that generous to all of its employees, all of them, I mean even the contractors who clean the micro-kitchens, they get these totally awesome "Google Micro-Kitchen Staff" shirts and fleeces.

There is nothing like it on the face of this earth. I could talk for hours, days about how amazing it is to work at Google, and I wouldn't be done. And they're not done either. Every week it seems like there's a new perk, a new benefit, a new improvement, a new survey asking us all if there's any possible way in which life at Google could be better.

I might have been mistaken, actually. Having your name and picture up on that big screen at End of Quarter may not be the biggest incentive. The thing that drives the right behavior at Google, more than anything else, more than all the other things combined, is gratitude. You can't help but want to do your absolute best for Google; you feel like you owe it to them for taking such incredibly good care of you.

Okay, incentives. You've got the idea. Sort of. I mean, you have a sketch of it. When friends who aren't at Google ask me how it is working at Google—and this applies to all my friends at all other companies equally, not just companies I've worked at—I feel just like how you'd feel if you'd just gotten out of prison, and your prison buddies, all of whom were sentenced in their early teens, are writing to you and asking you what it's like "on the outside." I mean, what would you tell them?

I tell 'em it's not too bad at all. Can't complain. Pretty decent, all in all.

Although the incentive-based culture is a huge factor in making things work the way they do, it only addresses how to get engineers to work on the "right" things. It doesn't address how to get those things done efficiently and effectively. So I'll tell you a little about how they approach projects.

Emergent Properties versus The Whip

The basic idea behind project management is that you drive a project to completion. It's an overt process, a shepherding: by dint of leadership, and organization, and sheer force of will, you cause something to happen that wouldn't otherwise have happened on its own.

Project management comes in many flavors, from lightweight to heavyweight, but all flavors share the property that they are external forces acting on an organization.

At Google, in contrast, projects launch because it's the least-energy state for the system.

Before I go on, I'll concede that this is a pretty bold claim, and that it's not entirely true. We do have project managers and product managers and people managers and tech leads and so on. But the amount of energy they need to add to the system is far less than what's typically needed in our industry. It's more of an occasional nudge than a full-fledged continuous push. Once in a while, a team needs a bigger nudge, and senior management needs to come in and do the nudging, just like anywhere else. But there's no pushing.

Incidentally, Google is a polite company, so there's no yelling, nor wailing and gnashing of teeth, nor escalation and finger-pointing, nor any of the artifacts produced at companies where senior management yells a lot. Hobbes tells us that organizations reflect their leaders; we all know that. The folks up top at Google are polite, hence so is everyone else.

Anyway, I claimed that launching projects is the natural state that Google's internal ecosystem tends towards, and it's because they pump so much energy into pointing people in that direction. All your needs are taken care of so that you can focus, and as I've described, there are lots of incentives for focusing on things that Google likes.

So launches become an emergent property of the system.

This eliminates the need for a bunch of standard project management ideas and methods: all the ones concerned with dealing with slackers, calling bluffs on estimates, forcing people to come to consensus on shared design issues, and so on. You don't need "war team meetings," and you don't need status reports. You don't need them because people already have incentive to do the right things and to work together well.

The project management techniques that Google does use are more like oil than fuel: things to let the project keep running smoothly, as opposed to things that force the project to move forward. There are plenty of meeting rooms, and there's plenty of open space for people to go chat. Teams are always situated close together in fishbowl-style open seating, so that pair programming happens exactly when it's needed (say 5 percent of the time), and never otherwise.

Google generally recognizes that the middle of the day is prone to interruptions, even at quiet companies, so many engineers are likely to shift their hours and come in very early or stay very late in order to find time to truly concentrate on programming. So meetings only happen in the middle of the day; it's very unusual to see a meeting start before 10 A.M. or after 4:30 P.M.. Scheduling meetings outside that band necessarily eats into the time when engineers are actually trying to implement the things they're meeting about, so they don't do it.

Google isn't the only place where projects are run this way. Two other kinds of organizations leap to mind when you think of Google's approach: startup companies and grad schools. Google can be considered a fusion of the startup and grad-school mentalities: on the one hand, it's a hurry-up, let's get something out now, do the simplest thing that could work, and we'll grow it later startup-style approach. On the other, it's relatively relaxed and low-key; we have hard problems to solve that nobody else has ever solved, but it's a marathon not a sprint, and focusing requires deep concentration, not frenzied meetings. And at the intersection of the two, startups and grad schools are both fertile innovation grounds in which the participants carry a great deal of individual responsibility for the outcome.

It's all been done before; the only thing that's really surprising is that Google has managed to make it scale.

The scaling is not an accident. Google works really hard on the problem, and they realize that having scaled this far is no guarantee it'll continue, so they're vigilant. That's a good word for it. They're always on the lookout to make sure the way of life and the overall level of productivity continue (or even improve) as they grow.

Google is an exceptionally disciplined company, from a software-engineering perspective. They take things like unit testing, design documents, and code reviews more seriously than any other company I've even heard about. They work hard to keep their house in order at all times, and there are strict rules and guidelines in place that prevent engineers and teams from doing things their own way. The result: the whole code base looks the same, so switching teams and sharing code are both far easier than they are at other places.

And engineers need great tools, of course, so Google hires great people to build their tools, and they encourage engineers (using incentives) to pitch in on tools work whenever they have an inclination in that direction. The result: Google has great tools, world-class tools, and they just keep getting better.

The list goes on. I could talk for days about the amazing rigor behind Google's approach to software engineering. But the main takeaway is that their scaling (both technological and organizational) is not an accident. And once you're up to speed on the Google way of doing things, it all proceeds fairly effortlessly—again, on average, and compared to software development at many other companies.

The Tyranny of the Calendar

We're almost done. The last thing I want to talk about here is dates. Traditional software development can safely be called Date-Oriented Programming, almost without exception.

Startup companies have a clock set by their investors and their budget. Big clients set target dates for their consultants. Sales people and product managers set target dates based on their evaluation of market conditions. Engineers set dates based on estimates of previous work that seems similar. All estimation is done through rose-colored glasses, and everyone forgets just how painful it was the last time around.

Everyone picks dates out of the air. "This feels like it should take about three weeks." "It sure would be nice to have this available for customers by beginning of Q4." "Let's try to have that done by tomorrow."

Most of us in our industry are date-driven. There's always a next milestone, always a deadline, always some date-driven goal to it.

The only exceptions I can think of to this rule are:

1. Open-source software projects.

2. Grad school projects.

3. Google.

Most people take it for granted that you want to pick a date. Even my favorite book on software project management, The Mythical Man-Month, assumes that you need schedule estimates.

If you're in the habit of pre-announcing your software, then the general public usually wants a timeframe, which implies a date. This is, I think, one of the reasons Google tends not to pre-announce. They really do understand that you can't rush good cooking, you can't rush babies out, and you can't rush software development.

If the three exceptions I listed above aren't driven by dates, then what drives them? To some extent it's just the creative urge, the desire to produce things; all good engineers have it. (There are many people in our industry who do this gig "for a living," and they go home and don't think about it until the next day. Open source software exists precisely because there are people who are more driven than that.)

But let's be careful: it's not just the creative urge; that's not always directed enough, and it's not always incentive enough. Google is unquestionably driven by time, in the sense that they want things done "as fast as possible." They have many fierce, brilliant competitors, and they have to slake their thirsty investors' need for growth, and each of us has some long-term plans and deliverables we'd like to see come to fruition in our lifetime.

The difference is that Google isn't foolish enough nor presumptuous enough to claim to know how long stuff should take. So the only company-wide dates I'm ever aware of are the ends of each quarter, because everyone's scrambling to get on that big launch screen and get the applause and gifts and bonuses and team trips and all the other good that comes of launching things with big impact at Google.

Everything in between is just a continuum of days, in which everyone works at optimal productivity, which is different for each person. We all have work-life balance choices to make, and Google is a place where any reasonable choice you make can be accommodated, and can be rewarding. Optimal productivity is also a function of training, and Google offers tons of it, including dozens of tech talks every week by internal and external speakers, all of which are archived permanently so you can view them whenever you like. Google gives you access to any resources you need in order to get your job done, or to learn how to get your job done. And optimal productivity is partly a function of the machine and context in which you're operating: the quality of your code base, your tools, your documentation, your computing platform, your teammates, even the quality of the time you have during the day, which should be food-filled and largely free of interruptions.

Then all you need is a work queue. That's it. You want hand-wavy math? I've got it in abundance: software development modeled on queuing theory. Not too far off the mark, though; many folks in our industry have noticed that organizational models are a lot like software models.

With nothing more than a work queue (a priority queue, of course), you immediately attain most of the supposedly magical benefits of Agile Methodologies. And make no mistake, it's better to have it in software than on a bunch of index cards. If you're not convinced, then I will steal your index cards.

With a priority queue, you have a dumping-ground for any and all ideas (and bugs) that people suggest as the project unfolds. No engineer is ever idle, unless the queue is empty, which by definition means the project has launched. Tasks can be suspended and resumed simply by putting them back in the queue with appropriate notes or documentation. You always know how much work is left, and if you like, you can make time estimates based on the remaining tasks. You can examine closed work items to infer anything from bug regression rates to (if you like) individual productivity. You can see which tasks are often passed over, which can help you discover root causes of pain in the organization. A work queue is completely transparent, so there is minimal risk of accidental duplication of work.

And so on. The list goes on, and on, and on.

Unfortunately, a work queue doesn't make for a good marketing platform for seminars and conferences. It's not glamorous. It sounds a lot like a pile of work, because that's exactly what it is.

Bad Agile in More Detail

I've outlined, at a very high level, one company's approach to software development that is neither an Agile Methodology, nor a Waterfall cycle, nor yet Cowboy Programming. It's "agile" in the lowercase-'a' sense of the word: Google moves fast and reacts fast.

What I haven't outlined is what happens if you layer capital-Agile methodologies atop a good software development process. You might be tempted to think: "well, it can't hurt!" I even had a brief fling with it myself last year.

The short answer is: it hurts. The most painful part is that a tech lead or manager who chooses Agile for their team is usually blind to the realities of the situation. Bad Agile hurts teams in several ways.

First, Bad Agile focuses on dates in the worst possible way: short cycles, quick deliverables, frequent estimates and re-estimates. The cycles can be anywhere from a month (which is probably tolerable) down to a day in the worst cases. It's a nicely idealistic view of the world.

In the real world, every single participant on a project is, as it turns out, a human being. We have up days and down days. Some days you have so much energy you feel you could code for 18 hours straight. Some days you have a ton of energy, but you just don't feel like focusing on coding. Some days you're just exhausted. Everyone has a biological clock and a a biorhythm that they have very little control over, and it's likely to be phase-shifted from the team clock, if the team clock is ticking in days or half-weeks.

Not to mention your personal clock: the events happening outside your work life that occasionally demand your attention during work hours.

None of that matters in Bad Agile. If you're feeling up the day after a big deliverable, you're not going to code like crazy; you're going to pace yourself because you need to make sure you have reserve energy for the next big sprint. This impedance mismatch drives great engineers to mediocrity.

There's also your extracurricular clock: the set of things you want to accomplish in addition to your main project: often important cleanups or other things that will ultimately improve your whole team's productivity. Bad Agile is exceptionally bad at handling this, and usually winds up reserving large blocks of time after big milestones for everyone to catch up on their side-project time, whether they're feeling creative or not. Bad Agile folks keep their eye on the goal, which hurts innovation. Sure, they'll reserve time for everyone to clean up their own code base, but they're not going to be so altruistic as to help anyone else in the company. How can you, when you're effectively operating in a permanent day-for-day slip?

Bad Agile seems for some reason to be embraced by early risers. I think there's some mystical relationship between the personality traits of "wakes up before dawn," "likes static typing but not type inference," "is organized to the point of being anal," "likes team meetings," and "likes Bad Agile." I'm not quite sure what it is, but I see it a lot.

Most engineers are not early risers. I know a team that has to come in for an 8:00 A.M. meeting at least once (maybe several times) a week. Then they sit like zombies in front of their email until lunch. Then they go home and take a nap. Then they come in at night and work, but they're bleary-eyed and look perpetually exhausted. When I talk to them, they're usually cheery enough, but they rarely finish their sentences.

I ask them (individually) if they like the Agile approach, and they say things like: "well, it seems like it's working, but I feel like there's some sort of conservation of work being violated…," and "I'm not sure; it's what we're trying I guess, but I don't really see the value," and so on. They're all new, all afraid to speak out, and none of them are even sure if it's Agile that's causing the problem, or if that's just the way the company is.

That, my friends, is not "agile;" it's a just load of hooey. And it's what you get whenever any manager anywhere decides to be a chump.

Good Agile Should Drop the Name

I would caution you to be skeptical of two kinds of claims:

- "All the good stuff he described is really Agile."

- "All the bad stuff he described is the fault of the team's execution of the process."

You'll hear them time and again. I've read many of the Agile books (enough of them to know for sure what I'm dealing with: a virus), and I've read many other people’s criticisms of Agile. Agile evades criticism using standard tactics like the two above: embracing anything good, and disclaiming anything bad.

If a process is potentially good, but 90 percent (or more) of the time, smart and well-intentioned people screw it up, then it's a bad process. They can only say it's the team's fault so many times before it's not really the team's fault.

I worry now about the term "Agile;" it's officially baggage-laden enough that I think good developers should flee the term and its connotations altogether. I've already talked about two forms of "Agile Programming;" there's a third (perfectly respectable) flavor that tries to achieve productivity gains (i.e., "Agility") through technology. Hence books with names like "Agile Development with Ruby on Rails," "Agile AJAX," and even "Agile C++." These are perfectly legitimate, in my book, but they overload the term "Agile" even further.

And frankly, most Agile out there is plain old Bad Agile.

So if I were you, I'd take Agile off your resume. I'd quietly close the SCRUM and XP books and lock them away. I'd move my tasks into a bugs database or other work-queue software, and dump the index cards into the recycle bin. I'd work as fast as I can to eliminate Agile from my organization.

And then I'd focus on being agile.

But that's just my take on it, and it's 4:00 A.M.. Feel free to draw your own conclusions. Either way, I don't think I'm going to be an Early Riser tomorrow.

Oh, I almost forgot the obvious disclaimer: I do not speak for Google. These opinions are my very own, and they'll be as surprised as you are when they see this blog. Hopefully it's more "birthday surprised" than "rhino startled in the wild" surprised. We'll see!

Author’s Note: Is Google going to stay relevant?

I haven’t published this one before, at least not outside Google. This was written exclusively for Googlers. It is the “prequel” to my Platforms Rant. It is Part One of an 11-part series in which I tackle nine separate, orthogonal dimensions of wrongness about Google’s strategy today. The Platforms Rant was Part Two, and I had the other ones all planned out.

Then the Platforms Rant accidentally posted externally, and it was a shot that echoed around the world. I narrowly avoided losing my job—not because of the rant, but because I could easily have mentioned something company-confidential, seeing as it was intended, after all, to be internal-only. So I’d dodged a bullet. But the publicity was a little much, even for me—a lot much, truth be told—and I decided to put my 11-part series on hold for now.

So here’s Part One of the series. It’s just my opinion, of course—given that I’m disagreeing with a key aspect of Google’s philosophical strategy here, it clearly represents my own opinion and not Google’s. They’re cool with people expressing dissenting opinions internally. They’re also cool with just ignoring you, and I’m fine with that too. I just want an “I told you so” paper trail once they finally realize I’m right.

I didn’t ask permission to publish this post externally (e.g., in this book). I could be fired over it! Hopefully that makes it at least moderately more titillating. Probably not, though. This stuff is pretty obvious to everyone on earth.

Jamie Zawinski’s been a repeat guest star in these intros, so I’ll channel him again. He had what I think is the best summary of the issue I’m arguing about here. He said, in his blog:

Google's statement is obvious bullshit, and here's why. The way you "support" pseudonyms is as follows:

1. Stop deleting peoples' accounts when you suspect that the name they are using is not their legal name.

2. There is no step 2.

Sigh. Someday I will be as awesome as JWZ. I keep telling myself that. It’s good to have a goal.

And someday I’ll be brave enough, and motivated enough, I hope, to write Parts Three through Nine, plus the two summary/conclusion posts that don’t really introduce new material, but serve to ensure the coffin-nails are pounded in nice and tight—most likely on my career.

Someday.

That, or I’ll just go do it right myself. We’ll see!

Is Google Going to Stay Relevant?

In my time at Google I've never before had occasion to worry about our fundamental ability to remain relevant.

We've made mistakes now and then, but I've never been concerned by our slips because I've always felt that Google's leadership has its finger on the world's pulse. There has never been a fundamental disconnect between Google and society, nor between Google and common sense.

When you hear people outside the company talk about Google, in the grocery store or on an airplane or at a soccer game, there has always been this sense of "Google's got my back" underlying the conversation, even when Google is doing something that people may not understand or agree with. People respect Google because regardless of whatever mistakes we might make, Google "gets it." Google is known for building things that are highly relevant to regular people.

We all know this is not true of Microsoft. They were supremely relevant in the 1980s, but their relevance started to fade before our eyes in the 1990s, and all traces of it had vanished by 2003 or so. With the notable exception of their XBox platform, which became their Pepsi to Sony's Coca-Cola, Microsoft has failed absolutely in their attempts to reconnect with regular people. Any non-technical person can look at any of Microsoft's products or services from Bob to Zune to Bing to Kin and deduce immediately that the executive leadership at Microsoft no longer "gets it."

All Microsoft can do today is copy others and play a frantic game of catch-up. But they rarely have more than a superficial understanding of the space they're entering. It's hard to play catch-up in a space you don't understand in the first place, so they generally do a poor job of it. They have ceased to be a relevant company, and they have gradually ceded their old turf to Apple, Google, Amazon, Facebook, and a host of other players who have managed to remain relevant.

I've worked for companies that sank from relevance into irrelevance. It happened, for example, at a company called Geoworks. Twenty of us Googlers used to work there, some 15 or 20 years back. Geoworks was a game company that reinvented itself as an operating systems company, and then again as a smartphone company. And for a brief time, working with companies like Nokia and Sharp and Matsushita and HP, Geoworks achieved a modest kind of relevance in the smartphone market.

But then 1993 happened, and Netscape and the Web came along and created the biggest new market opportunity since the Industrial Revolution, and Geoworks just flat-out ignored it. The 100-odd engineers at the company saw the huge opportunity, and we asked our CEO Gordon R. Elevant at an all-hands why we didn't do something with it. He mumbled something about it "not being our core competency." Gordon didn't "get" the Web. Over the next few years Geoworks was dropped by key partners, then delisted from the stock exchange, and the company was finally dismantled around 2001-ish.

I know this sounds like a bunch of rambling and pointless history lessons. But while I was at Geoworks I learned that strong companies can very easily lose their way and never find it again. It happened to DEC, to HP/Compaq, to IBM, and even to Microsoft.

Amazon.com even lost their way for a little while, during my seven-year tenure there. Jeff Bezos informed us at an all-hands that our staple product, BMVD (books/music/video/dvd) was digitizable and thus piratable. The margins were evaporating, and BMVD would soon become irrelevant as a revenue source, Bezos told us. But hardlines—things like clothes and consumer electronics—have short shelf lives: nobody wants last summer's barbecue model. And hardlines present complex supply and fulfillment issues. At the time Bezos was giving us this news, Amazon's bid to out-eBay eBay had just failed, and now it was looking like Amazon itself was going to become irrelevant.

Bezos has been fighting the loss of relevance with everything he's got. We've seen lots of stuff from them in the past decade, including the Kindle, Mechanical Turk, AWS, and EC2/S3. But that's just the beginning. Bezos does NOT want to become irrelevant, and he's so far been fairly successful at extending Amazon's brand into relevant new spaces: a feat that many marketing experts claim is near-impossible. In fact I suspect that Amazon will eventually succeed at a total reinvention of their brand, because Bezos still "gets it." He never did understand operations or employee retention, but he does okay at understanding markets.

Sometimes people call the search for renewed relevance an "existential crisis." Candidates from Microsoft use that term surprisingly often when they're interviewing with us.

As I watched Geoworks and Microsoft lose their relevance, never to regain it, and I watched Amazon struggle against the tide of their relevance evaporating along with their profit margins, I learned to be extremely sensitive to this issue. Working at a company mired in existential crisis is no fun at all. They tend to go into catch-up mode at the drop of a hat, whipping everyone into a frenzy to get this or that new product out because this will be the Big One that makes them relevant again.

Being super-sensitive to the issue, I have acquired something of a reputation among friends as a canary who can sense the impending advance of irrelevance long before it is broadly recognized. And I have also acquired something of a reputation for fleeing sinking ships, especially when I'm waist-deep in rising water and the captain tells me the ship is just fine.

And I tell you this: our Real Names policy has me very worried. The water's only lapping over my toes, but I was also in the US Navy as a nuclear reactor operator, and I learned to take even tiny leaks very seriously.

I've never been worried about Google before. Not once. I've seen us jump into projects that were ill-conceived—the now-defunct Google Next (GN) in Kirkland, for instance. I declined to work on GN because I looked at the mockups and decided I didn't understand it. It didn't resonate with me. I couldn't tell why were we doing it. But I knew, with a faith that had never been shaken, that common sense would eventually prevail, because Google only gets behind offerings that real people can bond with.

It was the same with Wave. I never started using it because I didn't understand it. At Goo Camp where the presenters got a huge ovation after their sneak-peak demo, I was in the back, honestly wondering what everyone was clapping about. What had I missed? In retrospect I think the applause was just tech people clapping for tech people, maybe, but I guess I'm not much of a tech adopter. I look at product offerings from the perspective of Plain Old Me: not a techie, but a regular person who needs pants and toothbrushes and stuff. And I just didn't "get" Wave.

Apparently other regular people didn't get Wave either, so we pulled the plug. I think it's cool that we launched Wave and let it run for a while. But what's really cool is that rather than desperately try to pretend that it was almost-relevant, and then spend years trying to fix it, we just pulled the plug. That's the kind of thing Microsoft just wouldn't be able to do, because they seem to have lost all their corporate common sense, if they ever really had any.

Our Real Names policy, and the thinking that underlies it, has me starting to worry about our future relevance. With GN, and Wave, and any number of other failed products we've built, I was never concerned about Google itself. That's what's different this time. This time I'm seeing what may be the first sign ever that "The Thing" has happened: whatever Thing it is that sent DEC, IBM, HP, Geoworks, Microsoft, and so many other companies off into the woods of irrelevance, often never to return. I don't have a real name for The Thing, but I assure you it is as real as Death itself. The Thing is an invisible beast that feeds on companies.

I am concerned because it seems like plain old common sense that Real Names is not the way the real world works. I'm not talking about restaurants here. I'm talking about the online world that has evolved over the past four decades from Usenet and MUDs and BBS systems into the online world we have today, which consists of extremely fancy, scalable versions of Usenet and MUDs and BBS systems.

This is a fairly astonishing outcome. You would expect the online world to be substantially different than it was two generations ago. But no, it's just a higher-tech version of the same old thing: discussion groups, chat rooms, real-time messaging (remember the Unix "talk" command?), offline messaging, media and content sharing, multiplayer games, document creation, for-sale lists, news feeds, knowledge databases, online shopping, spam, pornography.

That's pretty much it. That's the human condition.

Even the briefest of surveys of how accounts work in the real world will show you that Anonymity is King. Look at user accounts on shared knowledge bases like Wikipedia and IMDb. Or news aggregator communities like Slashdot, Reddit, Digg, Y-Combinator. Or online games—all of them, no exceptions. Or online gaming networks like PSN, XBox Live, PopCap. Take a look at online dating sites like eHarmony and OkCupid. Or P2P trading sites like eBay and craigslist, not to mention every hobbyist trading/sharing site in existence. Take a look at online chat rooms. IRC clients. Discussion boards. Forums. Blogs.

Look around and you'll see that nobody uses their real name anywhere. In fact communities often discourage you from doing it. Nobody wants to go adventuring with an elvish wizard named Doug Smith.

Take a look at the whole world, and you will be looking at Avatars. That's how people interact online. You can argue about the validity of this phenomenon, or the reasons behind it, but you can't ignore the phenomenon itself. The whole world interacts via Avatars—pseudonymous and persistent and anonymous to the extent that your real identity is hidden from the people you are interacting with, unless you choose to reveal it.

Facebook is the notorious standout exception to the Avatar rule, because the dude who started it (I refuse to bother remembering his name, but it's something like Jerry Bruckheimer) said, famously, that "anonymity is cowardice," by which he meant of course that anonymity does not pad his wallet as easily as real names. And Facebook has certainly enjoyed a measure of success.

But speaking as a regular person who doesn't use Facebook, and has never liked it nor understood it, I believe that they are successful not because of their policy, but in spite of it. Speaking here as a regular person who uses social networking sites every day, whether they be online games or knowledge databases or discussion boards or chat rooms or news aggregators or hobbyist sites, it's clear to me that there is a massive untapped global market that Facebook is missing out on because of their naming and identity policies.

So when I first heard about Google+ I was very excited, because I knew that we would be uniquely positioned to deliver safe-for-everyone Anonymity a world where Anonymity has been King for 40 years, and we could do it in a way that is every bit as monetizable and moderatable as Real Names.

The secret, of course, is no secret at all: with a Google Account tied to a credit card, a person can have any number of avatars, and only Google ever need know which real person is behind that avatar. It's an extraordinarily serendipitous solution to the universal problem with pseudonymous networks. Having an authority like Google manage the real identities without divulging them to the participants permits behind-the-scenes abuse control and moderator activity (both human and algorithmic) while giving people the freedom they crave to be whoever they want in any given social setting.

So yeah, I was extremely excited. I even had a bunch of ideas for stuff I wanted to write with this incredible new social network we were building. It seemed—and still seems—patently obvious to me that we have the near-unique ability to revolutionize and in fact commoditize the ad-hoc social networks that sprout up around every relevant web property that does not happen to be Facebook.

Hence it is with some dismay that I have learned—admittedly without following the discussions at all, so it's all third-hand information at best—of our Real Names policy and its underlying philosophy.

I am dismayed because our policy seems symptomatic of a complete disconnect with online reality and with the common intuition of regular people. And worse, we seem to be uninterested in learning whether it is the right policy for us to have. We are not saying things like: "We will evaluate it and figure out whether it is the Right Thing." Instead we are simply reciting strange mantras about shirts in restaurants, as if an offline metaphor can outweigh the evidence of four decades of online experience, and charging ahead with implementing our chosen policy in what seems to be a rather brutal and un-Googley fashion.

Worst of all, we seem intent on preventing people from building anything atop our APIs—at least as far as I can see from examining said APIs—that could constitute an experience similar to eBay, or reddit, or eHarmony, or World of Warcraft, or IMDb, or any other online site where real people hang out doing regular real-people stuff through their persistent avatars.

I am concerned that we seem to think this is a debate about nicknames. Nicknames are critically important, but they are far from sufficient. It bothers me a little that we have fought against nicknames, and that we are only grudgingly thinking of adding them. At least we seem to be quasi-responding to market pressure here. But it bothers me a great deal that we seem to believe that nicknames will "solve the problem" that people around the world are raising. Nicknames are not the same thing as Avatars. Only a company that has lost touch with everyday online social common sense could think otherwise.

A platform is only as relevant as the apps you can build on it. The Google+ platform is not relevant to me or my interests. I wanted it to be, but it defies the implementation of not just all my own ideas, but also all of the ideas presented to me by friends and family members. To anyone wishing to build a great avatar-based interaction system, or augment an existing system with avatars, the Google+ platform comes across as defiantly irrelevant.

I will not try to argue that Facebook is not relevant: clearly they have found a market, one that my wife and I colloquially refer to as "stalkers and show offs." And Facebook is making an impact in that space. More power to them. Unlike Microsoft, Facebook has at least found a way to be relevant to a particular subculture.

In a similar vein, I would not dream of arguing that the Google+ APIs are irrelevant in general simply because they are irrelevant to me personally.

But I think it is significant that for the first time in my experience, Google seems to be blind to the existence of a market. We are not saying: "We are choosing not to tackle that market at this time," or if we are saying that, I haven't heard it. Instead we seem to be saying: "This is how people want to interact online." To a regular person, that is how we are coming across. We appear to be saying: "This is how you want to interact with other people online."

And that is a strange thing for us to be saying. In the past we have not, to my knowledge, ever tried to be in the business of telling people what they want. We are not a marketing firm. Until now we have been in the business of figuring out what people already want, and making it easier for them to do it.

But our entire Google+ stack—from our policies to our APIs to our external communications—is saying: "You don't want this." Or perhaps it is saying: "If you want this, then we do not want you." But from a global relevance perspective, the two messages are identical.

It seems to me that we are frantically trying to play catch-up in a space that we never really understood in the first place. And to date, that is a phenomenon that I have only witnessed in companies undergoing an existential crisis.

My faith in this company's vision and leadership has not been broken. In fact it would be premature to say that it has even been shaken. All that has really happened is that I have been given occasion to wonder, for the first time, whether The Thing is happening again.

I sincerely hope that it is not.

Author’s Note: Google Platforms Rant

This was the Big One. The most famous thing I’ve ever written. I doubt very much that it’s the most famous thing I ever will write, at least if I live a couple handful more years, because I could dash off half a dozen similar rants on corporate culture that would all be every bit as entertaining and eye-opening.

That is to say, I could dash them off if I stopped valuing my nuts. And perhaps someday I will!

The story of how this post happened is amusing in its own right. I had gone down to Mountain View on a biz trip to Google headquarters for the whole week. My wife was heading down to join me the next day. I’d just spent an entire day arguing with my peer managers about the value of a platform. We were getting word from above that we were going to start doing Services in Infrastructure, and I was all excited because, you know, finally. And then there was all this pushback.

I mean, it’s not as if my peers were wrong. At Google there’s safety in building products. It’s the less risky route by far. Nobody at Google ever got fired for building products. And the benefits of a platform are sometimes pretty hard to quantify, whereas the costs are easy: it’s about 3x to 5x harder, give or take.

But we were so close. I’d been chafing about this cultural issue for years, ever since I’d left Amazon. I’m a Platforms guy. It’s just what I do. I’ve been doing them since college, twenty years ago. For me the benefits are just baked in, to be taken for granted, because I’ve seen them work so well. And here suddenly my management chain was interested in Services as a way to enable Googlers outside our group to build their own tools, rather than sitting around waiting for us to build their tools for them. (That’s what we do in my org—we build tools for engineers.)

We were so close, and here it was about to get derailed from the inside. I could just feel a rant coming. So I went back to my hotel, grabbed a bottle of wine, and busted out Part Two of my 11-part series on Things I Think Google is Screwing Up, Even Though I Love ‘Em. I wasn’t just going to appeal to my peers in Developer Infrastructure, no: I decided it was time for me to appeal to all Googlers. To every single person at our, what, 20k-employee company. Me. Mister Nobody.

So it was critical that I be respectful, and accurate, and comprehensive, and careful, and still manage to tell a compelling story that got people to see things my way.

And that approach worked right up to about halfway through the bottle, at which the storytelling sort of took over. Even slobbering drunk I still respect my fellow Googlers more than you could imagine—every day I feel like I’ve won the lottery to be working there—so the end result was still flavored with genuine, undisguised caring. I think that really resonated with a lot of outside-world readers.

But slobbering drunk, even if I can still write reasonably well, evidently I have trouble with buttons and UI widgets and such. All I remember is spending 2 hours writing it out in Emacs, followed by 90 minutes of editing (and giggling a lot), and finally dorking with Google+ for a good solid 30 minutes while I tried to figure out how to post the damn thing. It’s a lot different from Blogger, and my expectations were for something larger than a tiny text input box saying “Share what’s new.”

Google employees have an internal version of Google+ that looks exactly like the external version. The only way to tell the difference is by looking at your email address in the upper-right corner, or your profile pic (if they’re different). I remember doing quite a few basic sanity checks to make sure I didn’t post externally, but even more checks to figure out how to post at all.

After I finally figured out how to post it, I remember hitting “Share” (or whatever it is), declaring “That’ll show em!” and being asleep in bed within about 20 seconds—around midnight, give or take.

At 1:30am my phone rang. I was in a random hotel, one that I hadn’t told anyone about—you don’t say where you stayed until you file your expense report, sometimes weeks after the trip is over. I’d already said goodnight to my wife, who was still up in Kirkland. Who would be calling me at 1:30am?

It turned out to be a really nice guy in the Zurich office who was calling me on behalf of one of our PR departments, I think London maybe. He asked me if I realized my post had gone external, outside Google. “WHAT!? NOOO!” Was my first reaction. “NOT POSSIBLE!” But even intoxicated and tired as I was at 1:30 A.M., a tiny voice was telling me that it was unlikely that a random person in Zurich would have somehow found out where I was staying and called me at 1:30 A.M. if my post had not in fact gone rogue.

He was very nice, and PR was very nice, and the rest is reasonably well documented now in my follow-up post(s) and comments. It eventually blew over, and we all had a good laugh, and mostly what I have to show for it is just a lot more gray hair.

To this day, I have no idea how they knew where I was staying. I’m not sure I want to know.

It’s been 11 months since the post went viral, and in the meantime we’ve been slowly but surely building Platforms in my little corner of Google. And it’s nice. Other than that, life hasn’t changed too much, except that I get a lot more job offers than I used to. Even from Amazon, ironically enough. I ask them, “Are you sure you cleared this with the higher-ups?” And they say: “Oh yes, we’re well aware of your, uh, history, and it’s all good,” and I’m always on the verge of asking: “So, umm… how high up did you go, exactly?” Because, well… that’s one toasty bridge, is all I’m saying. It smells like a trap.

But Google is still the most awesome place to work on earth, so I suspect I’ll remain settled there for a pretty good stretch.

Google Platforms Rant

I was at Amazon for about six and a half years, and now I've been at Google for that long. One thing that struck me immediately about the two companies—an impression that has been reinforced almost daily—is that Amazon does everything wrong, and Google does everything right. Sure, it's a sweeping generalization, but a surprisingly accurate one. It's pretty crazy. There are probably a hundred or even two hundred different ways you can compare the two companies, and Google is superior in all but three of them, if I recall correctly. I actually did a spreadsheet at one point but Legal wouldn't let me show it to anyone, even though recruiting loved it.

I mean, just to give you a very brief taste: Amazon's recruiting process is fundamentally flawed by having teams hire for themselves, so their hiring bar is incredibly inconsistent across teams, despite various efforts they've made to level it out. And their operations are a mess; they don't really have SREs, and they make engineers pretty much do everything, which leaves almost no time for coding—though again this varies by group, so it's luck of the draw. They don't give a single shit about charity or helping the needy or community contributions or anything like that. Never comes up there, except maybe to laugh about it. Their facilities are dirt-smeared cube farms without a dime spent on decor or common meeting areas. Their pay and benefits suck, although much less so lately due to local competition from Google and Facebook. But they don't have any of our perks or extras—they just try to match the offer-letter numbers, and that's the end of it. Their code base is a disaster, with no engineering standards whatsoever except what individual teams choose to put in place. (Author’s note: I hear Amazon is somewhat better these days. Bully for them.)

To be fair, they do have a nice versioned-library system that we really ought to emulate, and a nice publish-subscribe system that we also have no equivalent for. But for the most part they just have a bunch of crappy tools that read and write state machine information into relational databases. We wouldn't take most of it even if it were free.

I think the pubsub system and their library-shelf system were two out of the grand total of three things Amazon does better than google. (Author’s note: I now think Google’s pubsub system is better. Amazon still does components better, though.)

I guess you could make an argument that their bias for launching early and iterating like mad is also something they do well, but you can argue it either way. They prioritize launching early over everything else, including retention and engineering discipline and a bunch of other stuff that turns out to matter in the long run. So even though it's given them some competitive advantages in the marketplace, it's created enough other problems to make it something less than a slam-dunk.

But there's one thing they do really really well that pretty much makes up for ALL of their political, philosophical, and technical screw-ups.

Jeff Bezos is an infamous micro-manager. He micromanages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally—wisely—left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they're all still there, and Larry is not.

Micromanaging isn't that third thing that Amazon does better than us, by the way. I mean, yeah, they micromanage really well, but I wouldn't list it as a strength or anything. I'm just trying to set the context here, to help you understand what happened. We're talking about a guy who in all seriousness has said on many public occasions that people should be paying him to work at Amazon. He hands out little yellow stickies with his name on them, reminding people "who runs the company" when they disagree with him. The guy is a regular… well, Steve Jobs, I guess. Except without the fashion or design sense. Bezos is super smart; don't get me wrong. He just makes ordinary control freaks look like stoned hippies.

So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion—back around 2002 I think, plus or minus a year—he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.

His Big Mandate went something along these lines:

1. All teams will henceforth expose their data and functionality through service interfaces.

2. Teams must communicate with each other through these interfaces.

3. There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4. It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols—doesn't matter. Bezos doesn't care.

5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6. Anyone who doesn't do this will be fired.

7. Thank you; have a nice day!

Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that number seven was a little joke I threw in, because Bezos most definitely does not give a shit about your day.

Number six, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an ex-Army Ranger, West Point Academy graduate, ex-boxer, ex-Chief Torturer slash CIO at WalMart, and is a big genial scary man who used the word "hardened interface" a lot. Rick was a walking, talking hardened interface himself, so needless to say, everyone made lots of forward progress and made sure Rick knew about it.

Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon's vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon's dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:

- Pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.

- Every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.

- Monitoring and QA are the same thing. You'd never think so until you try doing a big SOA. But when your service says, "oh yes, I'm fine," it may well be the case that the only thing still functioning in the server is the little component that knows how to say "I'm fine, roger roger, over and out" in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. So they're a continuum.

- If you have hundreds of services, and your code MUST communicate with other groups' code via these services, then you won't be able to find any of them without a service-discovery mechanism. And you can't have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.

- Debugging problems with someone else's code gets a lot harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.

That's just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers.

This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally.

At this point they don't even do it out of fear of being fired. I mean, they're still afraid of that; it's pretty much part of daily life there, working for the Dread Pirate Bezos and all. But they do services because they've come to understand that it's the Right Thing. There are without question pros and cons to the SOA approach, and some of the cons are pretty long. But overall it's the right thing because SOA-driven design enables Platforms.

That's what Bezos was up to with his edict, of course. He didn't (and doesn't) care even a tiny bit about the well-being of the teams, nor about what technologies they use, nor in fact any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.

You wouldn't really think that an online bookstore needs to be an extensible, programmable platform. Would you?

Well, the first big thing Bezos realized is that the infrastructure they'd built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel' o' other services browsable at aws.amazon.com. These services host the backends for some pretty successful companies, reddit being my personal favorite of the bunch.

The other big realization he had was that he can't always build the right thing. I think Larry Tesler might have struck some kind of chord in Bezos when he said his mom couldn't use the goddamn website. It's not even super clear whose mom he was talking about, and doesn't really matter, because nobody's mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I've just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.

I'm not really sure how Bezos came to this realization—the insight that he can't build one product and have it be right for everyone. But it doesn't matter, because he gets it. There's actually a formal name for this phenomenon. It's called Accessibility, and it's the most important thing in the computing world.

The. Most. Important. Thing.

If you're sorta thinking, "Huh? You mean like, blind and deaf people accessibility?" then you're not alone, because I've come to understand that there are lots and lots of people just like you: people for whom this idea does not have the right accessibility, so it hasn't been able to get through to you yet. It's not your fault for not understanding, any more than it would be your fault for being blind or deaf or motion-restricted or living with any other disability. When software—or idea-ware for that matter—fails to be accessible to anyone for any reason, it is the fault of the software or of the messaging of the idea. It is an accessibility failure.

Like anything else big and important in life, Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there's more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.

But I'll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network.

So yeah. In case you hadn't noticed, I could actually write a book on this topic. A fat one, filled with amusing anecdotes about ants and rubber mallets at companies I've worked at. But I will never get this little rant published, and you'll never get it read, unless I start to wrap up.

That one last thing that Google doesn't do well is Platforms. We don't understand platforms. We don't "get" platforms. Some of you do, but you are the minority. This has become painfully clear to me over the past six years. I was kind of hoping that competitive pressure from Microsoft and Amazon and more recently Facebook would make us wake up collectively and start doing universal services. Not in some sort of ad-hoc, half-assed way, but in more or less the same way Amazon did it: all at once, for real, no cheating, and treating it as our top priority from now on.

But no. No, it's like our tenth or eleventh priority. Or fifteenth, I don't know. It's pretty low. There are a few teams who treat the idea very seriously, but most teams either don't think about it all, ever, or only a small percentage of them think about it in a very small way.

It's a big stretch even to get most teams to offer a stubby service to get programmatic access to their data and computations. Most of them think they're building products. And a stubby service is a pretty pathetic service. Go back and look at that partial list of learnings from Amazon, and tell me which ones Stubby gives you out of the box. As far as I'm concerned, it's none of them. Stubby's great, but it's like parts when you need a car.

A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.

Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: "So is it the Stalker API?" She got all glum and said "Yeah." I mean, I was joking, but no… the only API call we offer is to get someone's stream. So I guess the joke was on me.

Microsoft has known about the Dogfood rule for at least twenty years. It's been part of their culture for a whole generation now. You don't eat People Food and give your developers Dog Food. Doing that is simply robbing your long-term platform value for short-term successes. Platforms are all about long-term thinking.

Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that's not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there's something there for everyone.

Our Google+ team took a look at the aftermarket and said: "Gosh, it looks like we need some games. Let's go contract someone to, um, write some games for us." Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.

You can't do that. Not really. Not reliably. There have been precious few people in the world, over the entire history of computing, who have been able to do it reliably. Steve Jobs was one of them. We don't have a Steve Jobs here. I'm sorry, but we don't.

Larry Tesler may have convinced Bezos that he was no Steve Jobs, but Bezos realized that he didn't need to be a Steve Jobs in order to provide everyone with the right products: interfaces and workflows that they liked and felt at ease with. He just needed to enable third-party developers to do it, and it would happen automatically.

I apologize to those (many) of you for whom all this stuff I'm saying is incredibly obvious, because yeah. It's incredibly frigging obvious. Except we're not doing it. We don't get Platforms, and we don't get Accessibility. The two are basically the same thing, because platforms solve accessibility. A platform is accessibility.

So yeah, Microsoft gets it. And you know as well as I do how surprising that is, because they don't "get" much of anything, really. But they understand platforms as a purely accidental outgrowth of having started life in the business of providing platforms. So they have thirty-plus years of learning in this space. And if you go to msdn.com, and spend some time browsing, and you've never seen it before, prepare to be amazed. Because it's staggeringly huge. They have thousands, and thousands of API calls. They have a huge platform. Too big in fact, because they can't design for squat, but at least they're doing it.

Amazon gets it. Amazon's AWS is incredible. Just go look at it. Click around. It's embarrassing. We don't have any of that stuff.

Apple gets it, obviously. They've made some fundamentally non-open choices, particularly around their mobile platform. But they understand accessibility and they understand the power of third-party development and they eat their dog food. And you know what? They make pretty good dog food. Their APIs are a hell of a lot cleaner than Microsoft's, and have been since time immemorial.

Facebook gets it. That's what really worries me. That's what got me off my lazy butt to write this thing. I hate blogging. I hate… plussing, or whatever it's called when you do a massive rant in Google+ even though it's a terrible venue for it, but you do it anyway because in the end you really do want Google to be successful. And I do! I mean, Facebook wants me there, and it'd be pretty easy to just go. But Google is home, so I'm insisting that we have this little family intervention, uncomfortable as it might be.

After you've marveled at the platform offerings of Microsoft and Amazon, and Facebook I guess (I didn't look because I didn't want to get too depressed), head over to developers.google.com and browse a little. Pretty big difference, eh? It's like what your fifth-grade nephew might mock up if he were doing an assignment to demonstrate what a big powerful platform company might be building if all they had, resource-wise, was one fifth grader.

Please don't get me wrong here—I know for a fact that the dev-rel team has had to FIGHT to get even this much available externally. They're kicking ass as far as I'm concerned, because they DO get platforms, and they are struggling heroically to try to create one in an environment that is at best platform-apathetic, and at worst often openly hostile to the idea.

I'm just frankly describing what developers.google.com looks like to an outsider. It looks childish. Where's the Maps APIs in there for Christ's sake? Some of the things in there are labs projects. And the APIs for everything I clicked were… they were paltry. They were obviously dog food. Not even good organic stuff. Compared to our internal APIs it's all snouts and horse hooves.

And also don't get me wrong about Google+. They're far from the only offenders. This is a cultural thing. What we have going on internally is basically a war, with the underdog minority Platformers fighting a more or less losing battle against the Mighty Funded Confident Producers.

Any teams that have successfully internalized the notion that they sh ould be externally programmable platforms from the ground up are underdogs—Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it's hard for them to get funding for it because it's not part of our culture. Maestro's funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it's a fluffy rabbit versus a T-Rex. The Docs team knows they'll never be competitive with Office until they can match its scripting facilities, but they're not getting any resource love. I mean, I assume they're not, given that Apps Script only works in Spreadsheet right now, and it doesn't even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.

Ironically enough, Wave was a great platform, may they rest in peace. But making something a platform is not going to make you an instant success. A platform needs a killer app. Facebook—that is, the stock service they offer with walls and friends and such—is the killer app for the Facebook Platform. And it is a very serious mistake to conclude that the Facebook App could have been anywhere near as successful without the Facebook Platform.

You know how people are always saying Google is arrogant? I'm a Googler, so I get as irritated as you do when people say that. We're not arrogant, by and large. We're, like, 99 percent Arrogance-Free. I did start this post—if you'll reach back into distant memory—by describing Google as "doing everything right." We do mean well, and for the most part when people say we're arrogant it's because we didn't hire them, or they're unhappy with our policies, or something along those lines. They're inferring arrogance because it makes them feel better.

But when we take the stance that we know how to design the perfect product for everyone, and believe you me, I hear that a lot, then we're being fools. You can attribute it to arrogance, or naivete, or whatever—it doesn't matter in the end, because it's foolishness. There is no perfect product for everyone.

And so we wind up with a browser that doesn't let you set the default font size. Talk about an affront to Accessibility. I mean, as I get older I'm actually going blind. For real. I've been nearsighted all my life, and once you hit 40 years old you stop being able to see things up close. So font selection becomes this life-or-death thing: it can lock you out of the product completely. But the Chrome team is flat-out arrogant here: they want to build a zero-configuration product, and they're quite brazen about it, and Fuck You if you're blind or deaf or whatever. Hit Ctrl-+ on every single page visit for the rest of your life.

It's not just them. It's everyone. The problem is that we're a Product Company through and through. We built a successful product with broad appeal—our search, that is—and that wild success has biased us.

Amazon was a product company too, so it took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers… if only they could be monetized somehow… you can see how he arrived at AWS, in hindsight.

Microsoft started out as a platform, so they've just had lots of practice at it.

Facebook, though: they worry me. I'm no expert, but I'm pretty sure they started off as a Product and they rode that success pretty far. So I'm not sure exactly how they made the transition to a platform. It was a relatively long time ago, since they had to be a platform before (now very old) things like Mafia Wars could come along.

Maybe they just looked at us and asked: "How can we beat Google? What are they missing?"

The problem we face is pretty huge, because it will take a dramatic cultural change in order for us to start catching up. We don't do internal service-oriented platforms, and we just as equally don't do external ones. This means that the "not getting it" is endemic across the company: the PMs don't get it, the engineers don't get it, the product teams don't get it, nobody gets it. Even if individuals do, even if you do, it doesn't matter one bit unless we're treating it as an all-hands-on-deck emergency. We can't keep launching products and pretending we'll turn them into magical beautiful extensible platforms later. We've tried that and it's not working.

The Golden Rule of Platforms, "Eat Your Own Dogfood," can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later. Certainly not easily at any rate—ask anyone who worked on platforming MS Office. Or anyone who worked on platforming Amazon. If you delay it, it'll be 10 times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for any reason. You need to solve the hard problems up front.

I'm not saying it's too late for us, but the longer we wait, the closer we get to being Too Late.

I honestly don't know how to wrap this up. I've said pretty much everything I came here to say today. This post has been six years in the making. I'm sorry if I wasn't gentle enough, or if I misrepresented some product or team or person, or if we're actually doing lots of platform stuff, and it just so happens that I and everyone I ever talk to have just never heard about it. I'm sorry.

But we've gotta start doing this right.

Become a Hyperink reader. Get a special surprise.

Like the book? Support our author and leave a comment!

V.

Conclusion

If you actually read this whole book, well, color me amazed and impressed. There’s a lot more where these came from, if you’re interested. We only included maybe ten or fifteen percent of all my material, give or take, to keep the book reasonably focused and manageable. You can find my other rants by searching for “Stevey’s Blog Rants” (my blog since June 2005) and “Stevey’s Drunken Blog Rants” (stuff from before June 2005).

If you just skipped to the end to see if I’d included a TL;DR, well, here you go, I guess.

Too Long; Didn’t Read

- There are lots of very different and equally legitimate ways to do software development. And they all hate each other.

- Great programmers are great because they practice.

- As soon as you get comfortable, drop your language and start coding in a new one.

- If you want more than anything to be a manager, you are likely to be a bad one.

- Lisp was hard to master, but it is the only language that continues to make me happy.

- Emacs was hard to master, but it is an investment that will pay dividends for life.

- Step out of your comfort zone. Learn something new once in awhile.

- Build things for yourself. It’s the only way you can be sure it’s the right thing.

- Laugh a lot. It’s healthy, and it feels good too.

- Make sure to laugh at yourself too, but not loudly in public, without explanation.

Please don’t hesitate to send me your thoughts. I’m steve.yegge@gmail.com. Sometimes things can get a little hectic, and I don’t always get a chance to reply to every single email I receive. But I do read them, and I reply to most, and I learn from all of them.

Become a Hyperink reader. Get a special surprise.

Like the book? Support our author and leave a comment!

About The Author

Steve Yegge

Steve Yegge is a programmer and blogger who is known for writing about "programming languages, productivity and software culture." He received a bachelor's degree in computer science from the University of Washington and has two decades of industry experience, developing across domains including embedded operating systems, scalable e-commerce systems, applications for mobile devices, and software productivity tools. In his career he has worked for Amazon.com and Google among others.

About the Publisher

Hyperink is the easiest way for anyone to publish a beautiful, high-quality book.

We work closely with subject matter experts to create each book. We cover topics ranging from higher education to job recruiting, from Android apps marketing to barefoot running.

Hyperink is based in SF and actively hiring people who want to shape publishing's future. Email us if you'd like to meet our team!

Note: If you're reading this book in print or on a device that's not web-enabled, please email books@hyperinkpress.com with the title of this book in the subject line. We'll send you a PDF copy, so you can access all of the great content we've included as clickable links.

Get in touch:

Copyright © 2012-Present. Hyperink Inc.

The standard legal stuff:

All rights reserved. No part of this book may be reproduced in any form or by any electronic or mechanical means, including information storage and retrieval systems, without permission in writing from Hyperink Inc., except for brief excerpts in reviews or analysis.

Our note:

Please don't make copies of this book. We work hard to provide the highest quality content possible - and we share a lot of it for free on our sites - but these books are how we support our authors and the whole enterprise. You're welcome to borrow (reasonable) pieces of it as needed, as long as you give us credit.

Thanks!

The Hyperink Team

Disclaimer

This ebook provides information that you read and use at your own risk. This book is not affiliated with or sponsored by any other works, authors, or publishers mentioned in the content.

Thanks for understanding. Good luck!

Table of Contents

Cover

Contents

Title Page

Preface

Programming-Language Religions

Author’s Note: Tour de Babel

Tour de Babel

Author’s Note: Execution in the Kingdom of Nouns

Execution in the Kingdom of Nouns

Author’s Note: Notes from the Mystery Machine Bus

Notes from the Mystery Machine Bus

Author’s Note: Moore’s Law Is Crap

Moore’s Law Is Crap

Author’s Note: Transformation

Transformation

Author’s Note: Is Weak Typing Strong Enough?

Is Weak Typing Strong Enough?

Code Philosophies

Author’s Note: Software Needs Philosophers

Software Needs Philosophers

Author’s Note: Code's Worst Enemy

Code's Worst Enemy

Author’s Note: A Little Anti-Anti-Hype

A Little Anti-Anti-Hype

Author’s Note: Bambi Meets Godzilla

Bambi Meets Godzilla

Author’s Note: Math For Programmers

Math For Programmers

Author’s Note: Rich Programmer Food

Rich Programmer Food

On Google

Author’s Note: Get that job at Google

Get that job at Google

Author’s Note: Good Agile, Bad Agile

Good Agile, Bad Agile

Author’s Note: Is Google going to stay relevant?

Is Google Going to Stay Relevant?

Author’s Note: Google Platforms Rant

Google Platforms Rant

Conclusion

About the Authors

About the Publisher

Copyright