Three Prongs and a Shelf Full of Parts

Recently, I was in a meeting with another technologist and two managers, none of whom were pointy-haired.  We were working on a problem that, in outline, sounded easy to resolve.  After recapping the problem description (yep, still sounded "easy") the manager who was responsible for the troubled area dropped his recommendation on us: he needed 6 months to research a solution, pilot it, negotiate a contract, get hardware approved and ordered, and implement it.  In the most respectful way possible, the rest of us called bullshit and tried to pick apart the one-page schedule he'd brought.  In the end, while we could have nickel-and-dimed him, the broad strokes were unassailable.

The other technologist left the meeting, spent half an hour between a search engine and our internal documentation store, and scoped a solution by taping together elements we already had on our corporate shelf.  His approach was implemented in a week.

Holy cow!

Collaborative tools -- email and wikis and databases -- make it possible for firms to be well run at larger sizes.  Coase talks about this in his 1937 Nobel Prize-winning paper The Nature of the Firm, and he hadn't seen nothin' yet.

But as companies get bigger, they secrete a nautilus shell of protective practices that slow down projects.  Anyone who has ever interacted with a big company -- working at one, selling to one, servicing one -- has seen the endless paperwork, review committees, and RFPs.  On the face of it, these things suck, so why do companies do them? Companies hate to make the same mistake twice.  (Which reminds me of a darkly funny story about monkeys.)  Companies hate mistakes more than they hate soul crushing paperwork, so they'll erect a mountain of paperwork to head off a molehill of mistakes.

The problem I'm describing lives in the intersection of these two tendencies.  Collaborative tools increase the possible size of companies, but big companies demand lengthy product evaluation and contract negotiations... which raises the costs of getting these tools in the first place.

This is not a completely original line of thought, and I'll pause here to acknowledge a debt to Joel Spolsky who got this far with camels instead of monkeys.  Joel's solution (fly under the minimum-noticeable-mistake barrier) is appropriate to the problem of selling to big companies.  My problem is watching the big company I work for piss away an unaccounted fortune trying to avoid technology purchase mistakes.  So I need a different solution.

The solution I've come up with takes three prongs.  They're not steps, they're not tips, they're prongs, and you can't buy a fork without getting all the prongs at the same time.  That said, you gotta write things in some order, so here's the order you'd want if you were building a brand new IT shop.

1. Stock your Shelf with a few flexible Parts.
2. Know thy Parts in depth.
3. Duct Tape holds the universe together.

Of course, that order doesn't help you at all if you're not building a new shop -- and your team has been around since OS2 Warp looked promising.  So I'm gonna describe them to you backwards, because you have projects you have to start tomorrow, and you're about to stop reading, and I need to tell you something right now that at least slows down the death spiral.

3) Duct Tape holds the universe together.
Lets pretend that you actually have requirements from your customer for that project you're starting tomorrow.  First things first, get a big fat Sharpie and strike out all the places where they describe the solution instead of the problem.  You're the expert here, if you wanted pushy ignorant consumers second guessing you, you'd have gone into medicine.

Look at the requirements again, in that inky sea of redaction.  This has two advantages: (a) it'll be considerably shorter, and (b) it'll be possible to spot places where this problem sounds awfully like things you've already solved.  Maybe it's about tracking documents as they change, maybe it's about working better with distant teammates, maybe it's about shepherding a workflow -- but it's going to sound eerily familiar.

Now try to sketch out a solution as a centaur-style amalgamation of Parts you already have.  "It's like our project management tool, if the data entry was more like our wiki tool." "It's like our free-busy tool, but with the search capabilities of our web-based personnel directory." "It's like every CRUD database application ever written, except with a cornflower blue style sheet."

How could you make that centaur sketch a reality?  We're living in the age of the Mash Up, and even the very worst Enterprise Bloatware seems to come with an API of some kind.  This is where the duct tape comes in. 

I don't mean duct tape in the negative sense, I just mean to evoke a substance that can bind two of anything together.  Of course the binding code needs to be itself robust, documented, and supportable.  It needs to be created by a professional, and cared for as a new Part itself.  After all, the binding is the product that you're producing now, instead of buying new Parts.

But in order for the binding to succeed, you have to understand the two (or more) surfaces that you're bringing together, and that brings me to my previous point...

2) Know thy Parts in Depth
For every Part you're taping together, you need an expert that knows, in his bones:

• What is this Part excellent at?  What is it bad at?
• How big can this Part scale? 
• What demands does this Part place on supporting systems like storage and network?
• How do I make this Part fault tolerant?
• What APIs does this Part have?
• How much care and feeding does this Part require?  Is it brittle?

These are all deep questions, and they're not answerable by scanning the Part's web site.  Some of them would be answered in professional training, but some of them can only truly be grokked by a live-in expert.  A Part without the tender loving care of a full time expert is not likely to be strong enough to be a vital component of what you're building.

And that leads to the first prong...

1) Stock your Shelf with a few flexible Parts.
It's not enough to say "we buy the most excellent parts that meet our budget."  Everybody does that.  In order to have the expertise that you need in Prong 2, to build the solutions that you're dreaming up in Prong 3, you have to have a manageable few Parts.

How few?  Well lets start with the maxim "One tool for each job."  If you have several different Parts that perform exactly the same function, migrate all the responsibility to your favorite, and kill all the others.  Even MBA's know this stuff; it's an unalloyed good and you'll get the easiest bonus of your career.

The hard part is when you have two tools that do similar jobs, or a tool that is excellent at one thing and mediocre at a jillion other things.  I agree with the MBA's that there is inherent value in having fewer tools, but I say that out of love for engineers, not Cost of Ownership.  Fewer tools will let the same staff achieve greater mastery.

But under what criteria would you give up (or avoid in the first place) an excellent point solution for a good-enough extension of a system you've mastered?  For one thing, consider how completely your expert believes he can cover the true requirements, at this time.  Resist the urge to buy a point solution that can do really shiny things that you don't need right now.

Flexibility is also an important criteria for picking a generalist over a specialist.  A good Part is broadly configurable, with API surfaces that are easy for your duct tape to adhere to.  Over time, inflexible specialist systems will always be overtaken by programmable generalist systems, it's a question of speed versus acceleration.  Moreover, picking the flexible Part today increases the breadth of tasks you can take on in the future with Prong 3.

Why is this a good idea?

The chief advantage of this approach is to speed up the delivery of solutions to new problems.  Taping together Parts already on your shelf does not wake your big company's fear of risk, and it flattens your learning curve.  This is where the money is, this is how you sell it to your boss.

A subtler good is that you create more experts, instead of a legion of mediocre staff looking after a gob of systems they don't deeply understand.  Experts are more edifying to work with, and they breed more experts.

Also, solutions built in this way get designed explicitly to solve the problem at hand.  They're not grown in some focus group, they're not jealous of market share or cannibalizing existing products, they're here to serve you.  They take on some of the flavor of Situated Software, and that's got its own set of virtues.

I'd also like to make explicit that the Parts you choose should reflect the character of your team.  Maybe your shelf contains Twiki, Zimbra, and MySQL; but it could as easily be SharePoint, Exchange, and SQL Server.  I dig Open Source, but it's not fundamental to this model.

So try it out!  Especially the Sharpie on the requirements doc, that's just good clean fun.

Yahooisms

Where did all these words come from?
At Yahoo!, there is an internal mailing list called 'devel-random.'  This list receives hundreds of messages a week from Yahoo! employees, on topics ranging from technology to company politics to global politics, and is read by an equally diverse population, including developers, project managers, marketers, and both founders.

So when I took a Linguistics class from Stanford's CSP program, I decided to use a copy of this list's archives I had laying around: 8,000+ messages sent between January 31st and May 28th, 2008.  I sketched up a script in PHP that downloaded these emails out of my Exchange account, split them up into words, then threw out all the words that matched the GNU Aspell dictionary.  The rest of the words I reviewed using an AJAXy front end, throwing out obvious misspellings, proper nouns, and bits of URLs, but providing me with enough context from the original message to spot the diamonds in the rough.  What's left were interesting enough to get a class paper out of.

If you are a developer or an IT nerd (like me!) a few of these are going to seem old hat to you.  Several of them were already in off-the-beaten-path dictionaries like the Hacker's Dictionary or the Urban Dictionary at the time of my project.  But hopefully you'll find something new to you, or bent into a new use, or maybe just a better definition here that helps you communicate with your "normal" friends.

So, what was interesting?
I’ll break down the interesting words into three categories.  The first is jargon, words that add detail or speed up conversation in our areas of expertise.  The second are terms that I think have the potential to spread beyond technologists and into mainstream usage.  The third are words used when talking about the generic use of lower-case "google" to mean "searching the internet" without necessarily referring to the Google brand (which, as you might imagine, really pisses Yahoos off).

Jargon
Dogfood (noun or verb) – This term has its origin in the expression “to eat our own dog food.”  In technology, this means “using our own products.” Dogfooding is, for programmers, a joyous milestone, because it means the product is usable -- at least by someone predisposed to see past the current bugs.  The term is also open to parody, since the expression literally means people are consuming something not fit for human consumption.

Mashup (noun), mashable (adjective) – Both these words came into my experience through the hack culture at Yahoo.  A mashup is an application that uses data or functionality from two existing applications, usually applications that you did not write and whose development you do not directly influence.  Douglas Crockford, a programming language architect at Yahoo, says “Mashups are the most interesting innovation in software development in decades.”   Mashable is a complement, meaning “lends itself to mashups.”

Spidered (verb) – Meaning “collected by a spider,” this usually comes up in context of things that search engines are capable of finding, as opposed to content that is deliberately or accidentally not visible to search engines.  Spider is a metaphorical term for the web crawlers that find content for inclusion in search engines.

Threadsafe (adjective) – A threadsafe program is one that is demonstrably error-free when several copies run simultaneously and share some resource, like memory. It is a fairly recent development for consumer-grade hardware to support this possibility, but it's not widely implemented in software.  Rasmus Lerdorf, creator of the PHP programming language has said "Most humans are simply not smart enough to write threadsafe code." 

Migrated (verb, passive voice) – The interesting nuance that I credit to the technology industry is the passive voice usage.  In general usage, the person or animal that migrates is the actor; in technology a user can be migrated, without their involvement or consent, between technologies or products.

AJAXy (adjective) – This term stems from the programming trick “Asynchronous JavaScript and XML” that powers modern web tools like GMail and Flickr.  The adjective form can either mean “literally uses AJAX” or more figuratively “showy, especially but not necessarily by using AJAX.”

Freetard (noun) – Refers to someone who conflates free as in open source and free as in zero cost; a freetard undervalues other people’s work, and demands that everything delivered digitally cost zero dollars, regardless of whether the creator wishes to participate in open licensing or not.

GA'ed (adjective) – A "GA release" is the milestone that makes a product "Generally Available."  This form, pronounced “gee-ayd” effectively means “made generally available.” 

Folksonomy (noun) – A taxonomy built by users.  Usually this implies that a rigid hierarchy will not be imposed, and that one item may live in many categories.

Mainstream Potential
Betamaxes (noun) – Used as a general term for any dead end technology, especially technologies that had strong vendor backing, but were killed either by user preference or published content.  The term primarily applies in winner-take-all fields powered by network effects: as betamax slipped in popularity, the tapes weren't stocked in stores, which made consumers not want the players, which caused a negative feedback loop.

Frankensteined (verb) – To put something together out of existing parts, but with a strong pejorative tone.  Frankensteined has the added connotations of shoddy workmanship, poor performance, and unmaintainability.  (Contrast with mashup.)

Offlist (adverb) – Compounding of “off of the list.”  E.g., “I will reply offlist.”  I suspect it owes it’s shape by comparison to “off-the-record,” and it tends to be used in either the “I don’t want to bother the list with this” or the “I would rather keep this private from the list” sense.

Spim (noun) – Spam delivered by instant messenger instead of email.  This appears to be one of a series of refinements on the general term spam, which also includes spit (Spam over Internet Telephony) and bacn. 

Spamfighting (gerund) - The act of fighting spam, by comparison with firefighting.  I like that his gives a heroic nuance to the forces arrayed against spam.

Verbing Weirds Language
Verbiness (noun) – A measure of how well a noun lends itself to unambiguous verb conversion.  Yahoo is considered to have low verbiness, Google has high verbiness.  The jury is still out on the verbiness of "Bing."

Kleenexification (noun) – The transition through which a brand name comes to stand in for a generic activity or product.  Like betamaxes above, using this example to stand in for a whole class of activity evokes the whole lore of the specific example.

Genericize (verb) – To make something generic.  This also implies “to erode the claim to trademark.”

How I discovered that I was building Situated Software

I'd been creating “situated” software for months before I first stumbled on Clay Shirky's article, which gave a name to what I was already up to. You should stop here and go read the original article, in part because he's a captivating writer, and in part because I've been subsequently marinating in these ideas for more than a year now, and some of what I found earth-shaking on first read I will have forgotten was novel enough to require explanation.

http://www.shirky.com/writings/situated_software.html

I wrote what follows to explore how situated software changes the design landscape at my day job: writing software inside of Yahoo's IT department. In some sense it's a situated essay; there are assumptions I get to make, working at a company in America, in that odd part called Silicon Valley, with a significant portion of my user base being other software authors – I don't think anybody is going to pick this up and learn much that's immediately and unequivocally applicable to any situation other than mine. But that's okay, even the exercise of thinking about “that wouldn't work here” is still helping you to explore what situated software would look like in your situation.

I would boil “Situated Software” down to “software that doesn't make sense outside of the situation it was designed for.” For contrast, think about Microsoft Word. It's hard to imagine a typed document that wouldn't basically fit in Microsoft Word -- it's as applicable to school papers as it is to filing for bankruptcy or creating Star Trek fan fiction. Microsoft pays a legion of programmers, testers, linguists, and sociologists to make sure Word is as broadly useful as possible; I expect this is why my seventh grade class took a mandatory course about Microsoft Word: whatever our future vocation, Word would be an acceptable (if not ideal) tool.

So if Word isn't situated software, what is? I've been working on two applications that have benefitted from a situated state of mind, and I'll be mining these for examples and anecdotes. One of them is a web front-end for an enterprise phone system, that we call Yahooized Asterisk (Asterisk is the open source phone system that underlies the custom front-end). The other is a web-based project management tool, named Smithers. Part of what's interesting about these examples is that you can buy non-situated software to meet both of these needs, so I'll be explaining where my software is superior, and where (hopefully less frequently) it falls short.

Barrier to Entry

Because both systems are designed to only be used by employees, I can assume that any user will already have an account in Yahoo's single-sign-on solution for employees. Users get a single ID and password that grant access to dozens of situated (and retro-fitted commercial) apps; all these applications already know who the user is without the sign-up speedbump.

Pre-Populated Data

Yahoo, like most large companies, knows volumes about its employees; phone numbers, org charts, geography, mailing list membership, etc. Yahooized Asterisk, for example, looks up all incoming caller ID numbers against the corporate phone directory; if it gets a match, your phone shows the caller's name in addition to his number. (Think of the contacts on your cell phone, but with 20,000 entries, kept rigorously up to date by human resources and the telecom team.)

Smithers uses information from the org chart to figure out which managers to keep in the loop on which projects. For example, if A has a project that is sponsored by a senior vice president D, Smithers knows to keep A's boss (B) and boss's boss (C) in the loop. If C quits and is replaced by C', Smithers learns about it from the HR database, and stays up to date.

A less utilitarian but more personal example in Smithers is the user's profile picture. This is a pretty common feature of consumer software; LinkedIn and StackOverflow and eBay all have it, but most people seem to keep the (ugly, useless) silhouette default. I actually don't recall ever seeing this feature on business software, which is a real shame, because inside a big company seems like one place you could actually get a high attach rate. At Yahoo, on your first day, we take your picture for your security badge, then upload that picture into a web service (which Smithers and other situated apps consume). If you don't like your mugshot, you can replace it with something more flattering, but even the default is meaningfully you.

Humor and In-Jokes

The whole point of situated software is to be more responsive to fewer users than commercial software. But situated software can represent more than a better-tailored feature list, it can participate in a community's culture. Situated software can even have a sense of humor.

For example, this is part of the dialog that greets a user who hasn't set up his voicemail pin in Yahooized Asterisk:

This PIN is all that stands between you and evil monkeys who want to listen to your voicemail and forward your calls. Our crack monkey defense researchers suggest these minimum PIN strength guidelines:

• the PIN must be all numbers
• at least 4 digits long
• not the same as your extension
• not something a monkey would guess (e.g., 1234 or 9876)

The factual part of the advice is pretty standard for any voicemail system.  Inside the company I can inject a little fun into it, without worrying about what Joel describes as "one old lady in Minnesota" getting bent out of shape about my monkey-incorrectness.

Smithers is named after the loving assistant to the evil Mr. Burns on the Simpsons. Late projects get a tiny icon of the Simpsons' character Krusty the Klown. The user who originally suggested this feature meant it as “your project is making you look like a clown” – a little bit of negative reinforcement to improve behavior. In the jargon of the users, it's now called “Getting a Krusty.”  I'm just grateful that isn't something horrific in Urban Dictionary.

Taking a cue from the Treehouse of Horror, Smithers even had a special Halloween edition. The project status icons changed from boring dots of red, yellow, and green to blood, pumpkins, and frankenstein icons. Even users names were updated in the Simpsons' tradition of “scary” names in the credits.  The change got a lot of buzz, and a lot of goodwill for the app. More importantly, users submitted 40% more updates than the day before.

Marketing

So if everyone can use these systems, how do they learn that they should? For Smithers, the key was having a patron. When he told the managers that report to him to use Smithers, they passed the edict down to their employees. So Smithers got a big push from a top-down directive, instead of relying on things like excellence to entice users. It's what happened next that I'm more proud of.

Smithers is not the only mechanism Yahoos use to track projects. Some teams use spreadsheets or whiteboards or swarms of sticky notes; others use Microsoft Project, or Serena Mariner, or VersionOne. The only way I could imagine that I could improve on all of these was to tap into the community that needed to use the tool.

One way I did that was to work with several power users: employees who would be generating a lot of input, and mangers who would need structured output. I set up a mailing list to discuss design decisions, drew paper prototypes with users, and even posted the development schedule. Between meeting their needs, and the leavening of humor I mentioned earlier, Smithers became popular with the users, which is great because they didn't actually have a choice in whether or not to use it. But because users liked it, they used it for more projects, and updated it more frequently.

At this writing, Smithers is almost a year and a half old, and 111 users have created 1,934 projects and submitted 13,712 updates.  The original patron has long since left the company, but Smithers is still going strong thanks to the user-generated momentum.

Too Much Flexibility, or Not Enough?

If the great virtue of situated software is that it can become whatever its situation requires, the great tragedy of situated software is that there are always more user needs than developer hours. The whole Yahooized Asterisk system originally grew out of the idea that 95% of Yahoos are using a tiny sliver of the functionality of our commercial phone system. So we decided that if we could replace our expensive commercial phone system with an open source one, and really kick ass on the few “essential” features, we could save a boatload of capital, and actually improve the service. And so far, we've delivered on that vision, for 95% of our users.

But that last 5% is a doozy. First off, that small fraction is actually a big number in absolute terms; 5% of 13,500 full time employees represents 675 not delighted people. And aside from not being happy with my software, they really don't have much else in common. Some are call-center managers, who want better sticks to beat their call-taking zombies with. A few dozen are secretaries and assistants that want to be able to field their bosses' calls, in scattered pockets that look more Madison Avenue than Silicon Valley. One guy wants to use the corporate long distance plan when he's making business calls from his personal cell phone, like a calling card. The M&A team wants a super-secure conference bridge.

This is, of course, not unique to situated software, this is general to design tradeoffs in every field. What is unique is that we don't have cash as a feedback mechanism, and we do have an amplified sense of who the squeaky wheels are. In a commercial software shop, I'd probably rank those features by the number of people who want them, and the price-per-head I thought I could extract for that feature, and complete them in that order. Missing that, I can be guided by politics or the features that are exciting to me.

So where does that leave mass-produced software?

To be clear, I do think that there is still a place -- lots of places -- for mass-produced software. The underpinnings of both Smithers and Yahooized Asterisk are general problems, like databases and web servers and phone systems.  The top level tool is situated, but we're standing on the shoulders of giants like Apache and YUI and Asterisk.

Maybe what's changing is that the unsolved problems in software are getting closer to the promise Hal Abelson lays out in the first lecture of Structure and Interpretation of Computer Programs. If software is actually about formally expressing a process, then it shouldn't be surprising that a company would express something so culture-specific as “how we run projects” in their own image, rather than trying to find something close.

I expect that software will continue to get easier to write, and that this will encourage more organizations to retain staff that can create excellent, custom software in increasingly narrow situations. I think eventually software creation will be mostly an exercise in codifying requirements, and communicating those to sympathetic tools. As we get closer to those tools and that understanding of software, I expect a larger percentage of new software to be situated.