If you ever learn about optimization, the second technique they teach you after “hill climbing” is something called “simulated annealing“. Forget trying to work through Wikipedia’s definition of it, here’s how I mentally visualize it:

Imagine you your job was to find the highest point on a random Frank Gehry designed building, say the Hotel Marques de Riscal. The problem is, you’re blind so you can only determine the height of one point at a time.

Hotel Marque de Riscal

One way to do it would be to take the entire outside shell of the structure and turn it upside down and then drop a ball bearing in it. When the ball bearing stops moving, you declare that point to be probably the lowest point.

Upside Down Hotel Marque de Riscal

In the field of optimization, this is called hill climbing and it sure is fast but the problem with it is obvious: the ball bearing will likely drop into some rut on the side of the building before reaching the bottom (aka: a local minima).

Simulated annealing is like adding a giant paint shaker and imagining the ball bearing getting heavier over time. At first, the ball bearing is light as a feather and every random vibration is going to make it bounce around, sometimes to a lower point but also sometimes to a higher point. Over time, as the ball bearing gets heavier, it becomes progressively harder to jostle out of it’s rut but when it does move, it moves to a lower rut some distance away. Finally, the ball bearing becomes so heavy it’s virtually impossible to budge, at which point, you declare that position to be probably the lowest point. Simulated annealing can still get trapped in local minima but it usually does a hell of a lot better than hill climbing.

Now, imagine that the ball bearing is society and it’s searching for the solution to a problem like, say, attaching sheets of paper together. If you’re living in any time since the 1930’s, it’s overwhelmingly likely that you’ll be reaching for a “Gem” style paperclip.

A Gem style paperclip

What many people don’t realize is that the Gem paperclip wasn’t the only type of paperclip invented. In fact, the 1909 Websters Dictionary entry on paperclips referred to the “Konaclip” version which was considered the prototypical paperclip of the time.

A Konaclip style paperclip

From 1864 when the first paperclip patent was granted, till 1930 when the gem became dominant, there were literally dozens of designs for paperclips (which, thankfully, has been documented on the Paperclip section of the Early Office Museum). But since the 1930’s, technological progress in the paperclip arena has ground to a halt with the entirety of society standardizing on the Gem and every other style dying in obscurity (For a much more detailed look into the paperclip’s fascinating history, I refer you to the book The Evolution of Useful Things).

To explain this history, I refer back to the simulated annealing analogy. In the early days of paperclipdom, society wasn’t particularly attached to any one design so there was very little momentum in the system. Even a tiny vibration was enough to make another design viable. However, as time went on, certain styles of paperclips started to have legacy effects and it became a race between a few, select alternatives, most notably the Konaclip & the Gem. Finally, society had settled so firmly upon the Gem that it it would take the most extraordinary effort to have any other design supplant it which is why it remains the paperclip of choice to this day. In Simulated Annealing terms, the entropy was now so low that it became practically impossible to escape from a local minima.

Indeed, one of the characteristics of simulated annealing is that it goes through 3 distinct phases. A period of fluid, diverse shifting, a long period of stagnation punctuated by occasional radical shifts and then finally stability from which only incremental improvement is possible.

Looking at the pattern of historical developments of other technologies, it’s possible to spot these same shifts in between phases.

Before 1973, there was a diverse ecosystem of physical interaction paradigms for desktop computing. The most notable examples from that era were Ivar Sutherland’s Sketchpad using light pen interaction and Alan Kay’s tablet style Dynabook.

Sketchpad circa 1963

Dynabook circa 1972

Then, in 1973, the Xerox Alto was released and virtually halted the progress of computer interfaces from that point forward. Nearly every desktop computer today is a recognizable descendant of the Xerox Alto.

Xerox Alto circa 1973

The list of genuine innovations in physical UI that have been adopted since the Alto are as laughably few in number as they are trivial in scope. In rough order of importance: Speakers, microphones, webcams, the mouse wheel, wireless networking, higher resolution screens, a numpad, flatter screens & the Windows/Command button (this is not a sampling, it’s the complete list).

The list of potential innovations are as staggering as they are futile: Pen computing, Tabletop computing, Tablet computing, Augmented & Virtual Reality, Touch based interaction, Multitouch, MultiMouse, Bimanual Interaction, Single Display Groupware, 3D displays and the list goes on and on. In the last 37 years, billions of dollars have been poured into these alternative technologies in a vain attempt to supplant the Xerox Alto as the dominant paradigm with pretty much nothing to show from all that work except a bunch of pretty pilot projects.

And with each passing year, it’s become more and more difficult to foster a viable alternative paradigm because more and more gets invested into keeping Alto style computing firmly entrenched. Software is designed for a keyboard+mouse+screen, people have invested time learning the intricacies of a WIMP OS and thousands of companies have a vested interest in keeping the system entrenched.

The same pattern can be found in keyboard layouts with QWERTY, programming languages with C, web standards with HTML+CSS+JS, Office Productivity with Microsoft Office, Email with Eudora and so forth. In every instance, a period of rapid innovation was brought abruptly to a close with a dominant technology and, from that point forward, genuine changes in the status quo happen at best every decade and only after extraordinary effort *.

Almost every discussion of a new innovation focuses on the details of the innovation almost exclusively without considering the broader social context. Such discussion manages to miss the vital point that adoption of an innovation depends only a tiny bit on the actual innovation and almost completely on the current progress of society. Once a technology has reached a certain maturity point, whatever local minima that society has currently reached will establish itself be the dominant technology until the end of time regardless of what technological progress has occurred or has yet to occur. Like a game of musical chairs, it’s all about when the music stops. If you’ve managed to grab a seat at the table, then you’ll stay there forever, if you missed your spot, then it’s going to take extraordinary effort to regain it, no matter how much of an improvement you are over the status quo.

* This same model also explains why European & Japanese mobile & broadband technology is so far in advance of the US even thought the US were the pioneers of both technologies. Because the US started early, it’s industry matured around an earlier technological & social standard which caused it to fall behind the less mature, more technologically limber peers.

A few days ago, I allowed a friend of mine to post a guest post titled Why women don’t understand friendship. So far, the reactions to the piece have been universally negative with “bitter”, “misogynistic” and “offensive” being a fairly representative sample of words used to describe it. I can’t say that I’m particularly surprised by the reaction. So given that, why post it?

Lets start with what were not the reasons I posted it.

It was not because I agreed with him. I differ with Aug Bohr on several substantial points and his conclusion has not managed to convince me.

I did not post it an a deliberate attempt to offend readers or drum up controversy. Yes, there was plenty to be offended by in the post but this was a side effect of the argument being made.

I posted it because it was heresy and, for the ideals of the free exchange of ideas to flourish, heresy must be allowed to stand on it’s own merits.

The free exchange of ideas is not a natural condition. We have fought to establish it and it’s existence is fragile at best. Heresy is where the battles of intellectual freedom are fought. Where heresy is allowed to stand, intellectual stagnation follows. Taboos become entrenched & advancement becomes more about adopting the “correct” position than the true position. The only way to prevent this intellectual stagnation is to become virulently allergic to judgment based on ideology and a dedication to focusing on content alone.

But humans, in their natural state, are supremely uncomfortable with heresy. The reaction, when confronted with an argument that you violently disagree with is to try and make it false as quickly and efficiently as possible. Oh, the author must be a bitter, misogynistic hack. Oh, the standards for evidence were not met. Oh, it relies on a poor grasp of evolutionary psychology. Anything to make the argument go away. This is not acceptable. Look past the ideology, evaluate it like you would evaluate any other argument, hold it to the same standards. Don’t flinch.

“Why women don’t understand friendship” isn’t a great argument. But it’s also not a bad one. It’s intriguing and also presents a perspective that I had never heard before. I posted it because if it had been on any other topic, I would have considered it worthy of posting. To have not posted it would have been a betrayal of an ideal that I hold dear.

This is a guest post by a friend who currently wishes to remain anonymous. Not everything in this post is reflective of my own personal views but I do believe there is something both provocative & interesting contained within it.

Friendship is a peculiar type of relationship. It has no formalization of status, no hard and fast beginning or conclusion. Even more strangely, a search through your memory for the segue from acquaintanceship for a specific friend will draw an amnesia-like blank. Yet even as it seems so elusive, it’s the strongest, most critical type of connection humans can forge with one another.

Can men and women be friends? No, but not for the reason that immediately comes to mind. Latent sexual desire is not the fork in the gears of an intersex friendship. It may be the scapegoat, but what is really at play is a woman’s incapacity to be friends with anyone, including another woman.

I will argue that fully half the population has no conception of friendship. It does not exist for them in the same way that childbirth does not exist for men. Instead, they are observing and approximating what men do, going through motions and playing along. Before vehemently coming out in favor or against this conclusion, consider the following.

Men are worthless. Not rhetorically worthless or metaphorically worthless. Actually worthless. You could toss 95% of the world’s men into a meat grinder to make hotdogs and the birthrate and overall quality of life of everyone would be unaffected. A single rich man can father and support hundreds of children in a single generation. Reducing the number of women, however, drops the birthrate and the desire and ability to care for children by that much.

Though the realization is not a conscious one, men understand it on some level, and their life is a continuous battle to avoid irrelevance. A man cannot create life, and due to his fast recycle time in the part he does play in creating life, another man could easily step in and take his place. This so defines man’s existence that the inability to ejaculate when the woman desires it has been medicalized, a subtle hint to his innate worthlessness. To keep his genetic destiny from disappearing because a better man could spare fifteen minutes and the equivalent of his salary, men struggle to assert their worth.

Monogamous marriage benefits the common man over both women and wealthy men. If Bill Gates were allowed to take as many wives as he could support, thousands of poorer men would be left out in the cold. Certainly not all women would accept a generous stipend and fatherly non-interference over the alternative, but that is not required for the numbers to work against the common man. Men agree to take only one wife so that everyone can at least have the one.

Men are the architects of civilization, the government leaders and founders, the discoverers, the inventors, the explorers. Not because they are better at these things or more suited to do them or even because they’re bullies. They do these things because if they don’t, they are worthless. Their life is a constant struggle to prove they are more valuable alive than put through a meat grinder. When women do undertake these endeavors, they often have above average level of testosterone, the chemical which seems to engender this sensation of worthlessness.

Men have always fought the wars, built the buildings, and hunted the game. These are activities which can only be done with a uniquely intimate cooperation. To this day, managing a war or construction project is an extraordinary problem not only in implementation but in trust. Men tens of thousands of years ago accomplished these tasks with no oversight and minimal communication. It was a remarkable feat, perhaps facilitated by a new form of intimate connection, friendship.

There is no need to define friendship in terms of what it is when we can define what it does through group selection. It is simply a type of human relationship which, when experienced by members of a tribe, allowed that tribe to outdo its foes. A tribe with friendship will always defeat a non-friendship tribe in a raid. A friendship tribe will return with more food from a hunt, will always build a shelter faster in a downpour. This was all that was required for us to become descendants of those with the capacity for friendship.

Women did forage for food, which actually provided most of the tribe’s sustenance. Foraging, however, does not benefit from cooperation in the same way as a three-day hunt. These men had to sacrifice for each other, even give up their lives for each other in raids, and to understand each others’ signals and intentions. This was equally true in war and construction. Women had no comparable activity for which cooperation of this degree was necessary. This, however, doesn’t explain the mystery of why women don’t possess even a vestigial sense of friendship. That is, unless the ability to sacrifice for one another actually handicaps their pursuit of genetically high quality children.

It does. Women primarily competed to get the best man, and men competed to get the best woman. However, the cheapness of sperm meant men had no motivation to ration it, whereas eggs, if successfully fertilized, mean at minimum a yearlong investment. This is what forces men into the role of putting on displays while women sit back and choose. While women stepped over each other to get the best man, men’s displays further improved, in the form of civilization, something that can’t be created alone.

You can witness this to this day when you go to the bar. A woman will sabotage her “friend” if that friend is hit on by a man she’s interested in. Even if she’s not interested, she’ll almost instinctually sabotage her. The wingman practice only exists because these women are only friends so long as a man does not come between them. Men must be such good friends because women are natural enemies, no matter how long they’ve known each other or how great a display of friendship they advertise.

And it is women doing all the advertising. “We are such good friends! Jenny is my best friend in the whole world!“ Men never speak of friendship, and they especially do not rank who their best friend is. For men, the concept is so fucking visceral, so tangible, that blathering on about who is and who isn’t a friend would be like talking about the ground they walk on. “Isn’t the ground great? This is the best ground!” It’s either ground or it isn’t. If you’re walking on it, it’s fulfilling its purpose, and its purpose is entirely encapsulated by the term “ground.” The only motivation to make noise about such an entity would be for exhibition. It does not exist for you, and you are trying your best to approximate something you see others partake in so naturally.

When pressed to define it, women display an almost infantile conception of friendship. A woman’s friends are the people she talks to often, who make her feel good, who she has things in common with. In other words, frequently-seen acquaintances. A five-year-old would list similar criteria. When women do challenge the friendship of two men, their questions betray their conception. “If you’re such good friends, what’s his favorite color/middle name?” Knowing details about someone has as much to do with friendship as the brand of a bottle of water has to do with whether it’ll keep you from dehydrating.

Women may claim their best friend and mate are one in the same. A woman does tend to absorb her mate entirely, going as far as integrating his preference in sports teams into her own identity, but this is an unrelated type of connection. Joining with someone so closely that your goals and future blend together is a truly deep connection, and one women are certainly capable of, but it is at the expense of friendship, not as a shining example. Women who follow the absorption pattern of pair bonding almost always require complete compliance from their friends. Anyone who questions her relationship with her mate is cut loose without hesitation. It is almost as if women have the capacity for exactly one relationship. If her mate emotionally drifts away 20%, she’ll reach out to someone to fill the gap. Once he returns, the new fill-in is placed in cold storage.

If what women call friendship is in fact acquaintanceship, surely an experiment can be designed to reveal this. How one understands exclusively online friendship is an excellent litmus. Talking to someone online allows intimacy, commonality, superficial shared adversity, and frequency of communication. It does not, however, allow friendship. An online acquaintance never has the chance to give you their last sip of water on a long hike or risk life and limb to save your child from a lion. If you have no conception of actual friendship, the objection you’re now having is, “Someone can’t lie forever. Eventually you do get to know them and what kind of person they are.”

This misses the point entirely. The issue is not what kind of person they are; it is that entire facets of their personality, both positive and negative, cannot express themselves without a real life connection. Online friendship may allow more commonality, honesty, and understanding than real life, but these are extremely low on the Maslow hierarchy of friendship. An artificial intelligence could supply these while being completely unconscious and lacking all integrity and character. Who knows, your online friends may in fact give up a finger so you can keep your hand, but without the opportunity for this to happen, it is irrelevant. This defining facet of friendship is completely shielded by lack of real life interaction. For this reason, men will overwhelmingly say that friendship cannot exist online, and women will say it can.

Evil Geniuses are so rare that it’s still surprising when one actually surfaces. In this case, the plot is so deviously brilliant I can’t do anything but heartily applaud it. I’ll let The Guardian take over from here:

The women were led to believe they were being filmed for a Big Brother-type television programme, according to the Dogan news agency and other news reports. Instead, their naked images were sold on the internet by their captors.

They were made to sign a contract which stipulated that they could have no contact with their families or the outside world, and would have to pay a fine of 50,000 Turkish lira (£20,000) if they left the show in the first two months, the agency reported.

She said the women were not abused or harassed sexually, but that they were told to fight each other, to wear bikinis and to dance by the villa’s pool.

Does any other discipline have problems establishing the basics of competency in the same way that software engineering does? Starting from Imran, to Reginald Braitwaite, to Jeff Atwood and now to Richard Banks, the story of FizzBuzz has become mildly famous among the circle of software programmers. What is it about the profession of programming in particular that allows for such stunning incompetence to be masked for so long?

Are there professional athletes who are unable to walk? Are there professional accountants who cannot do math? Are there professional postmodern english lit professors who occasionally let slip with a comprehensible sentence? I’d like to have some insightful insight at this point but the sheer lucridity of it has me baffled.

I’ve mentioned previously that the reason this company is named Bumblebee Labs is from a term I coined:

A bumblebee is an occurrence which cannot be explained by our current theory and thus, demands special attention. Bumblebees are the keys to uncovering areas where our understanding of the world drastically fail and how we can construct a better theory to explain what is happening.

and the FizzBuzz paradox is a Bumblebee of the highest order.

Just once, I’d like to hear the FizzBuzz story from the other side. I’d like to hear a veteran software engineer, confirmed to have been working on multiple projects and with largely positive references explain how he managed to fool so many people for so long about his basic competency in programming. Does this person know that they’re a bad programmer or are they suffering from a severe case of Dunning-Kreuger? How does knowing that you’re a fraud every single day weigh upon your psyche? How do your team mates treat you?

Sure, there are many other fields in which blatant frauds continue to go unexposed for many years: art, academia, holistic medicine, cult leaders etc. But everyone knows that fraud is endemic in these fields. Sure, there are fields in which people are completely self delusional about their ability: music, writing, acting. But everyone else knows those people are crap and never let them do anything of significance. Software Engineering shouldn’t be like those field. It should be like every other respectable, solid career. And yet it’s not.. why?

One of the hardest things about writing this blog is the feeling of talking into a void. I know there are people out there reading it because I can see the analytics data but it still feels lonely writing. So I’m going to experiment with a semi-regular feature where you, the readers respond.

Post a comment with how you found out about this blog, why you read it and also if there’s any particular area you’d like me to cover more.

As an added incentive , I’ll be donating 25 cents to charity for every person that responds. If you feel particularly strongly about any one charity, put that also in the comments.

The Dream Job is so simple I’m wondering why I’ve never heard anyone propose it before.

I’m going to start off describing The Dream Job by the most inconsequential and most easily changed details because, well… you’ll see.

The Dream Job pays one million dollars per year and only one is offered every year. Every application for The Dream Job must be made public. Once you’re hired, the only power the employer has over you is firing you. That’s it, those are the only overt constraint for The Dream Job, everything else flows from there.

You don’t apply for The Dream Job by sending in a resume. Well… you could but you’re not likely to get it. There’s only 1 Dream Job a year, you need to dazzle. The Dream Job is only for people who first of all love that company very very much but that love is tinged with a deep channel of ambivalence. It’s for people who are driven with the desire that this company, while great, could be so much greater if they could only fix this one thing and you are the right person to fix it.

I would be deeply, deeply tempted if Newegg offered The Dream Job (There are a half dozen other companies with which I would apply for a Dream Job without hesitation but I’m holding those closer to the chest as they’re related to stuff I’m actually working on). Classic usability isn’t even my field anymore but The UI behind online computer buying has essentially remained static since the mid 90’s and every time I want to purchase a computer online, it makes me deeply angry that the user experience is still so poor from a pure usability standpoint. I have so many ideas bursting in my about how you could revolutionize the user interface to make it orders of magnitude more productive and fitting with the tasks that people have. Give me a week to prepare and an hour to present to Newegg and I’m utterly confident I could convince them hiring me at a million dollars a year would be a bargain. Here’s a simple idea Newegg: Build a braindead reliable text parser so that I can paste the shopping cart from any other web store and you can tell me how much identical or similar items would cost at Newegg. Why should price comparing systems be a laborious half hour of hunting for equivilant components on multiple sites when some simple engineering could reduce it down to seconds?I managed to come up with 8 other ideas inside of an hour before I got bored at how easy it was.

The Dream Job is not for everybody. The freedom offered by it sounds alluring but when you consider the full implications of it, can also be slightly terrifying. Once you get in, you can do literally whatever the hell you want but that also means not a single person will actually know what you do until you sell them on it. By definition, you’re hired to do something at that company that’s never been considered before so you’re starting off with nobody obligated to give you the time of day. It’s up to you to build up the support within the company and selling people on your vision or your mission is dead before it’s even born. There’s a reason why a company full of otherwise smart people hasn’t been able to see a problem that’s so obvious to you and it most probably stems from a complete difference in cultures. You need to be not only a visionary but also an anthropologist and a translator. On top of that, you need to constantly justify your $1 million dollar expense or you will be swiftly canned. The Dream Job requires not only brilliance and passion but also deft people skills and the ability to work around showstopper obstacles.

So, given all this, why one million dollars? Simply because, even in this day and age, one million dollars still means something. It still has that allure when those crisp syllables roll off your tounge. The actual number is meaningless, mostly symbolic, and the bargain of the century to boot. Anyone that brilliant willing to work for one million dollars a year is clearly not in it for the money. Instead, the requirements for The Dream Job are simply the natural result of the observation that hiring only for the skills you know you need is a rather stupid way of doing things.

The conventional way of hiring is you first figure out what resources you need, how much you want to pay them, where they slot in the org chart and then you search for a candidate. This was great when your grandparents were busy climbing the corporate ladder but why don’t we shake it up a bit. How the hell is a company supposed to know what it needs anymore? If you’re a process nerd, then you’re going to hire a bunch of other process nerds and build a great process nerd company culture but how can you ever know what you really need is a deep design aesthetic as well? Similarly, if you’re a design person, how are you going to find out how an obsessive A/B tester can transform how you build? The simple answer is that you never will with a conventional hiring model. Instead, you need them to tell you how they want to do their job.

Everything about The Dream Job stems from transferring the onus of responsibility of defining your job from the employer to the employee. The limit of one a year is what gives it the specialness, the prestige and the cache neccesary to attract the rare people who could handle such responsibility. The million dollars and the enforced hands off approach is what gives them the confidence that The Dream Job is something the company is taking seriously and is committed to integrating as a core part of how they do business. The requirement that applications be public is a filter that screens out the chuckleheads and leaves only those who have a credible chance of deserving it. The requirements are not carved into stone, they’re simply my interpretation of what would be the minimum required for The Dream Job to even work.

 The Dream Job is so stunningly obvious that it must be wrong. I can’t possibly have been the first person to have come up with this. But if it’s wrong, it’s probably at least going to be wrong in an interesting way. If you’re in a position to, do you have the balls to offer a Dream Job? If so, you better hurry because I know the first chance I get to scrape a million spare dollars together, this is what I’m doing.

Pop quiz: Which is a better deal for the government? Replacing a 10 MPG truck with a 12 MPG truck under the cash for clunkers program or replacing a 24 MPG with a 36 MPG car?

The correct answer is of course replacing the truck.

Consider both of those vehicles driving 120 miles.

The truck goes from using 12 gallons of fuel to 10 gallons of fuel, saving 2 gallons.

The car goes from using 5 gallons of fuel to using 3 1/3 gallons of fuel, saving 1 2/3 gallons of fuel.

It’s an unfortunate byproduct of Americans measuring the inverse of fuel efficiency. Europe and Australia don’t have this problem because they use L/100km.

For example: Which is a better deal for the government? Replacing a 10 L/100km MPG truck with a 8.33L/100km truck under the cash for clunkers program or replacing a 4.16 L/100km with a 2.77 L/100km car?

In this case, all that is required is simple subtraction to see that the truck nets you a 1.66 L/100km gain while the car nets you a 1.39 L/100km gain.

• Professional corporate ladder climber: If I had an alternate 30 years, I’d love to start from the mailroom at some gigantic company and try to work my way up to CEO. The key to playing the corporate ladder game is that you not only have to be confident but also willing to play office politics and so many people are not only bad at office politics, they actively disdain it which I don’t understand at all. There’s something immensely fun to me about figuring out how to effectively neutralize your enemies from within  your organization and move up the ladder.
• Operations research person: The goal of operations research is to streamline your manufacturing process to make it just a little bit smoother and quicker and more efficient every single day. There’s something zen about that devout focus to the small details which I find relaxing.
• Clinical psychologist: Clinical psychology contains some of the most fascinating issues of epistemology that you have to grapple with on a day to day basis. How do you effectively deal with someone’s self-sabotage when their self-sabotage is sabotaging their ability to deal with their self-sabotage for example?
• Professional chef: I’m a pretty decent cook and people ask me all the time if I’d ever considered doing it professionally. I’d always answered no because professional cheffing is nothing like cooking at home but if I had an alternate 30 years, it’d be fun to see if I could really perform at the highest levels.