Featured Posts
July 20 2009

Kindle, 1984 and schadenfreude

by Hang

Everyone’s all atizzy about Amazon’s recent decision to removed unauthorized copies of 1984 from the Kindle. All of a sudden, we’re reminiscing nostalgically about the freedoms inherent in paper and how the new digital era represents a grave threat. Seriously?

When we were all napstering and torrenting away, digital information represented us sticking it to the man and a sign that RIAA was so desperately out of touch with the changing media landscape. But as soon as the same phenomena hits us, we end up responding exactly the same way that RIAA did, desperately trying to preserve the institutions of print media despite arguments from first principles about how such a thing is impossible.

I hope there’s some dude at RIAA right now who is fully appreciating the irony of all this.

June 29 2009

Statistics should be the foundation of mathematics education

by Hang

In this recent TED talk, Arthur Benjamin talks about how we should have our High School math classes directed towards statistics, not calculus, as an end goal. I’ve been a heavy advocate for this ever since I wrote my post about how statistics is a philosophy subject.

June 26 2009

Doubling my podcasting efficiency

by Hang

One lesser mentioned feature in the new iPhone 3.0 release is the ability to listen to songs at half, normal or double speed. Last night, I experimented with listening to my podcasts at double speed and, while it took a bit more mental load, I found that I could reasonably keep track of the conversation. Given the amount of podcasts I listen to, this feature is shaping up to be a serious productivity boost.

June 17 2009

Statistical vindication

by Hang

A few days ago, I wrote about a case of a seemingly fascinating graph which I felt was used inappropriately. I was rightfully castigated in the comments for being too harsh but, to me, it gave the impression of a pattern when there really was none. In reply to some of the comments, I made the observation that

The only reason I wrote about it was because, I was surprised that even I as a reasonable trained statistics guy was momentarily caught off guard by it. Clearly, you meant nothing malicious by it but it’s a technique that could be used for malicious purposes so I wrote about it.

Now, in the wake of the Iranian Elections, it seems like my speculation has been somewhat vindicated. Andrew Sullivan posted what he claimed was the red flag that proved the Iranian elections were a fraud. And it seems eminently convincing. Luckily, Nate Silver produced a null hypothesis graph based on the US elections and demonstrated that the “red flag” was just a case of the exact same statistically fallacy I wrote about a week earlier.

Pain, Gain & Brain Candy

by Hang

One of my close friends, when asked if she would read my latest post, replied:

Not really sure I want to be enlightened in that way, but since I’m such a motherfucking glutton for punishment, I’ll read it. How does it make you feel to know that many of the things you say and write are punishing or cruel to other people?

My instinctual response is that I view pain somewhat akin to how a personal trainer would view it. Yes, there is bad pain but there is also pain that leads to growth as well and if it wasn’t at least a little bit painful, I don’t think I did my job right. But to be honest, I’d never really considered it all that closely before. It forced me to confront a world view I was somewhat alien to and, in the process, define my own world view a little more closely.

To me, pain is how you know you’re alive. Pain is thrilling because it’s transformational and without transformation, what else is there beside marking time? Pain is integrity, the prospect of pain is where you prove to yourself who you really are. Pain ties deeply into notions of masculinity and what it means to be a man. The masculine concepts of courage, cowardice, stoicism & loyalty all have to deal with reactions to pain and fear. This is not to justify this world view, merely to explain it.

A lot of my writing stems from these premises. It’s confrontational and brash and requires a bit of heavy lifting to get. Until now, I’d never thought it could be any other way. Part of the reason for me starting this blog is to find a community of like minded thinkers who view the world from that same lens of intellectual masochism. It’s been a frustrating process for me that I’ve been at this for almost a year with so little to show for it from that regard but I’m going to keep on plugging away at it.

Anything you think is either unoriginal, wrong or both

by Hang

I first discovered this obviously wrong truth when I was doing my honors thesis. Time and again, I would come up with a novel idea or a neat algorithmic trick. Some of them, I would discover had already been invented 3, 5, sometimes 10 years before I came up with it. But the ones I was absolutely sure nobody had published before because I had scoured the literature and covered every approach. Well, all of those original ideas turned out to have some hidden, unforeseen flaw that rendered them either trivial or actively stupid. This lead me to formulate the belief that “anything you think is either unoriginal, wrong or both“. Like all obviously wrong truths, it has the paradoxical property of being obviously wrong and also true.

The premise for the statement comes from the simple observation that good ideas survive and bad ideas die. This means there exists an entire class of awful ideas that people come up with time and again only to eventually discover their wrongness and then abandon them. Every person who discovers them believes themselves to be wholly original since nothing of the sort exists in the world and each of them is met with disappointment, sometimes after many years of sweat and toil. But because failures are almost invisible, they leave no warning signs to future generations that this is an awful idea that should be avoided*.

Anything you think is either unoriginal, wrong or both” is an acknowledgment of your own stupidity. Your first instinct, when you come up with a new idea, should be to try and find out if anyone else has done it before. Your second instinct should be to try and find out if anyone’s done it before. Your third, forth and fifth instincts are to ask how come everyone else figured out this was a dumb idea and I haven’t? If you’ve gotten this far and you still haven’t discovered anything useful, you should start feeling a little bit uneasy, it probably means you weren’t smart enough to discover how wrong you are.

If you have discovered the prior art or the fatal flaw, then breathe a small sigh of relief. Unoriginal ideas are GOOD, wrong ideas are GOOD. An unoriginal but right idea is still valuable to all the other people who’ve never heard of it and chances are, if you’ve never heard of it, there will be a significant fraction of the population to which bringing this idea contributes value. Wrong ideas do more to teach you more about the world than right ideas because they teach you about some discrepancy between your expectations and the world, The corrective force of wrong ideas is what allows you to deftly cut to the core of any issue and tease out just where assumptions are weak and likely to fail.

But if you’re lucky, over the course of your life, you’re going to stumble across many ideas which are both original and right, in which case it’s still better to treat them as unoriginal and wrong. Believing an idea is unoriginal and wrong makes that idea do more work. You attack it more fiercely and from more angles. You keep on asking people if the idea sounds familiar and you’re eager to seek feedback because you’re so damn curious to discover why it could be so wrong yet elude you for so long. In doing so, you disassociate the idea from your ego so that you can take criticism about it calmly and dispassionately. Eventually, that drive of curiosity will force you to action, just to finally prove how this idea is flawed. Treating an idea as unoriginal and wrong means that the only standard you’re willing to accept is success. This brings a clarity or purpose that cuts through the confusion when executing upon that idea. Other people may be willing to make excuses or caveats that salve their ego but, as far as you’re concerned, if an idea is not successful, it’s not right**.

Anything you think is either unoriginal, wrong or both” is an idea that also applies to itself. I’ve been slowly chewing over this idea for almost four years now and it’s been frustrating to me that so far, I haven’t been able to find someone else that’s expressed it as a similar sentiment which by de facto, makes it wrong. I’m putting this out there to invite the embarrassment of someone pointing out the obvious source or the obvious flaw that I’ve managed to miss for so long. Please, tell me how I’m stupid, it would be a welcome relief.

*Some people, when first discovering this problem, come up with elaborate schemes of recording all of these common awful ideas so that future generations can avoid them. This, unfortunately, is a common awful idea.

** not right and wrong are different concepts in the same way that not being a millionaire is different from being homeless.

Advice to new college grads: figure out how something is produced

by Hang

This advice has no basis in anything but intuition but as this recession is deepening and more and more of my friends are facing the prospect of un or underemployment, I have one piece of advice: Figure out as much as possible about how one thing is produced. The actual details don’t matter so much, what’s important is to gain the holistic, birds eye view of the entire production process.

What do I mean by the entire production process? I mean try and look at everything. If it’s a physical product, figure out where it’s manufactured & what the manufacturing process is. Figure out the logistics of shipping it from the factory to the store. Figure out how vendors relations and the sales process works. Figure out the task of marketing, what are the various channels it’s marketed through. Figure out the corporate philosophy, both as it’s stated and as it’s applied on the ground and understand the impact that this has on the final product. Figure out the R&D stage and how the idea for it was shaped. Figure out all the design constraints and internal politics that lead to in looking like it’s final form. Figure out the legal landscape that it lies in. Figure out it’s competitive market and how it’s situated in the context of similar products. Figure out how the users think and feel about it, what narratives they’ve built up around it and if there’s a culture around this product. Figure out how the social mores of different cultures and socioeconomic classes play into differing amounts of acceptance for this product. Figure out how the internal accounting rules work and how that impacts the budgets for various departments. In short, just go exploring.

The choice of product is largely irrelevant, what I think is important is to see the diversity of effort that needs to happen for something to be produced, the complex web of connections that constitutes a modern economy. What’s more, it forces you to step outside of the box of abstractions and deal with real, concrete scenarios in all thier glorious messiness. But focus on one product and one product only so that you achieve another limited, but far more useful form of blindness. Of course, even with the tightest scoping possible, you’re still looking at a lifetime of work so it’s up to you to define you’re own personal stopping criteria but I can say for myself I’m nowhere near to stopping. Each step leads to a new step to explore, it adds another hundred things to my already massive to-understand list. Someday, I’ll assume I’ll stop breathing and that’s the day I’ll stop doing this.

What’s the benifit of this? It’s hard to express to someone who hasn’t already done it. Figuring out the entire process gives you perspective and context. It situates your own tiny role within a larger context. I know a lot of bright, enthusiastic, dedicated people who are leaving school and the thing that most frustrates me about them is that they just don’t *understand*. They’ve been trained within their particular discipline and culture for their entire schooling career and they’ve lost the ability to see the forest for the trees. What’s more, it’s given me a certain peace and groundedness. The modern economy is abstractions piled upon abstractions. There’s something viscerally solid about understanding an entire process. Before doing this, I felt like I was floating in a sea of clouds designed to insulate and protect me. Those clouds were great but there was something insubstantial about the entire thing. Now, with the feeling of at least one foot on secure, stable ground, I feel more confident in pushing my head much further into the clouds and maybe that’s the most important reason of all.

June 10 2009

Another way to mislead with statistics

by Hang

I ran into a great blog post this morning on Using Mechanical Turk to evaluate search engine quality and came across this seemingly fascinating graph:

Something about that graph just invites reflection. What do marlboro schools, fidelity and ford have to do with each other? Is Bing better at boring queries and Google better at sexy ones? It wasn’t until 5 minutes in that I thought “hang on, shouldn’t the null hypothesis generate a binomial distribution anyway?”

So I decided to run my own simulated Google vs Bing test in which people guessed at random which search engine they liked and got this:

Null Hypothesis for Google vs Bing

As you can see from the simulated graph, asking why marlboro public schools did so much better on Google and tax forms did so much better on Bing is essentially as useful as asking why Query 37 is so much more Google friendly that Query 22.

The blog entry claims that there was a minor but significant  (p < 0.04) difference in overall quality but it’s obvious from the null graph that no individual query is statistically different in quality (I’d unfortunately have to dig out my stats textbook to figure out what test I would need to run to verify this but I’m pretty confidant on my eyeball estimate).

I understand the urge, when you have an interesting dataset to throw up any and all cool visualisations you have, I’ve been guilty of doing it myself many times. But responsible presentation of data requires a certain discipline and responsibility. Each graph should tell you at least one interesting piece of true information and strive to minimize the amount of false information presented. Unfortunately, the aforementioned graph cannot possibly communicate any true information because there is no true information to present and the false information is amplified precisely because it is such a fascinating graph. The worst of both worlds.

If I were the poster of the original piece, the way I would have deliberately not included that graph but I would include the following sentence:

Given our small sample size, we could not find any particular type of queries in which either Google or Bing significantly excelled at. It may be that Bing is better at product searches or Google excels at medical queries but no evidence of this was found in our study.

Even this is problematic but at least it includes several pieces of true information.

Like I said in a previous post on lying through the use of not statistically significant:

Sometimes, I swear, the more statistically savvy a person thinks they are, the easier they are to manipulate. Give me a person who mindlessly parrots “Correlation does not imply causation” and I can make him believe any damn thing I want.

June 2 2009

People are twice as smart as they talk, three times as smart as they write

by Hang

I used to think that the point of editing was to eliminate all of the poor writing from an article so only strong arguments remained. It wasn’t until I started writing this blog that I began to appreciate how much good content routinely gets cut from the final product. Ideas will get cut if they detract from the focus of the piece, if there’s no logical place to put them anymore, some are even cut because they’re too interesting and merit their own separate post. I even cut a good line from the end of this paragraph because it wasn’t the appropriate sentence to close a paragraph with.

Of every post I write, I estimate that I cut two good lines for every one good line that makes it into the final piece.What this means is that if you’re reading a piece of writing that you think is pitched at your level, the writer had to be three times smarter than you to have written it. Similarly, I noticed that when I was speaking, I would only say about half of the clever stuff I wanted to say which meant anyone talking to me was probably twice as smart as I think they are.

This is, perhaps, why the internet is so full of assholes who think they’re the smartest person in the room. If you have a look at the comment section of almost any piece of content, at least half the comments tend to be some snide implication that the commenter is smarter than the author. Next time you read something, mentally assume that the author is three times as smart as they appear and think of how you would respond then. All I can say is ever since I started doing that, I’ve not only been far calmer, I’ve learned much more too.

May 12 2009

Missing the point of interviews

by Hang

I don’t know if two makes trend but I’m too impatient to wait for a 3rd example so I’m going to rail against the geek trend of being cleverly literal when attempting to answer interview questions. This practice strikes me as about as original and amusing as pointing out that silenced guns in movies don’t sound like they do in real life and there’s no sound in space. Like film, interviews have their own set of conventions and rituals and exposing the inherently unreality of the form doesn’t mark you out as clever, just ignorant.

In primary school, you were probably given a series of largely banal word puzzles in your math class because some educational bureaucrat decided that stories were more “relatable” than numbers. Now, if you had an absolutely stellar textbook author, they would have taken to the medium with a gusto and crafted an entirely different pedagogy centered around stories as an expressive medium. But chances are, you didn’t. Chances are, the author took the bog standard approach of first coming up with number questions and then pasting on a thin veneer of wording to get the job done.

“3 + 5, Jane had 3 apples, Chris had 5 apples, how many apples do they have together?”

> 8 apples

“Correct, 18 – 2, Reginald has 18 cookies but he eats two, how many cookies does Reginald have left?”

> 16 cookies

“Correct, 10 + 10, Heathcliff has 10 gallons of water, Shaniqua has 10 gallons of ethanol, how many gallons would they have if they combined it?”

> 19.2 gallons

“Corr… Wait, what?”

> Mixing equal volumes of ethanol and water results in only 1.92 volumes of mixture.

Congratulations, you discovered a leaky abstraction but you also kind of missed the point.

The purpose of a programming question in an interview is not to simulate a real life job decision, it’s simply a very basic skill test with a thin veneer of story pasted over the top of it to make it seem relatable. If you persist on treating it that way, don’t blame me when I give you full marks for cleverness and then zero marks for getting the point.

Copyright ©2009 BumblebeeLabs — Theme designed by Michael Amini