gist JS

Tuesday, December 08, 2009

Decrypting Ruby AES Encryption

Want to encrypt something in Ruby and decrypt it in another language like AS3 using As3Crypto? Sure you can encrypt 'test string' and 'foo, without any problems. But say you want to encode something long.. like 16 character long. After much suffering, the following worked for me.

The main gotcha is that Base64.encode64 has a nasty habit of sprinkling newline characters to your strings. I can't imagine why it does this, particularly when there's a method b64encode(bin, len = 60) that is meant to do this, but I can tell you that AES sure doesn't like you adding random characters to ciphertexts.

Of course rails already thought of this and a method:
encode64s exists. So you could use encode64s instead of encode64, although I would argue that they should have called it something a bit more explicit eg Base64.encode64InAWayThatIsActuallyDecodesToTheSameStringYouStartedWith(string). Just a thought.

Anyway this As3Crypto Demo is a sweet way to test out the decrypt-ability of your encryption complete with all manners of paddings, encryption modes and schemes. Highly recommended.

Tuesday, November 17, 2009


I SIT in a chair and wait for my IDE to respond.

Millions of lines of code are run unnecessarily, indices are created and forgotten, gigabytes of ram are expended, the developers of RubyMine, Netbeans, Aptana chase the windmills of autocompletion, smoke pours across the fields of circuits and the spectre of kill -9 looms.

I sit in a chair and wait for my IDE to respond.

(with infinite apologies to Carl Sandburg who wrote the thundering poem mimicked here)

Wednesday, June 24, 2009

Security Theater

In Fooled By Testing I looked at some general concerns I've had about testing and what we can expect from it. In this part I'd like expose myself to the Internet at large and let you in on:

With no further ado, here are the past ~5 bugs I've let slip into production:

Production Bug #1
One of the joys of 'crowd-sourcing' some aspects of our application is that we regularly need to merge the symptoms that our user's report due to mispellings. When we do this, we automatically email the affected users so they know what's up. Normally this is merging 'tramors' into 'tremors' with 2 people reporting 'tramors' and 1500 reporting 'tremors'. It recently came up that we had 'headache' and 'headaches' as separate symptoms and each were reported by thousands of users. The task fell to engineering that we wanted to be able to do this merge without sending thousands of emails to users. I spent a good while writing pretty good tests that email didn't get sent and upgrading the merge tests in general and I felt pretty confident in my code. We got to QA and we realized that doing a merge like this was going to take 10+ minutes so we decided to move on and check it later.

Bottom line: Not once did we check to see if the site worked after we merged headache into headaches, the very functionality we were attempting to achieve.

Good News: The code I wrote worked. The merge happened successfully and no email was sent.
Bad News: The second the merge completed, all hell broke loose. You see, headaches is a primary symptom in our mood community and, long story and a bit of meta-programming magic later it turns out that primary symptoms get methods like 'has_headache?' created for them. The merge blew this symptom away thence there was no method and by this declension we proceed into the madness whereupon Hoptoad lit up like a christmas tree.

Now you are perfectly justified in claiming that only an idiot would let this happen, but I declare that I am often an idiot. Furthermore I do solemnly swear that I would have picked up on this without TDD. The great joy and efficiency of TDD, that I can exercise code through tests, is a A+ way to avoid rigorously seeing if the site actually works.

Production Bug #2
Added a css file that broke some style things in a different part of the site.
I grant you, this isn't an example of the evils of testing, but I will say that from a user perspective CSS-fail can look an awful lot like site fail. writing a test case to prevent this would be a monster task with poor ROI. If you a page to render in IE, you'd best load the page in IE.

Production Bug #3
Nobody merged the old production branch, so we regressed (tests existed... but they we're part of what didn't get merged).
Again, not testings fault, but what we're looking at here is 'Things that break production' and whether tests helped and in this case they did not. Sometime there is no substitute for manual labor.

Production Bug #4
Cannot Update Frozen Hash. This was some squirly ActiveRecord nuissance. The kind of thing that cropped up in Hibernate every 3 seconds bit which ActiveRecord generally seems robust to. It's in the middle of a gross controller that no one is proud of. A coworker couldn't reproduce this bug using the site, but 'fixed' this bug by writing a functional test that produced the same exception. He then fixed that, but the bug was still there. Eventually we found a way to exercise it using the app and then exorcised it, but I would argue that we got a false sense of 'fixedness' from this test.

Production Bug #5
Performance Disaster when searching for multiple treatments.
I know I know, nobody ever said unit testing was performance testing and technically this got stopped just short of production, but it got darn close and it was another episode of me writing my tests and thinking all was well and not testing the app as thoroughly as I should.

Production Bug #6
Tested: User.for_disease(disease), but actual form submits looked like User.for_disease([disease]). Turns out that array was bad news and could have had a major affect in production though thankfully no one actually used the feature involved.

Bottom line: It is astoundingly easy to write tests that seem to exercise the full functionality of the code, but for which subtle differences in initial conditions have catastrophic effects. (See Atmospheric Disturbances for more info on the profound psychological effects of perturbations in initial conditions :)

So is testing dangerous?
Well no, obviously not, but also yes a little.

Let's take a little break from development work and ask a similar question:
"Were the past 20 years of sharpe's ratio's and Case/Schiller financial analytics dangerous?"
Of course not. Before these tools existed, financial holdings were opaque and this was indeed dangerous. Once again however, as we've all learned, it was dangerous. Banks blew up in the 60's and 70's but knowledge of imperfect information breeds caution. Knowing 50% more but have 90% more confidence is dangerous in the extreme. A 99% chance the site works means 3.6 days/year of the site not working.

Obviously most of our rails apps are not 'too big to fail' (besides Twitter) but as confidence rises and paranoia decreases the chance of a killer bug grows sharply. If our profession is to avoid a backlash at the reliability of online applications, we can not simply throw all our eggs in the automated testing basket.

Let's say it 3 times together:

Unit-Testing doesn't find bugs
Unit-Testing doesn't find bugs
Unit-Testing doesn't find bugs

Bottom Line: Saying 'we can trust this, it's been unit tested' is a lot like saying 'what could go wrong? That debt is insured by AIG'

Tuesday, June 23, 2009

Fooled By Testing

This post comes as the byproduct of taking a break from testing rails apps and reading Nicholas Nassim Taleb's The Black Swan (also author of the less preachy Fooled By Randomness, hence the title) while laying on the beach for my honeymoon. As such it should be noted that any lack of coherence can be directly attributed to the balmy Caribbean backdrop under which this was conceived.

In the software development community and particularly in the Rails world, unit testing is the dominant paradigm of professionalism. I don't think it's much of a stretch to say that most rails coders would agree that if you're not testing, you're doing it wrong and perhaps that it's possible classify coders as either 'software engineers' or 'programmers' and that this would basically be the split between 'unit testers' and 'non-testing, script-kiddie hacks'.

These posts are my attempt to think a bit about whether testing should really be sufficient to bestow the coveted status of 'engineering' on our profession and about dangers of relying on tests.

Don't be a turkey

The best graph in The Black Swan is (similar to) the following:

What can we tell about this graph, absent any other dimensional information? Well, it might seem like we can tell quite a lot. We have a generally increasing metric and it would seem we could be fairly confident that an extrapolation from this data would be a valid conclusion. Of course this actual graph looks like this:

and it turns out that it's a graph of Food Eaten over Time for turkeys. I'll leave it as an exercise for the reader to determine where Thanksgiving lies on the timeline.

In development terms, this reminds me very much of the first application I wrote using Freemarker as a templating language. The following worked brilliantly... until it didn't.

<#list products as product>
<a href="products/${product.id}">${product.name}</a>

Can you spot the bug? Does it help if I mention that it worked for the first 999 products? Sadly, product 1000 rendered as '1,000' which is not a integer, breaking the link. (the solution Travels-travails-freemarker)

Past performance not indicative of future results.

To quote NTT:
"our emotional apparatus is designed for linear causality".

He describes this sort of problem as a 'black swan'. An unpredictable event that has dramatic, game-changing effect. The central thesis of the Black Swan is that these events are, in some respects, the only things that matter. Worse, they happen much more often than your brain was built to expect. I think this jives with any seasoned developer's sense of the amazing ability of code to break under duress (numbers greater than 1000, demo-ing for the boss, butterfly wings, etc).

So what does this have to do with testing?

Testing is billed as a way to get early, continual positive feedback. NTT would say that we're attempting to 'platonicize' software: to wrest order and predictability from chaos. I think it does do a great job of that, or at least, I think it does a great job of making us feel like we've created order from chaos. TATFT == 'dopamine release'. It makes us feel good. Better yet, since our ancestral environment was made of of small, localized, linear events and it is thus hedonistically better to spread positive effects over time we've developed continuous integration and this compounds the hedonistic effect.

def gather(berry)
return :good_taste

Wash, rinse, repeat. After a long day of successful unit testing I feel like I'm the king of the world.

But let's face it: software couldn't be less linear if we tried, (and we're trying: ie concurrency). The big problem category: cascading fails, server flapping, the slashdot effect are all highly non-linear problems and the small problem category: unexpected input, unexpected side-effects is populated principally by unknown unknowns.

To write good software we need to stay vigilant.

Gaining confidence from testing is akin to like throwing a shawl on in a hurricane. I went to a talk from ThoughtBot a couple weeks ago about TDD and they showed an (admittedly awesome) slide of a cat walking calmly past a line of massive german shepards.

Look back at the turkey graph. That turkey felt probably felt pretty confident about getting fed right before he got led off behind the woodshed. That cat is not the kind of code I want to release. That cat trusts that her tests > uncertainty. I want my cat to be a paranoid, ex-CIA, wacko, wearing a bullet proof vest, tin foil hat and packing an uzi. Confidence is the enemy.

So, "just write awesome tests" you say? Let me pick a couple bones:

Testing is complex:

Test This:
def suggest_event?
user.can_report_events? &&
(effects > 0 || side_effects.any?{ |s| s.severity >= 2})

We can generalize this to the form: A && (B || C) and this humble one liner seems like it should be a great candidate for a test, easy setup.
In reality it has 8 separate execution paths, essentially forcing us to test at least a 2^3 truth table, and even that won't really capture the numerical comparators eg checking the >=.
(If you're thinking that you don't need to individually test ¬A, B, C and ¬A, ¬B, C, because ¬A should short-circuit the AND, then I would say you're not really thinking of this as a black box)

Now imagine how you might do set this up with shoulda or context? I shudder to think of it. Some custom truth table asserter would probably work, and maybe you do decide to spend a half hour writing your test frameworks. Good job, but at the end of the day all you're really doing is exercising the ruby interpreter's logical expressions evaluator.

So testing all possible code paths starts to get impossible for complex applications. So what do we do in response? I believe that all too often, the solution is to: test trivial applications! Yes, we mock out the expected interactions, stub the thing to death and are rewarded by the sweet success of our favorite color 'Test Pass Green'. The tests run fast and true, but there's just one little problem: the site doesn't work. I call as expert witness Yehuda Katz, who just had a nice post about this sort of problem on the Rail codebase. http://yehudakatz.com/2009/06/20/on-rails-testing/ Yehuda?

My general rule is “Don’t mock anything you own” and more strictly “Don’t mock anything happening inside your own process”.

Thanks, Yehuda. Just what I was trying to say and (unlike me) people presume you know what you're talking about.

So what is the point here? The point of this exercise is primarily to remind me that testing is not a panacea and that overconfidence is to be avoided. As I read & write more 'well-tested' code I'm becoming suspicious that it is more common for pernicious edge cases to slip towards production because as developers we're thinking too much inside the testing box. That is my thesis.

So what's next? Part 2 "Security Theatre", where we look at a few case studies including a list of the past 5 bugs I've let slip into production.

FWIW, I'm aware that Unit Testing and TDD were never truly billed as an answer to all these woes. I'm also aware that I'm not the first to think about this stuff:

Monday, April 13, 2009


Always enjoy doing my e-civic duty with a little Google-Bombing. Interested in Amazon Rank?

Or keep up to date on Amazon forgetting what age they live in with twitter.

Wednesday, April 08, 2009

git panic

Not sure about that rebase, merge or pull? Worried that listening to Kenny Loggins while you work might hose the repo? Don't let this happen to you.

Introducing Git Panic, your 1-step disaster prevention toolkit. If you wouldn't walk around a hallway in Doom without auto-saving, why would you use version control without the same measure of paranoid caution? Remember, git is one of the most dangerous tools on your system. While it is powerful, using git means that with one simple command you could easily trash the remote repository and simultaneously downgrade from OS X to Windows 98.

git config --global alias.panic '!tar cvf ../git_panic.tar *'

Now whenever you get nervous, just push the 'git panic' button and breath deeply, you can always restore from a tarball. Good ole tar.

Extra points if you hook it up to a red button on your keyboard.


Friday, March 20, 2009

git rebase interactive: maverick style

Re-writing history? You know you feel like a bad-ass. Why rock git in silence?

function git-mav {
afplay -s 0 12 ~/Music/iTunes/iTunes\ Music/Kenny\ Loggins/Top\ Gun/Danger\ Zone.mp3 &
git rebase -i $1

git-mav HEAD~10

Wednesday, January 21, 2009

vi in five minutes

For a long time I was pretty merciless with vi users. Me and my IDE posse would hide the icons on their desktop, pants them in front of girls, etc. Mostly though, everything was fine. I used my tools and they used theirs. But as we all know vi stands for virus (well, it's a shortcut for virus) and the vi plot is to take over all the editors on your system. That's why even if you're a loyal IDE-bound code monkey, honking away and generating thousands of lines of configuration XML, home interfaces and remote interfaces, vi still manages to sneak up on you when you do something silly like svn commit and leave off the -m to add your own message.

At this point I would scream out over the cubicle wall 'how in the @*#(! hell do I close vi!!' And they would tell me :q! and I would stop listening after the first colon and just close the offending terminal.

But sometimes we'd get into the issue in a little more depth.
  • I'd say 'vi sucks.'
  • And they'd say: "you just don't know how to use it."
  • And I'd say: "any bleeding editor where the arrow keys only work if your config file asks nicely and I need to switch modes just to type is an abomination."
  • And they say something like "If I want to start all the paragraphs in the document with '========' I just type 'a9 i8= . wtfbbq'."
  • And I just stand there and have pity over their poor abused souls.

But now that I'm on git, what was once a passing annoyance has now become a daily part of life.

Happily this time everything is different. Different because of the actions of one man: Jon.

Much as the heroes of Starship Troopers find that they need to understand the bug to beat the bug, it turns out that spending the five minutes it takes to get a handle on vi is actually worth it. Counterintuitive I know, but it must be done. Jon has written up something that very nicely explains Why, oh WHY, do those #?@! nutheads use vi?

Now, armed with a grounding of wtf vi is trying to do, we are now prepared to do battle with the beast.

And beast it is. With no further ado I present:

vi in 11 simple years

Year 0: Accidentally end up in vi. Close terminal.

Year 5: Tape a Post-It note with ":q!" to your monitor. This is a significant improvement.

Year 6: Type i to enter insert mode. Use it like notepad. Type "esc :wq" to save and quit. Add these to the post-it note.

Year 11: Read the aforementioned post. Decide it's time to take the plunge. Learn 5 new key combos and finally, finally get the sweet sweet power of copy and paste back!
Insert Mode: i
Cut line dd
Paste buffer p (or P to paste above)
Cut 3 lines at the same time!! d3
Delete word dw
Quit without saving :q!
Save and Quit :wq

There it is. vi in 5 minutes. Plus 10 something years. Who says you can't teach old dogs new tricks.