Fuzzing for Software Security Testing and Quality Assurance

Good Things

• He really puts fuzzing in context. Fuzz testing has been around for a long time, and this book gives you the full historical perspective, as well as a modern view.
• Fuzz testing is important. When Gary McGraw and company did their Building Security In Maturity Model, one of the activities that virtually everyone did was fuzz testing. Clearly we need books like this to get everyone onboard.
• Although Ari is CTO of Codenomicon, a commercial fuzz testing tool vendor, the book is not a pitch for his tool. He actually gives lots and lots of information on a broad variety of tools, including free tools. It’s a complete and honest vision that is not overly promoting his company’s product.
• I learned a lot of fundamentals that make a difference to how I fuzz test things. For example, I now understand mutational versus generational fuzzers. They each have benefits and you probably want some of both for good coverage.

The Not-So-Good

• I think he spends too much time talking about motherhood and apple pie security things. Things like security testing, risk analysis, code analysis, etc. There have been ample trees killed on these topics and I don’t think the treatment in this book really adds to that body of knowledge. I would have been happier with just some references to the rest of the world.
• The comparisons of commercial and free tools are intermixed with all this extra security discussion. So sometimes you have to read about security metrics or some other broad topic in order to find a specific example of a specific tool.
• The authors’ perspective is too much fuzzing über alles. They downplay the value of techniques like static code analysis and architecture risk analysis. Those techniques are complementary, not counter, to fuzz testing.

I like the book a lot and am glad I have it. I recommend it.

Some colleagues of mine have recently published a book Implementing Automated Software Testing.

Elfriede and Thom were key organizers of VERIFY 2007, a software testing conference.

Now I’ll make some comparisons that seem completely unrelated, but they’re not. There are people who smoke and do not get cancer. There are people who do drugs and do not get caught and do not have major health problems. There are people who get in car accidents when they’re not wearing their seat belts, but they survive with little or no injuries. There are examples of all kinds of things that you should not do on a regular basis, but yet someone gets away with it just fine. There are probably plenty of 13-year-olds who can handle the Deathly Hallows just fine. The existence of some does not mean that it is good for all.

Secondly, let me try to make one final point clear. (I’m about ready to give up on this point as one that is too subtle for younger readers). The fact that you can read the book, that you understand what it says, does not mean that it is good for you to read it. There are all kinds of things that you can find—online, in print, at movie theatres—that you would fully comprehend. That doesn’t mean that it’s good for you to learn those things at this age.

Finally, if you want to go on record as a representative of your age (i.e., “I can read it so everyone my age can”) then learn to write. Otherwise, we’ll all conclude “well, maybe 13-year-olds can read, but they sure can’t write.” Don’t go down on record as the illiterate representative who cannot spell, cannot use punctuation, and who types on a computer keyboard as if she were typing on a phone.

SPOILER ALERT

I will freely discuss anything and everything from the book from this point on.
It may contain spoilers. Don’t read farther if you don’t want to know how things end.

Good: I really love the character development. The kids are really coming into their own. Ron’s not just a prat and a foil for Harry. He’s been gradually stepping out of Harry’s shadow over the last couple books, and this one really does him credit. Hermione is amazing. She’s no longer a know-it-all annoying kid, but instead she puts her knowledge to work. She’s not putting it out in everyone’s face. She’s learned some humility. I love her character. Harry really turns out to have wisdom and intuition which is just awesome.

Bad: The whole middle of the book really dragged for me. A few important plot points were established: Ron leaving, the whole story about the Deathly Hallows, retrieving the locket, etc. They just took so long to get there. A friend of mine said he thought it was to emphasize time passing so that certain things could happen in the book: Tonks having her baby, etc. There’s a difference between time passing in the story and pages passing in the book. I think time could have passed differently.

Disappointing: I don’t care for the ending very much. One of the problems that many books like this face is the “You just saved the world. Now what?” problem. If you just saved the world, you don’t get to go back to normal the next day. There’s serious adjustments and life-altering decisions to be made. Life has been turned upside down and inside out—not only for the main characters, but for the whole wizarding world. You don’t just show up for work on the following monday and say “I read in the Prophet that you-know-who is dead. That’s nice, isn’t it?” Nor do you say to your business partner “I guess we need to take out want ads since half our staff was killed and the other half carted off to Azkaban.” Life is so NOT as we knew it. The ending of the book is just too short and too glib on this point. I crave a good, happy ending as much as the next reader, but this was just too disconnected. A lot of interesting life decisions got made in the weeks, months, and years immediately following The Dark Lord’s Return (or whatever the wizarding world would have called it, for it would certainly have earned a name as a historical event). It’s all glossed over, and everything’s happy.

You know who got this “you saved the world, now what?” thing right? Tolkien. When you read the end of The Return of the King, there’s like 150 pages of stuff that happens after the ring is destroyed. It was a real problem for the film makers, because it just doesn’t fit our concept of a feature film, but I found it tremendously satisfying to watch all the ends tied up and to learn how people came back from fighting a world-altering war and try to make their lives normal again.

Outstanding: There were a few choice events that were just outstanding. Dobby’s death was handled superbly. I really felt for Harry and for all of the characters who liked Dobby and who mourned his loss. Goblins and their relations with wizards were fascinating. I thought she really created a vivid and unique interaction there.

“I’m Basil Exposition”: If you watch any of the Austin Powers movies, you’ll know there’s a character whose name is Basil Exposition. His whole purpose in the movie is to fill the role of narrator. His name mocks the various spy movie characters that have had to do the same thing: deliver long monologues whose whole purpose is to fill the view in on details that won’t be shown on screen. (”Dr. Evil has built an underground lair in a hollowed out volcano…”). So what does Austin Powers have to do with Harry Potter? Well, Rowling left herself a lot of loose ends to tie up in the final book, and the only way to tie them up was to do long expository dialog in places. You can spot the exposition coming in situations like p. 406 when Hermione asks “But what are the Deathly Hallows?” For the next 10 pages, Xeno Lovegood goes into excruciating detail and very helpfully explains the Deathly Hallows to the kids. Remember: he’s just sold them up the river when he starts the explanation. He’s only buying time to keep them occupied. Yet he goes to great lengths to put them carefully on the right track. It makes little sense. You can chalk it up to his being eccentric, silly, or actually diabolical (consciously putting on friendliness so as not to spook them), but none of those explanations is really satisfying to me.

The whole Exposition thing happens again with Snape’s death and his memories. It’s vital that we understand some of what happened to Harry’s parents’ generation, but it becomes pretty much exposition. I wonder if there was a way to learn those things without it being so blunt.

This isn’t for my kids. You pretty much have ought to be Harry’s age to read the books about him. A 10 or 11 year old can read book 1, a 12-year-old can read book 2 and so on. Although young kids can read the books, I’m not sure they should read the later ones. I say this because the books get really, really dark as the series progresses. Book 7 opens with torture and death in the first scene, and characters that you’ve grown to know and love start dropping like flies. By the end of the book you’re seeing really important secondary characters dying, and it’s just too much to imagine a 12- or 13-year-old reading something this dark I worry about how young kids might be affected by reading something so dark. My kids are a long way from being old enough to read any of them, so perhaps my opinion will change with time. When I understand what my 10- or 11-year-old is capable of understanding, maybe I’ll see it differently. But my intuition says ‘eek! this is dark, evil stuff we’re talking about here, and I’m not sure a young kid can handle it.” That makes it tough to introduce a kid to book 1, which is such great fun and accessible and essentially happy. If I do, then I have to get into a really difficult and tricky process of determining when the kids are ready for the next book, even though they’ll be sure they’re ready immediately after finishing the first.

Alas, it’s over. So I know she’s said there will be no more HP. That’s fine. End a good thing while it’s still a good thing. I just wish there was one more half a book called Harry Potter and the Stuff that Happened Next. I feel pretty unsatisfied at how it all got pulled together really quickly.

[Revisions]

You’ll notice that there is some text with a line through it above, and some italicized text near it. I got a bunch of good comments from young readers. There’s a lesson to be learned here about writing what you mean and being clear. I changed my original review to make it clearer about what I meant. I left the original words, though, so you could see how it changed and why people were saying what they said.

Originally, I used the term “can” (which really implies being able) when I meant “should” (which implies a value judgment on doing something). Several readers wrote to remind me that they were perfectly capable of reading the book. Of course. My real point is that, although they can, I’m not sure they should. There are lots of things that one can read, but perhaps one should not read until one has reached a certain level of emotional maturity. Now, I bet the same people who wrote to tell me that they were capable of reading it at a young age would also assert that they were mature enough to handle the material. That’s fine. I can appreciate the disagreement. I don’t think I have the skills to persuade a 12-year-old that he or she shouldn’t read it. When I was 12 and was sure of something, I was pretty hard to persuade. Heck, I’m not even sure I’m right. (Notice that I point out that I’m not yet a parent of a 12-year-old, so I really don’t have a good idea what a 12-year-old can handle emotionally).

So I write this amendment to the review as a tip of the hat to my readers. I hear you and I’m glad you’re reading.

A few things on the web site stand out this way. We know that Tim outsources as much as he can. It’s 99% likely that he has outsourced 99% of the web site development. These folks don’t really think much about security, or maybe it’s really not important. The whole point of going to the web site is to give Tim your email address so he can start emailing you things. This book is not just about selling copies, it’s about giving him leads for people who might buy other products.

You can actually read the source code of the web for that asks for your email and discover that it’s just going to send you to an unprotected web page. That is, if you know to go to this web page, then you don’t have to give him your email address at all. Before you get too excited, there’s not actually much there. Consider that the person who knows the “password” is supposed to own (and therefore ostensibly have read) the book. A person who has actually read the book and then goes to this password protected section of his web site will be disappointed.

At the time I went, it had six links:

• Introduction – My Story
• Outsourcing Life
• How to Check E-Mail Twice a Day… or Once Every 10 Days
• Hour-long podcast presentation on “The 4-Hour Workweek” fundamentals
• Lifestyle Design FAQ
• Interview with Tim and Rolf Potts of Yahoo Travel: “Freeing Yourself From The Daily Grind”

None of it is value add. The first two are sample chapters (remember, we already own the book). The next one is a blog entry you could just navigate to by going to his blog (no need to go to the top secret site). The fourth link is a podcast on someone else’s site (again, googling for Tim Ferriss and the book title will get you there). The fifth is another sample chapter from the book (that we already bought and read) and the last one is a Yahoo! travel news story.

So, pretty disappointing until you realize that he practice what he preaches. He outsourced the development of the web site, reused some content with minimal effort, and hyped it up to make it appear like added value.

Another overarching thought to this theme of joining the “new rich.” If there weren’t a whole lot of “old poor” around willing to do the day-in, day-out work that he’s outsourcing, there couldn’t be a “new rich.” It’s a dessert-before-dinner (or maybe even dessert-instead-of-dinner) lifestyle because it can’t sustain a culture. You can’t have any significant fraction of society living this way, because the trains do have to run on time. Someone has to be a policeman, fireman, doctor, lawyer (OK, maybe we can live without the lawyers), teacher, pilot, etc.

I’ve done a lot of disparaging here. Do I reject it wholesale? No. One of the big things Tim opened my eyes to was the potential I am wasting. I have the potential of creating lots of small revenue streams that will add up to money and time for pursuing dreams much sooner than if I just wait until traditional retirement or the empty nest. Tune in for my incorporation and interpretation of his ideas.

Appendix A is a republished history lesson that Whittaker wrote for IEEE in 2003. It is an oblique bit of value. Not completely off-topic, but not squarely on-topic, either. To me it feels like filler. Appendix B is a list of bugs in their demo app. That’s handy. Appendix C is high value, but short. It’s a list of web hacking tools and brief descriptions of how to use them. This is the only section that gives you hope that you can do what they talk about in the book. The descriptions are brief, unfortunately, and don’t do the tools justice.

Everything they describe is totally Windows-focused. When they talk about, for example, protecting against sample code left by default installations, they have only one useful piece of advice: IISLockdown. That overlooks the fact that the majority of web servers (accoding to www.netcraft.com) do not run IIS, and that the sample code we’re worried about didn’t come from IIS in the first place. Vulnerable sample code comes from WebSphere, WebLogic, JBoss, .NET, or some other middleware. In further irony, they don’t talk about how to secure PHP against attacks on its defaults, even though their demo application is written in PHP. When they talk about SQL injection, most of the examples they give are Microsoft SQL Server specific, and do not work generally on many databases. The Microsoft focus isn’t just a technical criticism. One of the only security experts they refer to is Mike Howard at Microsoft who, although a bona fide security expert, is not the only source of decent security information in the world.

If you use a Mac, or Linux, or test any sort of system that is based on something other than Windows and IIS, this book has less value for you. Most of the tools on the CD are Windows-only and closed-source. It’s up to you to adapt their advice to fit your needs. For example, you have to know to get the LiveHTTPHeaders extension for Firefox. They only recommend the IELiveHeaders tool for IE.

There’s too much pining away for the old days and comparing to older programming paradigms. These authors have been around a long time, and their security pedigrees are good. Unfortunately, saying “the web is different” is not actually important or useful. For most people writing insecure web applications today, “the web” is all they’ve known in their programming career. It’s not “different” because they don’t know anything else.

The “How to Protect” discussion at the end of most attacks is usually useless. For example they have a whopping 2 sentences on p.35 about how important it is to protect against bad input. Duh. If only it were so simple that all we had to do was say “input *must* be validated” and leave it at that. The fact is, this book is about how to “break” software, not “do it right.” A text on “doing it right” is much larger (c.f., Viega & McGraw’s Building Secure Software or Hoglund and McGraw’s Software Security: Build Security In). These little snippets should have been omitted. They’re too short to be useful, and they just push up the page count needlessly.

The book has some pretty naive advice that is either clumsy, unrealistic, or just plain ignorant of what usually goes on in industry. When talking about bypassing JavaScript checking, they overlook the ability of Paros (a proxy they mention in other places) to “trap” requests and modify values. Instead, they suggest saving the HTML as a file and editing the JavaScript out by hand, or disabling JavaScript altogether. The former is cumbersome and the latter usually renders a page unusable. They also suggest that it’s better to write software with only one programming language, to avoid cross-language bugs (like NULL character interpretation). They immediately admit that this advice isn’t very practical, so why bother saying it in the first place? It’s kinda like saying “only hire left-handed programmers” or something else equally improbable. Either provide useful advice, or don’t bother.

Occasionally they make absolutely absurd statements. They say that the buffer overflow attack is the easiest of all the attacks to execute. I suppose that’s true if you have no interest in actually ’succeeding’ with it. It’s far more difficult (compared to, say cross-site scripting, SQL injection, and cookie poisoning) to actually exploit. Furthermore, when you’re a black-box tester (i.e. without access to server logs and insider information), determining that the attack string you sent caused a buffer overflow, instead of some other server side error, is not at all obvious in most apps. It’s not like it comes back displaying a web page saying “segmentation violation.” In the handful of places they acknowledge the existance of Unix, they give dumb examples of unix things to do, like looking at the /etc/passwd file (which hasn’t had passwords in it on modern UNIX for nearly 10 years), and running the “finger” command. I mean, really, come on.

My final criticism is that a software tester whose job is testing web software won’t get much out of this book unless she is starting at absolute zero. If you have never tested a web app for security before, this book will introduce a few new concepts. The attack descriptions, though, are brief and impractical. A software tester who has a job to do will want to take the lessons in this book and make regression tests, interactive security tests, and self-contained scripts that can be used to repeatedly demonstrate the performance of a web application. All the advice in this book is for interactive testing in a very ad hoc, one-off sort of way. Systematizing it is an exercise left entirely up to the reader.

Recognize my bias as a reviewer: I am a security consultant who does this stuff for a living, and I am also an author. My standards are high. I also, however, work with and train developers and testers who are not security specialists. The information in this book is beneath any security specialist, but is not sufficiently specific and practical for a software professional. I can’t really see a good audience for this book. For my money, the “Hacking Exposed” book by Scambray, Schema, and Sima is a better value. I have no relationship to them. I just bought both books, read them and like the “Hacking Exposed” book better.