Notes on DJB's vulnerability finding class project

| Comments (2) | TrackBacks (86) |
Via /., I see a story that Dan Bernstein assigned students in his MCS 494: Unix Security Holes class the project of finding 10 exploitable holes each in Unix programs. The class of 25 students found a total of 51 vulnerabilities (in 44 separate reports) which you can find here.

I've taken a preliminary look at these vulnerability reports and some observations jump out:

  1. The holes were in 42 separate programs, with only 7 programs showing more than one vulnerability and with only one showing more than 2 (CUPS with 4, 3 of which were found by the same person).
  2. There's an enormous variance in the number of holes found. Only two students, Ariel Berkman and Limin Wang, got 10 vulnerabilities. The next highest was Yosef Klein with 5. Only 17 students found any vulnerabilities at all and 9 of them found 2 or less.
  3. Only 7 vulnerabilities have more than one author listed, and those have only two authors. This tells us something about the degree of overlap between different auditors, but I'm not sure what because I don't know if the students worked completely independently. I've fired off mail to DJB but don't have a response yet.
  4. 45 of the vulnerabilities were remotely exploitable in some way. I haven't examined them too carefully, but it looks like a large number of them were some would let you more or less take over the victim's account. My impression is that a lot of these were classic memory handling errors. Note: I didn't really examine the bug reports. To a first order I just used the remote/local categorization in DJB's reports.

Points (2) and (3) are especially interesting for what they suggest about the population of vulnerabilities. It's pretty common for security types (myself included) to assume that software is so bug-riddled that any idiot can find an arbitrary number of vulnerabilities. Obviously, this was quite doable for some people, but others clearly found it very challenging. This project was 60% of the grade in the class, so they clearly had substantial incentive to find them.

On the other hand, the overlap between the vulnerabilities people found (even if we assume they worked totally independently) was quite small. Less than 15% of the vulnerabilities have more than one person listed. A small overlap is not what we'd expect if the reason it was hard to find vulnerabilities was that the total population was very small.1 The fact that some students were so successful suggests that perhaps the limiting factor in finding vulnerabilities isn't that there is a limited number but rather that they are hard to find and some people are just better than others. It would be interesting to know what the two people who found 10 vulnerabilities did differently from everyone else.

1. One caveat here is that I don't know how things were run. If, for instance, the bugs were posted somewhere as soon as they were found and you didn't get credit for finding a duplicate, then you would obviously see very little overlap. However, the fact that DJB obviously submitted the bugs all at the same time suggests otherwise.

86 TrackBacks

Listed below are links to blogs that reference this entry: Notes on DJB's vulnerability finding class project.

TrackBack URL for this entry: http://www.educatedguesswork.org/cgi-bin/mt/mt-tb.cgi/41

texas hold em from texas hold em on February 3, 2005 4:50 AM

You are invited to check some helpful info about texas hold em online poker texas holdem Read More

online poker from online poker on February 9, 2005 3:41 AM

You can also check some helpful info in the field of online poker poker Read More

Please check out the pages in the field of online poker loans poker Read More

You can also visit the sites dedicated to Poker Texas holdem Online poker Read More

free online poker from free online poker on February 23, 2005 12:50 PM

A spectre is haunting the world - the spectre of communism. by texas hold'em Read More

Please visit the sites about Texas hold em Empire poker Read More

3 card poker from 3 card poker on March 2, 2005 1:30 PM

In your free time, check the sites about 3 card poker antique poker chips Read More

mortgage loan from mortgage loan on March 4, 2005 9:11 PM

You may find it interesting to check some helpful info in the field of auto loan cash loan Read More

bankruptcy loan from bankruptcy loan on March 14, 2005 5:45 AM

You may find it interesting to check out some helpful info in the field of bankruptcy loan high risk loans Read More

texas hold em from texas hold em on March 16, 2005 4:41 AM

You may find it interesting to check some helpful info about texas hold em party poker texas hold em Read More

party poker from party poker on March 17, 2005 10:11 PM

You may find it interesting to visit the pages in the field of party poker online poker texas hold em Read More

texas holdem poker software from texas holdem poker software on March 19, 2005 12:15 AM

Please visit some relevant pages about full tilt poker 7 card stud poker stratagies Read More

blackjack from blackjack on March 20, 2005 4:02 AM

Please visit some relevant pages about blackjack online pharmacy Read More

internet pharmacy from internet pharmacy on March 23, 2005 11:39 AM

Please check some helpful info dedicated to internet pharmacy pharmacies online Read More

You can also check out some information about roulette blackjack casinos Read More

texas hold em from texas hold em on March 28, 2005 8:46 AM

You may find it interesting to check out some relevant pages dedicated to texas hold em play poker party poker Read More

pokercasino stud 8 or better from pokercasino stud 8 or better on March 29, 2005 3:06 PM

You may find it interesting to visit some helpful info about no download poker texas game Read More

poker tables for sale from poker tables for sale on April 1, 2005 10:29 AM

You can also visit some information about official poker rules free poker online poker games Read More

poker table from poker table on April 4, 2005 1:29 AM

You can also check the sites about on line poker free casino poker table for sale Read More

party poker from party poker on April 8, 2005 1:06 PM

Please check the sites dedicated to party poker poker online poker Read More

pacific poker from pacific poker on April 8, 2005 5:14 PM

You are invited to visit some helpful info about pacific poker texas hold em party poker Read More

party poker from party poker on April 8, 2005 8:23 PM

Please visit some relevant pages in the field of texas hold em online poker party poker Read More

texas holdem from texas holdem on April 8, 2005 9:22 PM

You are invited to check some helpful info about texas holdem online poker pacific poker Read More

online poker from online poker on April 11, 2005 12:52 PM

You are invited to visit some helpful info dedicated to online poker texas holdem Read More

TITLE: Cruises URL: http://123.xmix.net/ IP: 211.98.24.6 BLOG NAME: Cruises DATE: 04/11/2005 12:58:15 PM Read More

texas holdem from texas holdem on April 11, 2005 6:24 PM

You can also check out some relevant pages dedicated to texas holdem online poker poker Read More

TITLE: low blood pressure URL: http://bp.xmix.net/ IP: 202.14.68.240 BLOG NAME: Blood Pressure DATE: 04/13/2005 12:38:56 AM Read More

party poker from party poker on April 14, 2005 12:25 PM

You can also check some relevant pages about how to play texas holdem gambling poker Read More

texas holdem from texas holdem on April 15, 2005 10:55 PM

In your free time, check the sites about pacific poker online poker texas holdem Read More

In your free time, check out some relevant pages in the field of alprazolam Read More

video poker games from video poker games on April 25, 2005 7:31 AM

You are invited to check out some information in the field of texas hold um texas holdem online free $25 Read More

poker games from poker games on April 29, 2005 5:53 PM

In your free time, visit some relevant pages dedicated to online poker Read More

texas holdem from texas holdem on May 4, 2005 11:20 PM

In your free time, visit the sites about texas holdem Read More

texas holdem from texas holdem on May 6, 2005 3:56 AM

Please check the sites dedicated to texas holdem Read More

online poker from online poker on May 24, 2005 2:36 AM

You can also visit the sites about online poker texas hold em Read More

texas hold'em from texas hold'em on June 6, 2005 12:29 AM

Check these: Some texas hold'em poker For example play texas hold'em Read More

texas hold'em from texas hold'em on June 6, 2005 12:32 AM

Check these: Consolidate play texas hold'em Through texas hold'em poker Read More

texas holdem from texas holdem on June 7, 2005 4:29 AM

Check these: Note that poker It must be noted texas holdem Would You online poker Read More

poker rooms from poker rooms on June 8, 2005 1:35 AM

You are invited to visit some information about online poker Read More

free online poker from free online poker on June 9, 2005 12:18 AM

Check these: All the poker online Always free poker online Read More

mortgage payment calculator from mortgage payment calculator on June 10, 2005 8:46 AM

You can also check out the sites about payday loans home equity loan refinance Read More

free poker from free poker on June 20, 2005 9:05 PM

You are invited to check some relevant information in the field of free poker Read More

how to play poker from how to play poker on June 24, 2005 3:19 AM

Take your time to visit the pages about world series of poker free online poker Read More

oxycontin drug abuse withdrawl symptoms from drugs oxycontin amry ucmj oxycontin oxycontin 93 33 snorted sniffing oxycontin oxycontin identification genetic oxycontin oxycontin in mexico oxycontin and methadone oxycontin high generic version of oxycont... Read More

free casino from free casino on July 24, 2005 9:44 AM

You may find it interesting to take a look at some helpful info dedicated to casino chips casino poker chips Read More

In your free time, visit the pages dedicated to pacific poker poker poker games Read More

nudist friendfinder from indiana adult personals on August 26, 2005 10:07 AM

friend finder free hardcore tgp gallery sites adult friend finder butterscup adultfriendfinder aff xxx adult frend finders married friend finder nz bbw friend finder aff friend finder nudist friendfinder the adult frend finder asian adult friend finde... Read More

video poker machine from video poker machine on October 14, 2005 10:53 AM

You may find it interesting to take a look at some information in the field of omaha hi poker Read More

poker table Read More

online pharmacy merchant account, Panama from online pharmacy merchant account, Panama on October 29, 2005 8:58 PM

You may find it interesting to check out the pages about online pharmacy tech degree Read More

Lesbian incest erotic stories from Mother and daughter pictures on November 5, 2005 10:12 AM

Movie teacher fuck boy Old men young women homemade sex videos Daddy mom son spanking pics Prison gang rape pics Read More

health insurance from health insurance on November 8, 2005 3:00 PM

health insurance is a greatblog Read More

christmas cards Read More

Christmas Gift Baskets from Christmas Gift Baskets on November 13, 2005 2:43 PM

plasma tv is a niceblog. Read More

pacific poker bonus from pacific poker bonus on November 20, 2005 1:46 PM

Check these: pacific poker . Read More

party poker room from party poker room on November 21, 2005 5:21 PM

Check these: party poker . Read More

play texas holdem from play texas holdem on November 28, 2005 4:10 PM

Check these: texas holdem pairs . Read More

Samples free sex forced sex from Bestiality incest porno on January 4, 2006 6:46 PM

Desi sexy cartoon Real player porn download Stories of rape with pics Video gay f... Read More

2 Comments

The classification of "remotely exploitable" he used is expansive to the point where any buffer overflow in input processing is "remotely exploitable" since the program might be run on untrusted input received via e-mail. Making this argument for something like an assembler seems a little extreme.

Gee, this makes me wonder if vulnerability discovery "in the wild" is a good idea or not ;-). It seems like all the highly-skilled discoverers are doing the work for the "bad guys".

Leave a comment