I've taken a preliminary look at these vulnerability reports and some observations jump out:
- The holes were in 42 separate programs, with only 7 programs showing more than one vulnerability and with only one showing more than 2 (CUPS with 4, 3 of which were found by the same person).
- There's an enormous variance in the number of holes found. Only two students, Ariel Berkman and Limin Wang, got 10 vulnerabilities. The next highest was Yosef Klein with 5. Only 17 students found any vulnerabilities at all and 9 of them found 2 or less.
- Only 7 vulnerabilities have more than one author listed, and those have only two authors. This tells us something about the degree of overlap between different auditors, but I'm not sure what because I don't know if the students worked completely independently. I've fired off mail to DJB but don't have a response yet.
- 45 of the vulnerabilities were remotely exploitable in some way. I haven't examined them too carefully, but it looks like a large number of them were some would let you more or less take over the victim's account. My impression is that a lot of these were classic memory handling errors. Note: I didn't really examine the bug reports. To a first order I just used the remote/local categorization in DJB's reports.
Points (2) and (3) are especially interesting for what they suggest about the population of vulnerabilities. It's pretty common for security types (myself included) to assume that software is so bug-riddled that any idiot can find an arbitrary number of vulnerabilities. Obviously, this was quite doable for some people, but others clearly found it very challenging. This project was 60% of the grade in the class, so they clearly had substantial incentive to find them.
On the other hand, the overlap between the vulnerabilities people found (even if we assume they worked totally independently) was quite small. Less than 15% of the vulnerabilities have more than one person listed. A small overlap is not what we'd expect if the reason it was hard to find vulnerabilities was that the total population was very small.1 The fact that some students were so successful suggests that perhaps the limiting factor in finding vulnerabilities isn't that there is a limited number but rather that they are hard to find and some people are just better than others. It would be interesting to know what the two people who found 10 vulnerabilities did differently from everyone else.
1. One caveat here is that I don't know how things were run. If, for instance, the bugs were posted somewhere as soon as they were found and you didn't get credit for finding a duplicate, then you would obviously see very little overlap. However, the fact that DJB obviously submitted the bugs all at the same time suggests otherwise.
The classification of "remotely exploitable" he used is expansive to the point where any buffer overflow in input processing is "remotely exploitable" since the program might be run on untrusted input received via e-mail. Making this argument for something like an assembler seems a little extreme.
Gee, this makes me wonder if vulnerability discovery "in the wild" is a good idea or not ;-). It seems like all the highly-skilled discoverers are doing the work for the "bad guys".