November 2009 Archives


November 29, 2009

I'd been mostly avoiding developing an opinion on the H1N1 pandemic, but I recently had someone ask me about it and then watched this ABC reportCogitamus) and figured it might be time to read some of the literature. Your go-to site here for statistics is CDC FluView (the above report complains about availability, but I don't think anyone is claiming that CDC is actually misreporting the data). After reading through CDC's data, the situation seems to be as follows:
  • We're seeing over twice as much "influenza-like illness" (ILI) as normal this year (5.5% of visits versus 2.3% of visits).
  • When tested, the majority of these cases (70+%) aren't actually flu.
  • The vast majority of cases that are actually flu are H1N1.
  • Mortality from pneumonia and influenza is currently at 7.5%, what looks like about 50-75% over the seasonal baseline and over the 6.9% epidemic threshold. So, you're looking at like a 2.5% excess risk of death. This doesn't sound too bad for adults.
  • We're seeing something like 2x ordinary pediatric mortality: 20 deaths in the week Nov 7-14, 138 deaths since August 30 and 200 since April 30. It's difficult to compare here since the CDCs ordinarily mortality data is broken up into different cohorts. Ordinarily there are around 10,000 deaths in the US in 2006 in the 1-14 cohort and 35,000 in the 15-24 cohort [*]. It we crudely divide by 52, we get something like 200 deaths/week in the 1-14 cohort and maybe something like 400 deaths/week in the the 1-17 cohort (assuming its evenly distributed in the 15-24 range). So, this year's flu represents somewhere around 2-10% excess mortality over usual, which doesn't sound that bad.
  • It looks like the rate of flu is coming down both in the US and Canada. The Canadians say they may have reached their "epidemic peak".
  • Quebec appears to be plague central, with the highest rate of hospitalizations of any province in Canada.

I'm most struck by how high a fraction of people with disease suspicious enough to be tested actually turn out not to have flu. This isn't just a case of US health care overtreatment, either, The Canadian positive rate is a little over 40%. That said, the rate of positives varies really dramatically from region to region, with a low of 13.3% in region 6 (AR, LA, NM, OK, TX) and a high of 51.8 in Region 3 (DE, DC, MD, PA, VA, WV), but it's hard to tell (or rather I'm too lazy to run the numbers) if it's just a matter of basic incidence or of the amount of testing being done.

One question people seem to want to ask is: should I get the vaccine? I know there are concerns about side effects, though as far as I can tell, there isn't much to be worried about (you can find the CDC party line here and Wikipedia's rundown here): the US versions of the H1N1 vaccine seem to be made with the same methods as the seasonal vaccine, so if you were happy to get that, you should be OK with the H1N1 vaccine too. [There have been concerns about the vaccines used outside the US which contain adjuvants, and I haven't developed an opinion on that.].

As far as effectiveness goes, the studies that are available seem to use the stimulation of an immune response rather than actually getting H1N1 as a study endpoint. This is understandable, since randomized controlled trials with infection as the endpoint are slower and more expensive to do, but it makes the data less useful for decision making than it otherwise would be. This Atlantic article argues that the overall evidence for flu effectiveness is thin, but compare this CDC survey which seems to indicate effectiveness in the 55-85% range. Anyway, if we assume that your chance of getting the flu is 5-20% and everyone gets vaccinated, then we'll see something like a 3.5-15% reduction in disease cases (number to treat = 6-30). On the other hand, since apparently most cases of ILu aren't actually flu, the improvement in your chance of experiencing some sort of flu-like crud (which is after all what you care about) are only like 1-5%, ignoring, of course, that flu season seems to be mostly over now.


November 28, 2009

So I heard on the radio today that some Americans were killed in a plane crash. In China... On a Zimbabwean cargo airline... On its way to Kyrgyzstan.

Unfortunately, the spoilsports at CNN now report that the plane was on its way to Harare, which is rather less interesting.

Some non-blog materials I never bothered to point to:

One of my better party invitations

My standard introduction to communications security entitled SSH, SSL, and IPsec: wtf?


November 25, 2009

I like using history podcasts as background for commuting, hiking, etc. My favorites come from Isabelle Pafford a lecturer in Classics at Santa Clara University (previously at Berkeley and SFU). Pafford has a light, entertaining style, full of jokes and pop culture references.

Classics 11A: Heroes and Heroism
Classics 178: Greek Art and Architecture

Older:History 110: Roman Republic
Hitory 106B: The Roman Empire
History 4A: The Ancient Mediterranean World

Worth checking out.


November 22, 2009

It is a truth universally acknowledged [at least outside the US] that American chocolate sucks. This observation serves as synechdoche for the general sense that Americans are philistines who are only interested in McDonalds and Budweiser. In the particular case of chocolate, however, the story is rather more interesting. The dominant chocolate in the US is Hersheys and for historical reasons Hersheys has a very distinct flavor:
Hershey process milk chocolate, invented by Milton S. Hershey, founder of The Hershey Company, can be produced more economically since it is less sensitive to the freshness of the milk. Although the process is still a trade secret, experts speculate that the milk is partially lipolyzed, producing butyric acid, which stabilizes the milk from further fermentation. This compound gives the product a particular sour, "tangy" taste, to which the American public has become accustomed, to the point that other manufacturers now simply add butyric acid to their milk chocolates.

As with beer (another oft-maligned American product which I'll get to in a minute), it's not like Americans don't know how to make chocolate that's good by non-US standards (exhibits A-G: Scharffen Berger [now owned by Hersheys, btw.], Guittard, Devries, Askinosie, Amano, Patric, Tcho), it's just a case of path dependence. Americans are used to a particular flavor, and as the passage above suggests, actively reject chocolate without that. Interestingly, I just got back from the UK, where I picked up a few bars of the legendary Cadbury Dairy Milk only to find it not at all as I remembered. Terence Spies suggests that this may be a problem in product handling but the above Wikipedia article suggests that they may have actually screwed with the formula, which, if true, is baffling.

I spent a lot of my time in the UK drinking and came away with a single observation: British beer is astonishingly bad, overly sweet, flat, thin, and almost utterly without hop flavor. I was mostly drinking local beers but I also tried Guinness and Bass, both of which I've had in the US, and while the Guinness was all right, the Bass seemed to differ from the US version in the same way as all the others. I attribute part of the difference here to the temperature (British beers are served far warmer) and lower level of carbonation due to cask conditioning, with the resulting lower level of carbonic acid. But the primary difference seems to be far less hops, both in terms of bittering and aroma; conversely my British companion informed me that they find American beers way too hoppy. (For the record, when I was in Germany and the Czech Republic, I found their beers perfectly fine). I don't have a complete theory for this, but I'm guessing it's another path dependence issue, in this case due to prohibition gutting the American alcohol market and leaving only a small number of breweries focusing on Czech pilsner style beers. This created a population bottleneck, and when the beer market in the US re-expanded with the American craft brew movement in the 1980s and 1990s it drifted in a totally different direction from that found in its ancestral population, one focusing on much higher levels of bitterness and hoppiness.

I don't intend to offer a defense of McDonalds, however.


Acknowledgement: This post produced with assistance from Terence Spies of Cacaolab, who originally told me about the Hershey process.


November 21, 2009

Search and Rescue (SAR) workers are complaining that hikers overuse personal locator beaconsPhilosoraptor). Here's one of the cases they site:
The Grand Canyon's Royal Arch loop, the National Park Service warns, "has a million ways to get into serious trouble" for those lacking skill and good judgment. One evening the fathers-and-sons team activated their beacon when they ran out of water.

Rescuers, who did not know the nature of the call, could not launch the helicopter until morning. When the rescuers arrived, the group had found a stream and declined help.

That night, they activated the emergency beacon again. This time the Arizona Department of Public Safety helicopter, which has night vision capabilities, launched into emergency mode.

When rescuers found them, the hikers were worried they might become dehydrated because the water they found tasted salty. They declined an evacuation, and the crew left water.

The following morning the group called for help again. This time, according to a park service report, rescuers took them out and cited the leader for "creating a hazardous condition" for the rescue teams.

This does seem pretty egregious. That said, it's probably the most egregious case the reporter was able to find, so it's worth asking what fraction of times people light off their PLBs actually are inappropriate. Reading this article, one sort of gets the impression that SAR would prefer that people didn't really have PLBs at all:

"There's controversy over these devices in the first place because it removes the self sufficiency that's required in the back country," Scharper says. "But we are a society of services, and every service you need you can get by calling."


"In the past, people who got in trouble self-rescued; they got on their hands and knees and crawled out," says John Amrhein, the county's emergency coordinator. "We saw the increase in non-emergencies with cell phones: people called saying 'I'm cold and damp. Come get me out.' These take it to another level."

Obviously, I don't know the details of any of these cases, but it's not clear to me that having people crawling out of the backcountry on their hands and knees is really desirable (and it's pretty clear to me that people having to cut their own arm off because they couldn't call for rescue isn't really optimal). As for being "cold and damp", that covers a lot of territory from "vaguely damp" to "severe hypothermia". Hypothermia is no joke: if you fall into a river and all your gear is down and gets soaked and it's going to be sub-freezing at night, it might be very appropriate to call for rescue (though I should mention that you might not get it in time depending on how far out you are).

It seems to me that there are really two separate issues here:

  1. People calling for help in situations that are inconvenient rather than emergent.
  2. People acting in unsafe ways because they think they can rely on the beacon to get them out of trouble.

The first issue seems to me to be at least partly one of incomplete communication: the major existing beacons just send out a homing signal, so SAR has no idea what's wrong with you and has to come to you in order to find out whether you've broken both legs or just run out of beef jerky. On the other hand, an interactive channel such as a cell phone would allow SAR to find out what the problem is and determine how to prioritize/triage their response, something that first responders already have to do in response to 911 calls. In the limit, they could just say "this isn't important enough, we're not coming." We're already seeing some movement in this direction: the SPOT messenger allows preprogrammed messages to be sent, but isn't interactive and doesn't allow arbitrary messages. What you really want is a PLB combined with a sat phone, so you could have enough battery for something like 10 minutes of sat phone conversation and 24 hours of beacon transmit time. I imagine we'll see something like this eventually, at which point problem (1) will be a lot simpler.

I have mixed feelings about problem (2). One the one hand, if you go out into the wilderness with nothing but a leatherman and a tube of sunscreen, you pretty much deserve whatever happens to you. On the other hand, one does get a sense that a lot of the people complaining here feel that danger ought to be an inherent part of the wilderness experience, which is a position I don't endorse. I'm not saying that it's something people should necessarily shun: if you want to go backpacking without the means to call for backup that's your business [and I've done that plenty of times, so I'm not saying it's that dangerous, certainly not the way rock climbing without a rope is, for instance], but that doesn't mean that it's the only option that should be open to people. One of the great things about technology is that it makes things that were previously unsafe safer and I'm not sure that it's a great idea to force people to reject that in favor of some vision of what an authentic experience is.

The natural solution is of course to charge people for rescue (in economics rational-choice land we would charge everyone a Pigouvian tax for rescue and let them purchase insurance; back in reality, we're not going to let people die of exposure on some mountain because they won't fork over $10k for a helicopter). As I understand it, some SAR organizations charge people for rescues when it's felt that they weren't prepared enough. I'm not sure what kind of fee structure makes sense here: ideally you would scale the fee to how much the rescue cost and how unprepared people seemed to be, but maintaining that sort of system (in particular investigating exactly what people did and didn't do to prepare) seems like it's probably impractical. Maybe what's needed here is just a medium sized fee (~1-5k); enough to make people think but not enough to deter people from calling for help when they're in serious trouble.


November 20, 2009

Sequoia Voting Systems recently announced that it will be publishing the source code to their Frontier opscan voting system. Reaction in the security community seems generally positive. Here's Ed Felten:
The trend toward publishing election system source code has been building over the last few years. Security experts have long argued that public scrutiny tends to increase security, and is one of the best ways to justify public trust in a system. Independent studies of major voting vendors' source code have found code quality to be disappointing at best, and vendors' all-out resistance to any disclosure has eroded confidence further. Add to this an increasing number of independent open-source voting systems, and secret voting technologies start to look less and less viable, as the public starts insisting that longstanding principles of election transparency be extended to election technology. In short, the time had come for this step.

I'm less sanguine. I'm not saying this is a bad thing necessarily, but I'm not sure it's a good thing either. As always, it's important to consider what threats we're trying to defend against. We need to consider two kinds of vulnerabilities that might be present in the code:

  • Backdoors intentionally introduced by Sequoia or their engineers.
  • Design and or implementation errors accidentally introduced by Sequoia engineers.

A lot of the focus on open voting systems has focused on the first kind of threat (corporations are stealing your votes, etc.) I think there's certainly a credible argument to be made that having to publish the source code does make this form of attack somewhat harder. If people are looking at your code, then you probably can't put a naked backdoor ("if someone types 1111, give them operator control") into the code because that might get caught in a review. On the other hand, it would be a pretty risky move to put that kind of backdoor into a piece of software anyway, since even closed voting source code does get reviewed, both as part of the system certification process and in private reviews like those conducted by Califoria and Ohio. More likely, you'd want to hide your back door so it looked like an accidentally introduced vulnerability, both to make it harder to find and to give you plausible deniability.

This brings us to the second form of vulnerability: those introduced as errors/misfeatures in Sequoia's development process. These aren't necessarily a sign of incompetence; as Steve Bellovin says, "all software has bugs and security software has security relevant bugs." Having access to the source code makes it easier to find those vulnerabilities (though as Checkoway et al. have shown it's quite possible to find exploitable vulnerabilities in voting systems without access to the source code). This of course applies both to attackers and defenders. There's an active debate about whether or not on balance this makes open source inherently more or less secure. I'm not aware of any data which settles the question definitively, but I don't think that anyone in the security community believes that a previously insecure piece of software will suddenly become substantially more secure just because the source is disclosed; there are just too many security vulnerabilities for the sort of low-level uncoordinated review that you get in practice to stamp out. On the other hand, it does provide a pretty clear near-term benefit to attackers, who, after all, just need to find one vulnerability.

Now, that's not what Sequoia is doing. According to their press release, Frontier is an entirely new system which they say has been "developed from the ground up with the full intention of releasing all of the source code to any member of the public who wishes to download it - from computer scientists and election officials to students, security experts and the voting public". This syncs up better with another argument about why openness is important, which is more about incentives: if vendors know their code will be open to scrutiny they will be motivated to be more careful with their designs and implementations. Reviews of Sequoia's previous systems have been pretty negative; it will be interesting to see if the new one is any better. On the other hand, we have the confounding factor that modern standards for what it means for a piece of software to be secure are a lot higher than those which applied when the original SVS system was written, so it will be hard to tell whether it's really openness which provided the value, or just that they started from scratch.

One more thing: suppose that the source code is published and the code is full of problems. What then?


November 7, 2009

I happened to be leafing through Stroustrup and noticed that you can overload <code>< and >. This motivated me to write the following program:
#include <iostream>
#include <vector>
typedef int UINT4;

using namespace std; 

class Hack 

Hack & operator< (Hack &a , Hack &b)
 std::cerr << "LT operator\n";

 return a;

Hack & operator> (Hack &a, Hack &b)
 std::cerr << "RT operator\n";

 return a;

int main(int argc, char ** argv)
 Hack vector;
 Hack UINT4;
 Hack foo;
 vector<UINT4> foo;

Ask yourself what this code does.

The answer is that it outputs:

LT operator
RT operator

If you focus just on the line vector<UINT4> foo; this looks like a relatively ordinary template instantiation of a vector of type UINT4. This is perfectly normal C++ stuff. However if we expand the scope, it becomes clear that something different is going on: we've defined a new class called Hack and vector, UINT4, and foo are actually objects of type Hack. We've also overloaded the < and > operators. So, what's actually happening here is that we are doing function chaining: We perform operator > on the pair of objects UINT4 and foo. This returns a temporary object of type Hack (in this case the first argument but it doesn't matter). We then perform operator < on set and the temporary variable. And of course since these operators are just function calls, we can do any work we want in them. The examples print stuff to stderr, but that's just an example; you could do anything. And of course this code was written to be moderately transparent while making the point. You could obfuscate it much further with a little effort.



Acknowledgment: Steve Checkoway pointed out to me that whatever crazy type resolution rules C++ follows here make the code work even with the definition of vendor and UINT4 at the top of file. My original version didn't have these and so alleged vector declaration in main wasn't really valid without the definition of Hack

UPDATE Oh great. HTML screws up anything with <foo>. Fixed now.


November 5, 2009

Marsh Ray has published a new attack on the TLS renegotiation logic. The high level impact of the attack is that an attacker can arrange to inject traffic into a legitimate client-server exchange such that the TLS server will accept it as if it came from the client. This may allow the attacker to execute operations on the server using the client's credentials (e.g., order a pizza as the client). However, the attacker does not (generally) get to see the response. Obviously this isn't good, but it's not the end of the world. More details below.

TLS Details
The attack exploits TLS's renegotiation feature, which allows a client and server who already have a TLS connection to negotiate new parameters, generate new keys, etc. Renegotiation is carried out in the existing TLS connection, with the new handshake packets being encrypted along with application packets. The difficulty is that they're not otherwise tied to the channel, which gives the attacker a window. The simplest form of the attack is shown below [cribbed from draft-rescorla-tls-renegotiate.txt]

Client                        Attacker                        Server
------                        -------                         ------
                                  <----------- Handshake ---------->
                                  <======= Initial Traffic ========>
<--------------------------  Handshake ============================>
<======================== Client Traffic ==========================>

So, in order to mount the attack, the attacker first connects to the TLS server. He can communicate with the server as much as he wants, including making an arbitrary number of requests/responses, etc. This traffic is all encrypted and shown as ==. Then when he's ready, he hijacks the client's connection to the server (in practice he might start by hijacking the connection and then connect to the server but it doesn't matter) and just proxies the client's traffic over the encrypted channel. The client negotiates with the server and from that point on the client and the server communicate directly. Note that the client is communicating with the attacker in the clear but the second handshake is encrypted and goes over the attacker's channel. Thus, the client does not know that he is renegotiating. However, the server thinks that the initial traffic with the attacker is also from the client. There are also other (probably less useful) variants where both sides see a renegotiation but of different connections.

Impact on Existing Applications
TLS itself is just a security protocol, so the impact of this attack depends on the application protocol running over TLS. The most important of these protocols is of course HTTP over TLS (HTTPS). Most Web applications do initial authentication via a username/password pair and then persist that authentication state with HTTP cookies (a secret token that is sent with any request). An attacker might exploit this issue by sending a partial HTTP request of his own that requested some resource. This then gets prefixed to the client's real request.

E.g., the attacker would send:

GET /pizza?toppings=pepperoni;address=attackersaddress HTTP/1.1 

And leave the last line empty without a carriage return line feed. Then when the client makes his own request

GET /pizza?toppings=sausage;address=victimssaddress HTTP/1.1 
Cookie: victimscookie

the two requests get glued together into:

GET /pizza?toppings=pepperoni;address=attackersaddress HTTP/1.1 
X-Ignore-This: GET /pizza?toppings=sausage;address=victimssaddress HTTP/1.1 
Cookie: victimscookie

And the server uses the victim's account to send a pizza to the attacker.

You can mount a similar attack if the server uses certificate-based client authentication: it's common (well, common in the small number of cases where client certs are used at all) for the server to let the client connect and request a resource and then if the resource is protected renegotiate asking for a certificate. The attacker can exploit this by doing the initial handshake and asking for the resource and then letting the client do the renegotiation, at which point the server acts as if the initial request was made by the client.

It's important to note that in both the cases we just described the attacker doesn't get to see any sensitive information directly: that's just sent back encrypted to the client. However, he can exploit side effects of the exchange, e.g., to get a pizza. It may also be possible to exploit HTTP features to directly access the data. For instance, he might be able to generate a combined request that would cause the server (or perhaps somehow mirrored through the client) to send the original client request to the attacker. If the request contains the client's cookie or password, this would cause their credentials to leak. It's not clear to me if this is possible, but I'm hoping some of the Web security specialists will weigh in.

The impact on other protocols (IMAP, SIP, etc.) would depend on those protocols and I haven't seen complete analyses of these yet.

Eventually there will be a TLS level protocol fix (see below). However, in the meantime options are limited.

For 99+% of applications, the mitigation is very simple: the server should simply disable all renegotiation, which stops the attack. (OpenSSL will helpfully automatically renegotiate which facilitates the attack even if the application isn't otherwise set up to do renegotiation). Unfortunately, there is no similar defense on the client side. In the example I showed above, the client is unaware that renegotiation happened. Moreover, the client can't tell that the server will refuse renegotiation (it could explicitly probe but the attacker could of course fake a failure). So, the client can't really do anything useful to protect itself.

The second drawback is that there are a small number of cases (e.g., the certificate authentication one I described above) where renegotiation actually should happen. The most practical defense on the server side is to restructure the site so that requests which require client auth are redirected to a different address or port which always requests a certificate and itself refuses renegotiation. However, this obviously requires major work on many sites.

There are a few other legitimate reasons for renegotiation, but they're mostly things one doesn't need to do. For instance, people sometimes renegotiate to force the generation of fresh keying material. This is not necessary with modern cryptographic algorithms. Another example provided by Ray is sites that support algorithms with differential strength. Again, this isn't really necessary. Unless you're doing certificate-based client authentication it's very unlikely you need to renegotiate and if you are using it, the workaround above is the simplest.

Long-Term Defense
Despite the existence of some defenses, it seems clear that TLS should really defend against this. There's a fairly obvious defense that at least three people have independently come up with: carry information about handshake n-1 in handshake n (if n==1 you just carry an empty field). This prevents attacks where the client and server have different views of the negotiation state. The predisclosure group that worked on this after Ray discovered this worked to develop a draft describing this technique which can found at draft-rescorla-tls-renegotiate.txt and we'll be submitting it to the TLS working group shortly. Of course, this will take a very long time to make it into enough clients and servers to make a real difference, so for now we're mostly stuck with the mitigations I've described.

The Software Situation
People have also developed patches for existing implementations. These fall into two categories:

  • Just disable renegotiation (really important for OpenSSL because of the aforementioned automatic renegotiation) issue.
  • Implement the long-term fix.

As far as I know none of these have been formally released yet. Ben Laurie's OpenSSL disabling patch can be found here. I've developed a patch that performs the long-term fix protocol pieces but it doesn't disable non-fixed renegotiation so it's usable primarily for interop testing right now. That can be found here. Expect both of these to eventually appear in official OpenSSL distributions/patches. I don't know the release status of other fixes.

Big Picture Perspective
Now that we've covered how this attack works, how bad is it? It's being billed as a man-in-the-middle attack, but it's really far more limited than your classic MITM. Rather, it's a plaintext injection attack. I'm not saying that's good, but we need to put it into perspective: over the past two years, we have seen two separate issues that allow a complete TLS server impersonation: the Debian PRNG bug (see our paper here for a description of the issue [which we did not discover] and its impact) and the TLS certificate "NULL" issue. Both of these issues had much more severe impact since they allowed the attacker access to all the data sent by the client. However, it's worth noting that they differed from this issue in that they were detectable/fixable on the client side so people could in principle protect themselves. As a practical matter, however, most clients never checked the Debian keys against a blacklist, so that protection was minimal at best.

Second, it's important to remember that even simple MITM attacks with fake certificates don't seem that hard to mount. Cranor et al. report that users very frequently ignore browser warnings of bad certificates, which more or less obviates any security protections TLS provides. If the user will accept the attacker's certificate, there's no need to do anything as limited as the attack described here. Obviously, that's not information that should give you a huge amount of confidence in any form of Internet communications security, but it does provide some perspective on the marginal impact of this attack.

One other comparison point is a remote vulnerability in the server. TLS implementations have historically had and continue to have remote vulnerabilities which would allow an attacker to subvert the server. Such an attacker could read all your traffic to that server. As with this issue, you can't really detect that from the client side and need to rely on the server being upgraded. So, this issue is somewhat less serious than a remote vulnerability in a really popular server stack like OpenSSL. It's primarily unusual because it's an issue in the protocol not the software and those are quite rare.

Bottom Line
As I said at the beginning, this obviously isn't good. We'd like SSL/TLS to provide it's nominal security guarantees and it clearly doesn't. There likely to be exploitable attack paths using this issue. It's far less clear whether they will actually see exploitation, given that these attacks aren't necessarily as powerful as other already known paths. In the not too distant future you should expect most important servers to have fixed, at which point most of your Web server interactions will be safe (even though you can't verify it easily). In the long term, most clients and servers will adopt the protocol fix at which point clients will be able to be sure that the servers they are interacting with are safe.