Recently in Networking Category

I've complained before about Farhad Manjoo's shallow analysis of the social implications of technical decisions, which seems to begin and end with what would be convenient for him. His latest post is an argument against anonymous comments on Internet forums/message boards/etc. Manjoo writes:

I can't speak for my bosses, who might feel differently than I do. But as a writer, my answer is no-I don't want anonymous commenters. Everyone who works online knows that there's a direct correlation between the hurdles a site puts up in front of potential commenters and the number and quality of the comments it receives. The harder a site makes it for someone to post a comment, the fewer comments it gets, and those comments are generally better.

I can appreciate how Manjoo might feel like that. No doubt as a writer it's annoying to get anonymous people telling you that you suck (and much as I find Manjoo's writing annoying, I'm forced to admit that even good writing gets that sort of reaction from time to time). However, this claim simply isn't true—or at least isn't supported by any evidence I know of—to the contrary, the Slate comments section (which Manjoo endorses later in his article) isn't really that great and one of the most highly regarded blog comment sections, Obsidian Wings is almost completely anonymous (though moderated), with the only barrier to posting being a CAPTCHA. Similarly, some of the most entertaining pure-comments sites such as Fark only require e-mail confirm, which, as Manjoo admits, is virtually anonymous. I don't really know everything that makes a good comments section work, but it's a lot more complicated than just requiring people to use their real names.

I think Slate's commenting requirements-and those of many other sites-aren't stringent enough. Slate lets people log in with accounts from Google and Yahoo, which are essentially anonymous; if you want to be a jerk in Slate's comments, create a Google account and knock yourself out. If I ruled the Web, I'd change this. I'd make all commenters log in with Facebook or some equivalent third-party site, meaning they'd have to reveal their real names to say something in a public forum. Facebook has just revamped its third-party commenting "plug-in," making it easier for sites to outsource their commenting system to Facebook. Dozens of sites-including, most prominently, the blog TechCrunch-recently switched over to the Facebook system. Their results are encouraging: At TechCrunch, the movement to require real names has significantly reduced the number of trolls who tar the site with stupid comments.

This is an odd claim since Facebook actually makes no real attempt to verify your full name. Like most sites, they just verify that there is some e-mail addres that you can respond at. It's not even clear how Facebook would go about verifying people's real names. Obviously, they could prune out people who claim to be Alan Smithee, (though consider this) but the world is full of real John Smiths, so why shouldn't I be another one of them?

What's my beef with anonymity? For one thing, several social science studies have shown that when people know their identities are secret (whether offline or online), they behave much worse than they otherwise would have. Formally, this has been called the "online disinhibition effect," but in 2004, the Web comic Penny Arcade coined a much better name: The Greater Internet Fuckwad Theory. If you give a normal person anonymity and an audience, this theory posits, you turn him into a total fuckwad. Proof can be found in the comments section on YouTube, in multiplayer Xbox games, and under nearly every politics story on the Web. With so many fuckwads everywhere, sometimes it's hard to understand how anyone gets anything out of the Web.

I don't disagree that this is to some extent true, though I would observe that (a) the link Manjoo points to doesn't actually contain any studies as far as I can tell, just an article oriented towards the lay public and (b) it's not clear to what extent people's bad online behavior is a result of anonymity. Some of the most vicious behavior I've seen online has been on mailing lists where people's real-world identities (and employers!) are well-known and in some cases the participants actually know each other personally and are polite face-to-face.

As I said above, I don't think anyone really knows exactly what makes a good online community (though see here for some thoughts on it by others), but my intuition is that it's less an issue of anonymity than of getting the initial culture right, in a way that it resists trolling, flamewars, etc., or at least has a way to contain them. In comments sections that work, when someone shows up and starts trolling (even where this is easy and anonymous), the posters mostly ignore it and the moderators deal with it swiftly, so it never gets out of hand. Once the heat gets above some critical point on a regular basis, though, these social controls break down and it takes a really big hammer to get things back under control. It's not clear to me that knowing people's real names has much of an impact on any of that.

Slate has published another Farhad Manjoo screed against unlimited Internet service.

And say hooray, too, because unlimited data plans deserve to die. Letting everyone use the Internet as often as they like for no extra charge is unfair to all but the data-hoggiest among usand it's not even that great for those people, either. Why is it unfair? For one thing, unlimited plans are more expensive than pay-as-you-go plans for most people. That's because a carrier has to set the price of an unlimited plan high enough to make money from the few people who use the Internet like there's no tomorrow. But most of us aren't such heavy users. AT&T says that 65 percent of its smartphone customers consume less than 200 MB of broadband per month and 98 percent use less than 2 GB. This means that if AT&T offered only a $30 unlimited iPhone plan (as it once did, and as Verizon will soon do), the 65 percent of customers who can get by with a $15 planto say nothing of the 98 percent who'd be fine on the $25 planwould be overpaying.

This seems extremely confused. First, it's generally true that whenever a business offers a limited number of product offerings with each at a fixed price that some people overpay because they only want some cheaper offering that the company doesn't provide. For instance, when I bought my last car, Audi insisted on selling me the "winter sports package" (heated seats and a ski bag). Now, I don't do a lot of skiing and I didn't want either but thats the way the thing came. Now by Manjoo's logic, it was unfair that I had to pay more for a ski bag I would never use (the heated seats are great, by the way) but that's just the way the product comes. Sure, I'd rather the company offered exactly the package I wanted but a limited number of offerings is just a standard feature of capitalism.

Its worth observing that there's nothing special about the "unlimited" plan in Manjoo's logic (It's not really unlimited anyway, since the network has some finite amount of bandwidth available so that provides a hard upper limit on how much data you can transfer in a month; it's just that that limit is really high.) Say Verizon offered only a 2GB plan, would he be whining that he only used 200 MB of bandwidth and so he was being made to overpay so Verizon can make money on the 2GB-using bandwidth hogs? So, this objection is pretty hard to take seriously.

Manjoo goes on:

But it's not just that unlimited plans raise prices. They also ruin service. Imagine what would happen to your town's power grid if everyone paid a flat rate for electricity: You and your neighbors would set your thermostats really high in the winter and low in the summer, you'd keep your pool heated year-round, you'd switch to plug-in electric cars, and you'd never consider replacing your ancient, energy-hogging appliances. As a result, you'd suffer frequent brownouts, you'd curse your power company, and you'd all wish for a better way. Economists call this a tragedy of the commons, and it can happen on data networks just as easily as the power grid--faced with no marginal cost, it's in everyone's interest to use as much of the service as they can. When that happens, the network goes down for everyone.

So, first this is just wrong: it's actually reasonably common for utilities to be included in people's leases and yet when that happens people don't automatically switch to plug-in cars or start up home aluminum refineries. That isn't to say that at having to pay for each watt of power doesn't have some impact on your consumption, but there is only so much power that it's really convenient for people to use; it's not like power being free causes consumption to spin off into infinity. To take another example, it's absolutely standard for local voice telephony service to be sold flat rate and yet practically nobody leaves their phone line tied up 24x7 just in case they want to say something to Mom and don't feel like taking the trouble to dial the phone. (Full disclosure, I actually have used dialup internet as a replacement for a leased line this way, but that's a pretty rare use case.)

The second problem with this claim is that computer networks don't behave the way the electrical grid does in the face of contention. Like the electrical grid, computer networks are sized for a certain capacity, but unlike the grid, computers aren't built with the assumption that that capacity is effectively infinite. If the electrical grid in your area is operating at full capacity, and you turn on your AC, this can cause a brownout because there is no way for the power company to tell everyone to use 1% less power and even if there was, many of the devices in question are just designed to operate in a way where they draw constant power. By contrast, computer network protocols are already designed to operate in conditions where they can't use as much bandwidth as they would like because non-infinite bandwidth is a basic feature of the system. Even if there is no contention for the network, applications need to work behind a variety of connection types so people who build applications typically build them to automatically adapt to how much throughput they are actually getting. For instance, Netflix has adaptive streaming which means that it tries to detect how fast your network is and if it's slow it compresses the media harder to reduce the amount of data to send. What this means is that unlike the electrical grid where your computer may just crash if it doesn't get enough power, if the network suddenly gets slower, performance degrades relatively smoothly.

The second thing you need to know is that in data networks congestion is (almost) the only thing that matters. If nobody else is trying to use the network right now then it's fairly harmless if you decide to consume all the available capacity. What's important is that when other people do want to use the network you back off to give them room. So, to the extent to which there is a scarce resource it's not total download capacity but rather use of the network at times when it's actually congested. To a great extent network protocols (especially TCP) already do attempt to back off in the face of congestion but there's also nothing stopping the provider from deliberately imposing balance on you (cf. fair queueing). In either case, this is a relatively orthogonal issue to the volume of data transferred; a cap on total transfer is an extremely crude proxy for the kind of externality Manjoo is talking about. Not only is it crude, it's inefficient: it discourages use of the network which would be cost-free for others and of value to the customer using the network.

All this stuff has of course been hashed out endlessly in the networking economics literature and the above is only the barest sketch. Suffice to say that just applying this sort of naive "tragedy of the commons" analysis doesn't really get you very far.

David Coursey complains about how long it took IEEE to develop 802.11n:

802.11n is the poster child for a standards process gone wrong. Seven years after it began and at least two years after 802.11 "draft" devices arrived, the IEEE has finally adopted a final standard for faster, stronger, more secure wireless.

Ideally, standards arrive before the products that implement them. However, the IEEE process moved so slowly that vendors adopted a draft standard and started manufacturing hardware. After a few little glitches, the hardware became compatible and many of us have--for years--been running multivendor 802.11n networks despite the lack of an approved standard.

...

If standards bodies expect to be taken seriously, they need to do their work in reasonable periods. Releasing a "final" standard long after customer adoption has begun is not only anti-climatic but undercuts the value of the standards making process.

In this case, the process failed. The IEEE should either improve its process or get out of the way and left industry leaders just create de facto standards as they see fit. That is not preferable, but if the IEEE process is stuck, it will be what happens.

My experience with IEEE standards making is limited, but I have extensive experience with IETF's process, and I'm a little puzzled as to what Coursey thinks the problem is here. Developing standards is like developing any other technical artifact: you start out with an idea, do some initial prototypes, test those prototypes, modify the design in response to the testing, and iterate till you're satisfied. Now, in the case of a protocol standard, the artifact is the document that defines how implementations are supposed to behave, and the testing phase, at least in part, is implementors building systems that (nominally) conform the the spec and seeing how well they work, whether they interoperate, etc. With any complicated system, this process needs to include building systems which will be used by end-users and seeing how they function in the field. If you don't do this, you end up with systems which only work in the lab.

There's not too much you can do to avoid going through these steps; it's just really hard to build workable systems without a certain level of testing. Of course, that still leaves you with the question of when you call the document done. Roughly speaking, there are two strategies: you can stamp the document "standard" before it's seen any real deployment and then churn out a revision a few years later in response to your deployment experience. Alternately, you can go through a series of drafts, refining them in response to experience, until eventually you just publish a finished standard, but it's based on what people have been using for years. An intermediate possibility is to have different maturity levels. For instance, IETF has "proposed standards", "draft standards", and then "standards". This doesn't work that well in practice: it takes so long to develop each revision that many important protocols never make it past "proposed standard." In all three cases, you go through mostly the same system development process, you just label the documents differently.

With that in mind, it's not clear to me that IEEE has done anything wrong here: if they decided to take the second approach and publish a really polished document and 802.11n is indeed nice and polished and the new document won't need a revision for 5+ years, then this seems like a fairly successful effort. I should hasten to add that I don't know that this is true: 802.11n could be totally broken. However, the facts that Coursey presents sound like pretty normal standards development.

One of the results of Joe Wilson (R-South Carolina) calling President Obama a liar on national TV was that money started pouring in, both to Wilson and his likely opponent in 2010 (Rob Miller). Piryx, who hosts Wilson's site, claims that on Friday and Saturday they were then subject to a 10 hour DoS attack against their systems:

Yesterday (Friday) around 3:12pm CST we noticed the bandwidth spike on the downstream connections to Piryx.com server collocation facility. Our bandwidth and packet rate threshold monitors went off and we saw both traditional DOS bandwidth based attacks as well as very high packet rate, low bandwidth ICMP floods all destined for our IP address.

...At this point we have spent 40+ man hours, with 10 external techs fully monopolized in researching and mitigating this attack.

To give a sense of scale, the attacks were sending us 500+ Mbps of traffic, which would run about $147,500 per month in bandwidth overages.

I think most people would agree that technical attacks on candidates Web sites, donation systems, etc. aren't good for democracy—just as it would be bad if candidates were regularly assassinated—and it would be good if they didn't happen. While there are technical countermeasures against, DoS, they're expensive and only really work well if you have a site with a lot of capacity so that you can absorb the attack, which isn't necessarily something that every HSP has.

This may turn out to be a bad idea, but it occurred to me that one way to deal with this kind of attack might be for the federal government to simply run its own HSP, dedicated solely to hosting sites for candidates and to accepting payments on their behalf. Such a site could be large enough—though compared to big service providers, comparatively small—to resist most DoS attacks. Also, to the extent to which everyone ran their candidate sites there, it would remove the differential effect of DoS attacks: sure you can DoS the site, but you're damaging your own preferred candidate as much as the opposition. Obviously, this doesn't help if the event that precipitates the surge of donations massively favors one side, but in this case, at least, both sides saw a surge. I don't know if this is universally true though.

Of course, this would put the site operator (either the feds or whoever they outsourced it to) in a position to know who donated to which candidate, but in many cases this must be disclosed anyway, and presumably if the operation was outsourced, one could put a firewall in to keep the information not subject to disclosure away from the feds.

Hovav Shacham just alerted me to an Internet emergency: AT&T is blocking 4chan. I don't know any more than you, but I think it's probably time to upgrade to threatcon orange.

Ed Felten writes about the economic forces that drive cloud computing, arguing that a prime driver is the desire to reduce administrative costs:

Why, then, are we moving into the cloud? The key issue is the cost of management. Thus far we focused only on computing resources such as storage, computation, and data transfer; but the cost of managing all of this -- making sure the right software version is installed, that data is backed up, that spam filters are updated, and so on -- is a significant part of the picture. Indeed, as the cost of computing resources, on both client and server sides, continues to fall rapidly, management becomes a bigger and bigger fraction of the total cost. And so we move toward an approach that minimizes management cost, even if that approach is relatively wasteful of computing resources. The key is not that we're moving computation from client to server, but that we're moving management to the server, where a team of experts can manage matters for many users.

This certainly is true to an extent and it's one of the driving factors behind all sorts of outsourced hosting. Educated Guesswork, for instance, is hosted on Dreamhost, in large part because I didn't want the hassle of maintaining yet another public Internet-accessible server. I'm not sure I would call this "cloud computing", though, except retroactively.

That said, the term "cloud computing" covers a lot of ground (see the Wikipedia article), and I don't think Felten's argument holds up as well when we look at examples that look less like outsourced applications. Consider, for example Amazon's Elastic Compute Cluster (EC2). EC2 lets you rapidly spin up a large number of identical servers on Amazon's hardware and bring them up and down as required to service your load. Now, there is a substantial amount of management overhead reduction at the hardware level in that you don't need to contract for Internet, power, HVAC, etc., but since you're running a virtualized machine, you still have all the software management issues Ed mentions, and they're somewhat worse since you have to work within Amazon's infrastructure (see here for some complaining about this). Much of the benefit of an EC2-type solution is extreme resource flexibility: if you have a sudden load spike, you don't need to quickly roll out a bunch of new hardware, you just bring up some EC2 instances. When the spike goes away, you shut them down.

A related benefit is that this reduces resource consumption via a crude form of stochastic multiplexing: if EC2 is running a large number of Web sites, they're probably not all experiencing spikes at the same time, so the total amount of spare capacity required in the system is a lot smaller.

Both of these benefits apply as well to applications in the cloud (for instance, Ed's Gmail example). If you run your own mail server, it's idle almost all the time. On the other hand, if you use Gmail (or even a hosted service), then you are sharing that resource with a whole bunch of different people and so Amazon just needs enough capacity to service the projected aggregate usage of all those people, most of whom aren't using the system very hard (what, you thought that Amazon really had 8G of disk for each user?). At the end of the day, I suspect that the management cost Ed sites is the dominant issue here, though, which, I suppose argues that lumping outsourced applications ("software as a service") together with outsourced/virtualized hardware as "cloud computing" isn't really that helpful.

You may or may not have seen this article (Bill here courtesy of Lauren Weinstein; þ Joe Hall):

Key lawmakers are pushing to dramatically escalate U.S. defenses against cyberattacks, crafting proposals that would empower the government to set and enforce security standards for private industry for the first time.

OK, I'm going to stop you right there. I spend a large fraction of my time with computer security people and I don't think I've ever heard any of them use the term "cybersecurity", "cyberattacks", or pretty much "cyber-anything", except for when they're making fun of govspeak like this. Next they'll be talking about setting up speed traps on the Information Superhighway. Anyway, moving on...

The Rockefeller-Snowe measure would create the Office of the National Cybersecurity Adviser, whose leader would report directly to the president and would coordinate defense efforts across government agencies. It would require the National Institute of Standards and Technology to establish "measurable and auditable cybersecurity standards" that would apply to private companies as well as the government. It also would require licensing and certification of cybersecurity professionals.

So, it's sort of credible that NIST would generate some computer security standards. They've already done quite a few, especially in cryptography and communications security, with, I think it's fair to say, pretty mixed results. Some of their standards, especially the cryptographic ones like DES, AES, and SHA-1 have turned out OK, but as you start to move up the stack towards protocols and especially systems, the standards seem increasingly overconstrained and poorly matched to the kinds of practices that people actually engage in. In particular, there have been several attempts by USG to write standards about systems security (e.g., Common Criteria, and Rainbow Books) I think it's fair to say that uptake in the private sector has been minimal at best. Even more limited efforts like FIPS-140 (targeted at cryptographic systems) are widely seen as incredibly onerous and a hoop that developers have to jump through, rather than a best practice that they actually believe in.

I haven't gone through the bill completely, but check out this fun bit:

(4) SOFTWARE CONFIGURATION SPECIFICATION LANGUAGE.--The Institute shall, establish standard computer-readable language for completely specifying the configuration of software on computer systems widely used in the Federal government, by government contractors and grantees, and in private sector owned critical infrastructure information systems and networks.

I don't really know what this means but it sounds pretty hard. Even UNIX systems, which are extremely text-oriented, don't have what you'd call a standard computer readable configuration language. More like 10 such languages, I guess. I'm definitely looking forward to hearing about NIST's efforts to standardize sendmail.cf

The licensing and certification clause seems even sillier. There are plenty of professional security certifications you can get, but most people I know view them as more a form of rent seeking by the people who run the certifying classes than as a meaningful credential. I don't know of anyone that I know has one of these certifications. I'm just imaginine the day when we're told Bruce Schneier and Ed Felten aren't allowed to work on critical infrastructure systems because they're not certified.

More as I read through the actual document.

Leslie Daigle just summed up the situation with IPv6 at today's ISOC IPv6 press event: "It's [IPv6] sort of a broccoli technology; good for you but not necessarily attractive in its own right."

UPDATE: Corrected the quote a bit. Thanks to Greg Lebovitz for the correction.

I've got some code that needs to convert an IP address into a string. This is one of those cases where there's a twisty maze of APIs, all slightly different. The traditional API here is:

    char *
    inet_ntoa(struct in_addr in);

inet_ntoa() has two deficiencies, one important and one trivial: it doesn't support IPv6 and it returns a pointer to a statically allocated buffer, so it's not thread safe (I'll let you figure out which is which). Luckily, there's another API: addr2ascii():

    char *
    addr2ascii(int af, const void *addrp, int len, char *buf);

If you pass buf=0, addr2ascii() will return a pointer to a static buffer like inet_ntoa(). However, if you pass it an allocated buffer it will return the result in buf. Unfortunately, if you actually try to use addr2ascii() in threaded code you will quickly discover something unpleasant, at least on FreeBSD: you occasionally get the result "[inet_ntoa error]" or some fraction thereof. The answer is hidden in the EXAMPLES section of the man page:

In actuality, this cannot be done because addr2ascii() and ascii2addr() are implemented in terms of the inet(3) functions, rather than the other way around.

More specifically, on FreeBSD, it looks like this:

    case AF_INET:
        if (len != sizeof(struct in_addr)) {
	    errno = ENAMETOOLONG;
            return 0;
        }
        strcpy(buf, inet_ntoa(*(const struct in_addr *)addrp));
        break;

In other words, even though addr2ascii() doesn't explicitly use a static buffer, since it depends on inet_ntoa() it's still not thread safe. In order to get thread safety, you need to use yet another API:

    const char *
    inet_ntop(int af, const void *restrict src, char *restrict dst,
        socklen_t size);

Outstanding!

UPDATE: Clarified that this is a problem on FreeBSD. I don't know if it's an issue on all other platforms. Linux, for instance, doesn't have addr2ascii()
UPDATE2: Trivial vs. important.

This is an interesting development. The California SoS posts a list of donors to various political campaigns, but it's pretty un-user friendly. Someone has done a mashup with Google Maps so you can see everyone in your area who donated to Proposition Eight. Obviously, this is trivially extensible to any arbitrary political issue; it's just that Prop 8 has generated a huge amount of heat in a relatively tech savvy community. I wonder how long it is before a site like this is up for every issue or, perhaps more interestingly, before you can profile every donation for everyone who lives near you. For all I know you can do it now.