This site is moving to a new server. The old hosting service had too many problems, so we’re switching to something that should be better.
It’ll take a few days until everybody sees the new server. In theory you shouldn’t notice much difference — other than improved performance and reliability.
(If you see this post, you’re still seeing the old server. In the meantime you can try the new one.)
Sunday was the tenth anniversary of the government filing its antitrust case against Microsoft. The date passed almost unnoticed, though echoes of the case continue to reverberate. This week I want to reflect on the case, with the benefit of ten years’ hindsight. I’ll write at least three posts: today, on the overall legacy of the case; Wednesday, on how the case affected the public view of Microsoft and software companies generally; and Friday, on how the government’s theory of the software market (which the courts accepted) looks in hindsight.
(Before starting, I should clarify that although I worked with the DoJ trial team through virtually the entire case — from before the case was filed, through the negotiation of the final settlement — I can’t say anything about what happened behind closed doors. My opinion is informed by everything I saw and heard, but unfortunately some of the most interesting details have to stay secret.)
Today I want to consider the overall legacy of the case. The purpose of antitrust law is to protect market competition, for the good of consumers. Thus Microsoft’s ultimate success in crushing Netscape and blunting the effect of Java only matters to the extent that it might have harmed consumers. The relevant questions are these: (1) Are the markets for operating systems and browsers healthier and more competitive than they would have been had the case not been brought? (2) Are consumers better off than they would have been had the case not been brought?
I see the case as a success by these standards, not so much because of the settlement, which most people saw as weak, but because the case taught Microsoft that ignoring antitrust concerns can be dangerous. Microsoft was routed in court and faced the possibility (though never the likelihood) of a court-ordered break-up; but the company managed to negotiate a favorable settlement when the government was distracted after the 9/11 attacks. Apparently worried that it might not be so lucky the next time, the company has moderated its behavior. It still dominates the operating system and browser markets — and it is still a fierce technical competitor, but its business and legal behavior is more moderate.
This kinder, gentler Microsoft is one of the two main legacies of the case. The other is the consensus that antitrust laws do in fact apply to high-tech companies. Though the law moves slowly — and sometimes can only deter via the possibility of after-the-fact sanctions — companies are not immune to its discipline just because they are in high-tech markets. Other powerful companies, such as Intel and Google, have learned this lesson too.
Tomorrow: how the case affected the public view of Microsoft and the software industry.
We’re going to do a live webcast of our workshop on “The Future of News“, which will be held tomorrow and Thursday (May 14-15) in Princeton. Attending the workshop (free registration) gives you access to the speakers and other attendees over lunch and between sessions, but if that isn’t practical, the webcast is available.
Here are the links you need:
Sessions are scheduled for 10:45-noon and 1:30-5:00 on Wed., May 14; and 9:30-12:30 and 1:30-3:15 on Thur., May 15.
Last Friday, the New York Times published an article about counterfeit Cisco products that have been sold as if they were genuine and are widely used throughout the U.S. government. The article also raised the concern that these counterfeits could well be engineered with malicious intent, but that this appears not to have been the case. There was an immediate Slashdot thread as well, but a number of issues are still worth commenting on.
First things first: the facts, as best we understand them. The New York Times reports that approximately 3500 counterfeit Cisco components (worth $3.5M) have been discovered as a result of a two-year FBI investigation. A Cisco spokesman is quoted saying that they found “no evidence of re-engineering.” In other words, we’re talking about faithful knock-offs of legitimate products.
If you go to the FBI’s unclassified PowerPoint presentation (dated January 11, 2008), you’ll see all the actual information. This is a fascinating read. For starters, let’s talk about the cost. The slides claim you can get a counterfeit router for approximately 1/6 the cost of a genuine router. (You can do similarly well buying used gear on eBay.) The counterfeit gear looks an awful lot like the genuine article. Detecting differences here is as difficult as detecting counterfeit money, counterfeit Rolex watches, or counterfeit signatures from sports stars. Given the apparent discrepancy between component cost and street value, we should be no more surprised to find knock-off Cisco gear than we are to find knock-off everything else.
It’s claimed that these counterfeits are built to lower manufacturing standards than the original equipment, causing higher failure rates. One even caught fire due to a faulty power supply. Likewise, the fakers are making stupid errors, like building multiple components with the same MAC address. (MAC addresses, by design, are meant to be unique — no two ever the same.)
The really interesting story is all about the supply chain. Consider how you might buy yourself a new Mac. You could go to your local Apple store. Or you could get it from any of a variety of other stores, who in turn may have gotten it from Apple directly or may have gone through a distributor. Apparently, for Cisco gear, it’s much more complicated than that. The U.S. government buys from “approved” vendors, who might then buy from multiple tiers of sub-contractors. In one case, one person bought shady gear from eBay and resold it to the government, moving a total of $1M in gear before he was caught. In a more complicated case, Lockheed Martin won a bid for a U.S. Navy project. They contracted with an unauthorized Cisco reseller who in turn contracted with somebody else, who used a sub-contractor, who then directly shipped the counterfeit gear to the Navy. (The slides say that $250K worth of counterfeit gear was sold; duplicate serial numbers were discovered.)
Why is this happening? The Government wants to save money, so they look for contractors who can give them the best price, and their contracts allow for subcontracts, direct third-party shipping, and so forth. There is no serious vetting of this supply chain by either Cisco or the government. Apparently, Cisco doesn’t do direct sales except for high-end, specialized gear. You’d think Cisco would follow the lead of the airline industry, among others, and cut out the distributors to keep the profit for themselves.
Okay, on to the speculation. Both the New York Times and the FBI presentation concern themselves with Trojan Horses. Even though there’s no evidence that any of this counterfeit gear was actually malicious, the weak controls in the supply chain make it awfully easy for such compromised gear to be sold into sensitive parts of the government, raising all the obvious concerns.
Consider a recent paper by U. Illinois’s Sam King et al. where they built a “malicious processor”. The idea is pretty clever. You send along a “secret knock” (e.g., a network packet with a particular header) which triggers a sensor that enables “shadow code” to start running alongside the real operating system. The Illinois team built shadow code that compromised the Linux login program, adding a backdoor password. After the backdoor was tripped, it would disable the shadow code, thus going back to “normal” operation.
The military is awfully worried about this sort of threat, as well they should be. For that matter, so are voting machine critics. It’s awfully easy for “stealth” malicious behavior to exist in legitimate systems, regardless of how carefully you might analyze or test it. Ken Thompson’s classic paper, Reflections on Trusting Trust, shows how he designed a clever Trojan Horse for Unix. [Edit: it's unclear that it ever got released into the wild.]
Okay everybody, let’s put on our evil hats. If your goal was to get a Trojan Horse router into a sensitive military environment, how would you do it and how would it behave? Clearly, the weak supply chain is an excellent vector for getting the gear into place. Given the resources of a nation-state intelligence agency, you could afford to buy genuine Cisco parts and modify them, rather than using low-cost, counterfeit gear. Nobody would detect you; you wouldn’t screw up and ship multiple boxes with the same serial number.
How will you implement your Trojan Horse logic? Pretty much any gear you’ll ever find of any modest complexity will have software running inside it. Even line cards have embedded processors of some sort. For all that hardware, there’s software, and that’s what you’d go to install your logic bomb. The increasing use of FPGAs in industrial designs means you could also “rewire” those parts to behave arbitrarily, much like the Illinois hack; you’d really want to get a hold of the original VHDL “source code”, leveraging your aforementioned spying prowess, to simplify the design and implementation of your malicious behavior. Hacking the raw netlists (the FPGA-equivalent of machine code) would be possible, but would be far more painful. [See Sidebar.]
What sort of behavior would you build in? The New York Times raises the idea of a kill switch. I send your router a magic packet and it dies. That’s too easy. How about I send your router a magic packet, it then forwards it on to all of its peers, repeatedly, and then they all die a few seconds later? That’s a pretty good denial of service attack (nevermind a plot device that was the basis of a popular science fiction television series). Alternatively, following the Illinois idea, we could imagine that the magic packet turns on a monitoring feature, allowing our intelligence agency to gather all kinds of information, reconfigure the router, and so forth. If they don’t want to generate extra traffic, which might be detected, they could instead weaken the encryption of a VPN tunnel, perhaps publishing the session key through a subliminal channel of some sort, acquiring the ciphertext through “other” means.
In summary, it’s probably a good thing, from the perspective of the U.S. military, to discover that their supply chain is allowing counterfeit gear into production. This will help them clean up the supply chain, and will also provide an extra push to consider just how much they trust the sources of their equipment to ship clean software and hardware.
[Sidebar: Xilinx supports a notion of "encrypting" a netlist. Broadly speaking, the idea behind the technology is to encrypt the description of your FPGA configuration with a crypto key, such that anybody who reads the file out of your board gets encrypted garbage. However, the FPGA has the key material to decrypt the configuration and then initialize itself normally. This sort of technology is meant to serve an anti-piracy / anti-reverse-engineering purpose. It could ostensibly also serve an anti-Trojan Horse purpose, although at that point it's really no more or less secure, semantically, than Microsoft's Authenticode. This technology, more broadly, is also an active research area (see, for example, Roy et al.'s EPIC: Ending Piracy of Integrated Circuits). Again, if we've got a nation-state intelligence service tampering with the system, none of this is going to provide meaningful protection for the end-user against Trojan Horses.]
The RIAA’s head technology guy says that the move away from DRM (anti-copying) technology by record labels is just a phase, according to a Greg Sandoval story at News.com:
“(Recently) I made a list of the 22 ways to sell music, and 20 of them still require DRM,” said David Hughes, who heads up the RIAA’s technology unit, during a panel discussion at the Digital Hollywood conference. “Any form of subscription service or limited play-per-view or advertising offer still requires DRM. So DRM is not dead.”
…
Last January, when Sony BMG became the last major recording company to sell DRM-free tracks at Amazon, plenty of observers considered the technology buried. Since then, a growing number of online stores have begun offering at least some open MP3s, including Walmart.com, Zune’s Marketplace, Amazon, as well as iTunes.
Not so fast, said Hughes, who predicted that DRM would reemerge in a big way. “I think there is going to be a shift,” he told the audience. “I think there will be a movement towards subscription services, and (that) will eventually mean the return of DRM.”
The imminent success of subscription services with DRM is more or less what the record industry was predicting several years ago. It didn’t happen, mostly because customers found the services clunky and inflexible — DRM at its worst. Nothing has changed to make DRMed subscription services more attractive. If anything, these services look even worse in light of the trend toward selling DRM-free tracks.
I can see the argument for selling large bundles of music rather than selling one track at a time. Bundling makes economic sense, given the huge storage capacity of today’s devices. The iPod of the future won’t be filled one track at a time.
But clunky DRM-based subscription services aren’t the only way to sell bundles of songs, and there are probably good ways to sell subscriptions without DRM. If you’re worried that a customer will subscribe for one month, download a zillion songs, cancel the subscription and keep the songs,then you can limit the number of downloads per month, or require a longer subscription period. If you can sell songs without DRM — and we know now that you can — there ought to be a way to sell a friendly subscription service too.
On this issue, the RIAA’s members may be ahead of the RIAA itself. There are encouraging signs that some of the major record companies are recognizing the need to rebuild their business strategy for the Internet era.
James Fallows at the Atlantic recently ran a reader contest to nominate the worst public policy decision of the past fifty years. () I’d like to do the same for technology policy.
Readers, please submit your suggestions for the stupidest infotech policy ever. An ideal submission is an infotech policy that (1) was established by a government, (2) did serious damage, (3) had wide support across the political spectrum, (4) failed for reasons that should have been obvious at the time, (5) failed even by the standards of its own supporters. It’s not enough that you would have chosen differently, or that you would have weighed competing public goods differently — we’re looking for a policy that no reasonable person, with the benefit of hindsight, would support.
Submit your suggestions in the comments. Once the discussion has died down, I’ll choose a winner. If this contest is successful, we’ll follow it up with a best policy contest.
Today marks the 30th anniversary of (what is reputed to be) the first spam email. Here’s the body of the email:
DIGITAL WILL BE GIVING A PRODUCT PRESENTATION OF THE NEWEST MEMBERS OF THE DECSYSTEM-20 FAMILY; THE DECSYSTEM-2020, 2020T, 2060, AND 2060T. THE DECSYSTEM-20 FAMILY OF COMPUTERS HAS EVOLVED FROM THE TENEX OPERATING SYSTEM AND THE DECSYSTEM-10 (PDP-10) COMPUTER ARCHITECTURE. BOTH THE DECSYSTEM-2060T AND 2020T OFFER FULL ARPANET SUPPORT UNDER THE TOPS-20 OPERATING SYSTEM. THE DECSYSTEM-2060 IS AN UPWARD EXTENSION OF THE CURRENT DECSYSTEM 2040 AND 2050 FAMILY. THE DECSYSTEM-2020 IS A NEW LOW END MEMBER OF THE DECSYSTEM-20 FAMILY AND FULLY SOFTWARE COMPATIBLE WITH ALL OF THE OTHER DECSYSTEM-20 MODELS.
WE INVITE YOU TO COME SEE THE 2020 AND HEAR ABOUT THE DECSYSTEM-20 FAMILY AT THE TWO PRODUCT PRESENTATIONS WE WILL BE GIVING IN CALIFORNIA THIS MONTH. THE LOCATIONS WILL BE:
TUESDAY, MAY 9, 1978 - 2 PM
HYATT HOUSE (NEAR THE L.A. AIRPORT)
LOS ANGELES, CATHURSDAY, MAY 11, 1978 - 2 PM
DUNFEY’S ROYAL COACH
SAN MATEO, CA
(4 MILES SOUTH OF S.F. AIRPORT AT BAYSHORE, RT 101 AND RT 92)A 2020 WILL BE THERE FOR YOU TO VIEW. ALSO TERMINALS ON-LINE TO OTHER DECSYSTEM-20 SYSTEMS THROUGH THE ARPANET. IF YOU ARE UNABLE TO ATTEND, PLEASE FEEL FREE TO CONTACT THE NEAREST DEC OFFICE FOR MORE INFORMATION ABOUT THE EXCITING DECSYSTEM-20 FAMILY.
This is relatively mild by the standards of today’s spam. The message announced legitimate events relating to legitimate products in which the recipients might plausibly be interested. The sender was apparently unaware that this kind of message was against the rules.
Yet this message has much in common with today’s spam. The message used ALL CAPS, which was more common in those days but not the universal practice for email. The list of recipients was long. The message was incorrectly formatted — the original had more recipients than the email software of the day could handle, so what was supposed to be the recipient list actually spilled over into the body of the email, apparently unnoticed by the sender.
At the time, the Net’s rules forbade commercial activity, so the message was against the rules. Beyond the rule violation,the message’s propriety was widely questioned, and people debated what to do about it. (Brad Templeton has posted parts of the debate.)
Thirty years later, there is more spam than ever and no end is in sight. This shouldn’t be surprising, because the spam problem is fundamentally driven by economics. If anyone can send to anyone, and the cost of sending is nearly zero, many messages will be sent. Distinguishing unwanted email from wanted email is notoriously difficult — often you have to read a message to decide whether reading it was a waste of time. In this environment, spam will be a fact of life. The surprise, if anything, is that we have done as well as we have in coping with it.
I’m sure this sort of behavior is old news, but it’s still really annoying. Starting last night and continuing as I’m writing this, some annoying spammer has been forging my email address as the “From” line of a variety of spams. This is causing a staggering volume of backscatter, mostly of the “Delivery Status Notification (failure)” variety. Sampling these messages, I’m seeing several interesting things.
EDIT: fixed post that had gone out before it was done.
In France, most municipalities use paper ballots in elections, but a few places have begun using DRE (direct-recording electronic) machines. Pierre Muller, a French computer scientist, has recently sent me a report of a malfunction by an ES&S iVotronic machine in a recent municipal election.
In this spring’s elections (and he believes this also happened last year), there have been some unexplained “undervotes” on iVotronic machines. Below is a printout from an iVotronic machine. There’s a line “UnderVotes For Above Contest: 1″. Since the voter is required by the user-interface to choose between a candidate and the choice “vote blanc” [none of the above], undervotes should not be possible.
This event is similar in some ways to the Sequoia AVC Advantage bug observed in New Jersey on February 5, 2008. In both cases it appears that the machine is producing results that should not be possible, and in both cases local election officials are unable to explain how these results could legitimately be obtained.
Here is the relevant portion of the printout:
I’ve also prepared a larger image of the full printout, annotated with my English translation.
Last week, I posted here about voter ID requirements. There was a case pending before the U.S. Supreme Court on the same topic. It seems Indiana was trying to require voters to present ID in order to vote. Lawsuit. In the end, the court found that the requirement wasn’t particularly onerous (the New York Times’s article is as good as any for a basic summary, or go straight to the ruling).
Unsurprisingly, there has been a lot of hang-wringing on this (see, for example, this New York Times unsigned editorial). We can expect similar legislation elsewhere now that the Court has made it pretty difficult to challenge these sorts of laws (see, for example, the ongoing battle to pass this sort of legislation in Texas).
As I wrote last time, I’m not particularly opposed to voters being required to present ID. However, ID needs to be easy to get for anybody who is elgible to vote. For most people, this is easy. The big question we’d all like to know is the size of the population for which it’s not easy. Consider, as a hypothetical example, an elderly Texas woman who never drove a car. If she’s over 75 years old, the state’s centralized birth certificate registry won’t (officially) have her records. It could well require detective work to produce sufficient documentation to get her a state ID card. Who’s going to pay for that?
The big technical question, of course, is whether the root desires behind the voter ID requirement can be addressed in some more effective fashion than ID requirement. What are those root desires?
Which problems can be solved by purple ink on a voter’s thumb? #1 and #2 are readily solved, since a second attempt to vote will be forbidden. #3 is disincentivized, because the impersonator will be unable to vote under his or her own name. #4-6 will require other technologies.
Okay, which problems can be solved by having required voter ID? Let’s assume, for the sake of discussion, we have a centralized state database keyed off the voter’s ID card number, but individual polling places do not have real-time access to this database. Also, let’s assume that voter ID cards do not have any computational power: no smart cards, no crypto, etc. #1 is ostensibly solved by the central database. #2 cannot be prevented (at least, in a world with early voting or voting centers, where a voter has multiple places where he or she can legitimately vote), but it can be detected, and is thus disincentivized. #3 is solved. #4 is largely unsolved: if malicious poll workers want to forge signatures in the poll book, they may or may not be detected. (In a recount situation, written signatures should be verified, but it’s unclear what the accuracy of that checking process might be.)
You could try to solve #4 with smartcards that issue digital signatures, but that’s a whole different can of worms. Since the smartcard doesn’t really know what it’s being asked to sign, this could be exploited by an attacker. (Example: you need to present your ID in a variety of different circumstances, such as proving your age to enter a bar. The bouncer could “swipe” your card and use that as a way of getting a forged signature on an election record.)
What about #5 and #6? These are really back-end database problems. Requiring voters to present ID doesn’t have any impact. However, having a database that is keyed off the voters’ ID cards significantly improves #5 and #6 and could ostensibly help reduce a variety of errors in the process.
Curiously, it seems that most of the benefit of requiring ID occurs in the back-end database, rather than on the day of the election. The only real benefit of presenting ID, on election day, occurs in vote centers, early voting locations, and so forth. When there may be millions of eligible voters who could use a vote center, traditional paper poll books are unworkable. With a database keyed from ID card numbers, a voter’s records can be efficiently looked up and verified. While this isn’t a security problem, improving the efficiency of the voting process is still a worthwhile goal.
