Evolving Technology in Crisis

Flash crowds are something that I think about a lot. This is mainly because it’s one of the prime challenges of building distributed systems.

Consider what happened in 1999 when Victoria’s Secret ran a Super Bowl ad announcing an online webcast of its Spring Fashion Show. The result was a sudden large volume of traffic to their site to view the webcast, so much that many customers were unable to view the webcast because the server could not handle the flash crowd.

A similar problem occurred after 9/11/2001, when everyone went to their favorite online news outlets for the emerging story.

What separates the two, of course, is that Victoria’s Secret planned their webcast (but failed to forsee the limits of their servers), where crisis situations are unpredicted, and generally not provisioned for.

This was clear in handling the San Diego firestorm last week in several ways, two of which I’ll mention here. What I find fascinating is how the people involved here had to adapt their technologies to handle the Crisis. In general, unexpected situations may always lead to this, and the people involved should largely be applauded. But at the same time, this presents us an opportunity to look at what happened to try and prepare automated systems for next time. Specifically, we need to improve or GIS/Mapping techniques, and our transparent web-content scalability techniques.

1. Basic information: Those of us who did not want our TVs constantly glued to a local news source (or radio) turned to the internet for updates. Furthermore, people worldwide with ties to San Diego, who don’t get local San Diego stations, tried to get that information online. This overloaded local servers and required adaptations.
   • San Diego County Emergency: This website was overloaded early on. There was a similar “official” website and phone system (211) for getting updates, but the phones apparenty had a 5 minute wait time, and website took over 15 minutes to load. At least the county emergency site would eventually load. This website hasn’t changed much since the beginning, despite a painfully small news clip box and downloads box which don’t work very well. The main changes here are (1) maps, which I’ll talk about later, and (2) apparently at some point Qualcomm is rumored to have been donating resources (bandwidth) to this site. That’s probably why it started loading better. I can only speculate that they wanted to keep information on this site so people would know it was authentic.
   • KPBS fire blog. KPBS realized their site would not stand the load, so they switched to using Twitter for updates. I didn’t try their site before they switched to Twitter, so I don’t know much about this one. I also don’t know much about Twitter, but it seems to be something designed for very short messages — like the size of an SMS. Using Twitter they were able to withstand the load, and people were able to use Twitter to subscribe, get content pushed to them, etc. KPBS was also the one to create a decent Google Map, which I’ll cover later.
   • The San Diego Union-Tribune fire blog. The San Diego U-T, a.k.a. SignOnSanDiego, started, like other sources, using headlines on their site. They quickly moved to a blog format to support the frequency of updates, but hosted at a url under their main site. Next, they realized that was too loaded, and split the blog out into a separate hostname (firestorm.signonsandiego.com), which allowed them to host it on other servers. This too could not keep up with the load, so finally they moved it to Blogger, the Google blog service. I was fascinated (and pleased) that both news sources had gone to public infrastructure broader than they alone could provide, and saw this as a success for big content installations. But still, this suggests we should look at server technologies which automatically replicate and scale content availability to match its demand, to avoid reconfigurations that might lose users.
2. Maps/Evacuation info. Perhaps the most important piece of information everyone wanted was where the fire was and where its headed. Much talk was made of the reverse-911 system (where 911 calls you to tell you their’s an emergency and you need to evacuate), which was very successful in alerting residents when they needed to evacuate. Of course, didn’t work for all — those of us without a landline would not get one (though now we know you can go here to add our non-land-lines (BEWARE, ONLY WORKS IN INTERNET EXPLORER). It also didn’t work for residents near Ramona, after their phone service was out due to the fire. In these cases, the old fashioned publis-safety-officer with loadspeaker and banging on door was the remaining solution. I myself signed up for updates as SMS messags on my phone (though I would have preferred a less expensive email option). But what we all wanted a place where we could just go and look at a map of evacuated and fire ares, which would be updated as time progressed. Here were our options:
   • No map: headlines only. If you know your regions of the town, you could mostly figure out what was going on from TV and radio news descriptions of where the fire was. But if you didn’t know the neighborhood names, you then had to go look them up, very inefficient.
   • TV maps. These maps were generally very bad. There was this one map they kept putting up with the evacuation areas, which was like 7 city names, and a fire symbol next to their name. Totally useless. Eventually, these switched to the county emergency maps (below), though they didn’t do much for giving credit here.
   • Site hosted maps. The U-T briefly tried to host a map on their site, but the load was too great, and it was updated too infrequently to be of any use.
   • Google maps. These were what I had great hopes for, but apparently the technology wasn’t quite there yet. The LA Times site had a map with the burn areas both of San Diego and LA, but it wasn’t very precise, and the fire icons on the map just didn’t give very good details. The best Google map was the one KPBS maintained, which they updated when they got the chance. It was fairly decent, and showed evacuation and burned areas, though there were a number of glitches. For example, a couple of times markers were placed in Las Vegas or LA, when they were supposed to be within San Diego County. Also, the Google Map never got to the point where it showed the active burn areas. It did, however, get linked from the main google maps page.

     Most interestingly though, twice on the KPBS twitter, they indicated they couldn’t update the map because of limitations, and they were working with Google to get improvements made. I’m not sure what (if anything) changed, but I was pleased to see they didn’t feel limited by the features, and that Google was willing to get involved. The google maps to me really showed the greated promise, because they were able to show satelite overlaid with fire information, street-level details, traffic information and also news blurbs on map icons. Problems, however, were that it was apparently too hard to update, the news blurbs were too big and thus were displayed poorly, and that you couldn’t turn on and off layers of the map (e.g. maybe you didn’t want to see the cluttered evacuation center icons or home destruction icons right now). As an attempt though, this gets the best credit, and should be the starting piont for future efforts. The SD County Emergency site needed to be able to easily update one or more map layers which people could download (from a scalable hosting site), and transparently add to the mapping program of their choice (be it google maps, MS live, or an offline map program. Over the county’s own maps, it provided very important details such as streets. GIS (Geographic information services) should be a major research front in this for future crises.

   • PDF downloads from the SD County Emergency site. In my opinion, these were the best of the maps, because they were most accurate, timely, and detailed. Unlike the other maps, these were the only ones to show active fire areas (though that took a few days to be added). At one point I realized that these were essentially the root of all maps — they were the input that KPBS was using for their google map, and all other news sources were using them and redistributing them. In fact, NBC San Diego actually was using these maps on the air, actually interacting with them in adobe acrobat reader. Meanwhile, they were telling viewers they too could download them from the NBC San Diego site (on the one hand, giving viewers the impression they made them, ont he other hand, taking important load away from the county emergency site, which was strained). Within this series of maps (and I’ve downloaded them all so I can see the progression of both the fire and the mapping techniques), it’s fascinating to see the progress. Originally, it was just the burn and evacuation areas. It broke up mandatory and voluntary evacuation areas with colors, and showed the burned area in solid red. But as the burn area grew, and some areas (some of which were burned because reopened), there were problems. Lack of transparency made it hard to see, and there were getting to be too many colors. (Is it burned, or is it ok to return?). So they eventually progressed to making the burned area translucent, and showing the active fire areas so (1) you could understand why they were letting people back into certain areas, and (2) you could see that an area had been both burned and was reopened. But the overlapping made it appear as a separate color, and was confusing unless you zoomed in. So eventually they just showed the fire perimiter with the active fire areas, so they could clearly show which areas were evacuated, and which were re-opened.

     Some confusing things remained to me. For example, un-inhabited areas were never shown as evacuated, but it was never obvious to me whether they were not evacuated because they weren’t inhabited, or because the fire wasn’t expected to go there. This because I didn’t know the geography of the county that well. In the most recent map I noticed they’ve marked some national forests, which make it easier to see this. Also, the lack of street-level details made it hard to tell where on the map you were. It did at least break up the map into the “Thomas-brothers” map grid, though since I don’t have one it didn’t do me any good. Next, I thought it was strange that the only download was a ~1MB PDF file, which no-doubt was getting a lot of hits. Seems like they could have saved a lot of bandwidth by offering the map in a 300KB JPEG format as well, which would have sufficed for many of us. Furthermore, the PDF had many layers, was large, and slow to render, having to be redrawn every time you zoomed or moved in the map.

     Example maps (for all maps, go to http://www.sdcountyemergency.com/).

     • First Emergency Map, 10/22, 9:00 am: This map was the first county-wide map with both Witch and Harris fires. It shows burn zone, evacuation zone, and voluntary evacuation zone. It also shows the 2003 fire perimeters, which was very interesting, but not so useful for evacuation. It is however, the best indicator of how the fire burned a different area this time. Problems include the different layerings, lack of Thomas Guide grid, too many dots for municipal stuff.
     • 10/23, 12:15 pm Map: Now they added the Thomas Guide. But there are spots where all three of fire, mandatory, and voluntary evacuations are layered, leaving one confused.
     • 10/23, 10:30 pm: So I’m noticing the 10:30 trend here, which I’m wondering if its the news cycle release. This map has areas people can return to, which is good, but they are adjacent in come cases to fire areas, leaving me confused as to why they can return. Soon the return areas will include burned areas, making it worse.
     • 10/24, 05:00 am: Now they’ve added the active burn area colors, which makes it much more clear why certain areas are allowed to return. But the solid fire color won’t work once they let people back in.
     • 10/25, 2:30 am: Here we see the overlapping burn and repopulation areas, which is very confusing, as it shows many colors in overlap. But otherwise, the map is looking pretty good.
     • 10/26, 7:00 pm: In this update, we now see the fire perimiter separately. The evacuation and repopulation areas are clear, as well as the active fire.
   • Several days into the event, they (San Diego County Emergency) released a Google earth data file. I have no idea what this contained or looked like, as I don’t have Google earth on my computer.