So after 28 years of never using a Mac, I now have one. I got it for work, in my continuing endeavor to try new things on work computers. My desktop will still be a linux PC, but my laptop is now a Macbook Pro. I’ve only had it a few days, so I don’t have a lot of thoughts about it—though I still have an open mind.

Pros:
- It looks shiny
- With a massive amount of RAM, it runs quite smoothly
- Better integration with X and native X open-source applications than achieved through Windows (and e.g. CygWin)

Cons:
- Whole new keyboard with different buttons which do different things (will take time to get used to)
- It hides what it’s doing even more than Windows, leaving the user to have no clue what’s happening behind the scenes
- New multi-touch pad is different and doesn’t have a second mouse button
- Did not come pre-installed with any games
- I really miss the home and end keys. What does exist for home and end are, first two-key combos, and second, they are more like a top-and-bottom. I don’t yet know how to go to the beginning and ending of a dialog box I’m typing in.

So anyway, there are things I like and don’t like, but there’s a lot which I won’t know until I get used to it more.

Flash crowds are something that I think about a lot. This is mainly because it’s one of the prime challenges of building distributed systems.

Consider what happened in 1999 when Victoria’s Secret ran a Super Bowl ad announcing an online webcast of its Spring Fashion Show. The result was a sudden large volume of traffic to their site to view the webcast, so much that many customers were unable to view the webcast because the server could not handle the flash crowd.

A similar problem occurred after 9/11/2001, when everyone went to their favorite online news outlets for the emerging story.

What separates the two, of course, is that Victoria’s Secret planned their webcast (but failed to forsee the limits of their servers), where crisis situations are unpredicted, and generally not provisioned for.

This was clear in handling the San Diego firestorm last week in several ways, two of which I’ll mention here. What I find fascinating is how the people involved here had to adapt their technologies to handle the Crisis. In general, unexpected situations may always lead to this, and the people involved should largely be applauded. But at the same time, this presents us an opportunity to look at what happened to try and prepare automated systems for next time. Specifically, we need to improve or GIS/Mapping techniques, and our transparent web-content scalability techniques.

1. Basic information: Those of us who did not want our TVs constantly glued to a local news source (or radio) turned to the internet for updates. Furthermore, people worldwide with ties to San Diego, who don’t get local San Diego stations, tried to get that information online. This overloaded local servers and required adaptations.
   • San Diego County Emergency: This website was overloaded early on. There was a similar “official” website and phone system (211) for getting updates, but the phones apparenty had a 5 minute wait time, and website took over 15 minutes to load. At least the county emergency site would eventually load. This website hasn’t changed much since the beginning, despite a painfully small news clip box and downloads box which don’t work very well. The main changes here are (1) maps, which I’ll talk about later, and (2) apparently at some point Qualcomm is rumored to have been donating resources (bandwidth) to this site. That’s probably why it started loading better. I can only speculate that they wanted to keep information on this site so people would know it was authentic.
   • KPBS fire blog. KPBS realized their site would not stand the load, so they switched to using Twitter for updates. I didn’t try their site before they switched to Twitter, so I don’t know much about this one. I also don’t know much about Twitter, but it seems to be something designed for very short messages — like the size of an SMS. Using Twitter they were able to withstand the load, and people were able to use Twitter to subscribe, get content pushed to them, etc. KPBS was also the one to create a decent Google Map, which I’ll cover later.
   • The San Diego Union-Tribune fire blog. The San Diego U-T, a.k.a. SignOnSanDiego, started, like other sources, using headlines on their site. They quickly moved to a blog format to support the frequency of updates, but hosted at a url under their main site. Next, they realized that was too loaded, and split the blog out into a separate hostname (firestorm.signonsandiego.com), which allowed them to host it on other servers. This too could not keep up with the load, so finally they moved it to Blogger, the Google blog service. I was fascinated (and pleased) that both news sources had gone to public infrastructure broader than they alone could provide, and saw this as a success for big content installations. But still, this suggests we should look at server technologies which automatically replicate and scale content availability to match its demand, to avoid reconfigurations that might lose users.
2. Maps/Evacuation info. Perhaps the most important piece of information everyone wanted was where the fire was and where its headed. Much talk was made of the reverse-911 system (where 911 calls you to tell you their’s an emergency and you need to evacuate), which was very successful in alerting residents when they needed to evacuate. Of course, didn’t work for all — those of us without a landline would not get one (though now we know you can go here to add our non-land-lines (BEWARE, ONLY WORKS IN INTERNET EXPLORER). It also didn’t work for residents near Ramona, after their phone service was out due to the fire. In these cases, the old fashioned publis-safety-officer with loadspeaker and banging on door was the remaining solution. I myself signed up for updates as SMS messags on my phone (though I would have preferred a less expensive email option). But what we all wanted a place where we could just go and look at a map of evacuated and fire ares, which would be updated as time progressed. Here were our options:
   • No map: headlines only. If you know your regions of the town, you could mostly figure out what was going on from TV and radio news descriptions of where the fire was. But if you didn’t know the neighborhood names, you then had to go look them up, very inefficient.
   • TV maps. These maps were generally very bad. There was this one map they kept putting up with the evacuation areas, which was like 7 city names, and a fire symbol next to their name. Totally useless. Eventually, these switched to the county emergency maps (below), though they didn’t do much for giving credit here.
   • Site hosted maps. The U-T briefly tried to host a map on their site, but the load was too great, and it was updated too infrequently to be of any use.
   • Google maps. These were what I had great hopes for, but apparently the technology wasn’t quite there yet. The LA Times site had a map with the burn areas both of San Diego and LA, but it wasn’t very precise, and the fire icons on the map just didn’t give very good details. The best Google map was the one KPBS maintained, which they updated when they got the chance. It was fairly decent, and showed evacuation and burned areas, though there were a number of glitches. For example, a couple of times markers were placed in Las Vegas or LA, when they were supposed to be within San Diego County. Also, the Google Map never got to the point where it showed the active burn areas. It did, however, get linked from the main google maps page.

     Most interestingly though, twice on the KPBS twitter, they indicated they couldn’t update the map because of limitations, and they were working with Google to get improvements made. I’m not sure what (if anything) changed, but I was pleased to see they didn’t feel limited by the features, and that Google was willing to get involved. The google maps to me really showed the greated promise, because they were able to show satelite overlaid with fire information, street-level details, traffic information and also news blurbs on map icons. Problems, however, were that it was apparently too hard to update, the news blurbs were too big and thus were displayed poorly, and that you couldn’t turn on and off layers of the map (e.g. maybe you didn’t want to see the cluttered evacuation center icons or home destruction icons right now). As an attempt though, this gets the best credit, and should be the starting piont for future efforts. The SD County Emergency site needed to be able to easily update one or more map layers which people could download (from a scalable hosting site), and transparently add to the mapping program of their choice (be it google maps, MS live, or an offline map program. Over the county’s own maps, it provided very important details such as streets. GIS (Geographic information services) should be a major research front in this for future crises.

   • PDF downloads from the SD County Emergency site. In my opinion, these were the best of the maps, because they were most accurate, timely, and detailed. Unlike the other maps, these were the only ones to show active fire areas (though that took a few days to be added). At one point I realized that these were essentially the root of all maps — they were the input that KPBS was using for their google map, and all other news sources were using them and redistributing them. In fact, NBC San Diego actually was using these maps on the air, actually interacting with them in adobe acrobat reader. Meanwhile, they were telling viewers they too could download them from the NBC San Diego site (on the one hand, giving viewers the impression they made them, ont he other hand, taking important load away from the county emergency site, which was strained). Within this series of maps (and I’ve downloaded them all so I can see the progression of both the fire and the mapping techniques), it’s fascinating to see the progress. Originally, it was just the burn and evacuation areas. It broke up mandatory and voluntary evacuation areas with colors, and showed the burned area in solid red. But as the burn area grew, and some areas (some of which were burned because reopened), there were problems. Lack of transparency made it hard to see, and there were getting to be too many colors. (Is it burned, or is it ok to return?). So they eventually progressed to making the burned area translucent, and showing the active fire areas so (1) you could understand why they were letting people back into certain areas, and (2) you could see that an area had been both burned and was reopened. But the overlapping made it appear as a separate color, and was confusing unless you zoomed in. So eventually they just showed the fire perimiter with the active fire areas, so they could clearly show which areas were evacuated, and which were re-opened.

     Some confusing things remained to me. For example, un-inhabited areas were never shown as evacuated, but it was never obvious to me whether they were not evacuated because they weren’t inhabited, or because the fire wasn’t expected to go there. This because I didn’t know the geography of the county that well. In the most recent map I noticed they’ve marked some national forests, which make it easier to see this. Also, the lack of street-level details made it hard to tell where on the map you were. It did at least break up the map into the “Thomas-brothers” map grid, though since I don’t have one it didn’t do me any good. Next, I thought it was strange that the only download was a ~1MB PDF file, which no-doubt was getting a lot of hits. Seems like they could have saved a lot of bandwidth by offering the map in a 300KB JPEG format as well, which would have sufficed for many of us. Furthermore, the PDF had many layers, was large, and slow to render, having to be redrawn every time you zoomed or moved in the map.

     Example maps (for all maps, go to http://www.sdcountyemergency.com/).

     • First Emergency Map, 10/22, 9:00 am: This map was the first county-wide map with both Witch and Harris fires. It shows burn zone, evacuation zone, and voluntary evacuation zone. It also shows the 2003 fire perimeters, which was very interesting, but not so useful for evacuation. It is however, the best indicator of how the fire burned a different area this time. Problems include the different layerings, lack of Thomas Guide grid, too many dots for municipal stuff.
     • 10/23, 12:15 pm Map: Now they added the Thomas Guide. But there are spots where all three of fire, mandatory, and voluntary evacuations are layered, leaving one confused.
     • 10/23, 10:30 pm: So I’m noticing the 10:30 trend here, which I’m wondering if its the news cycle release. This map has areas people can return to, which is good, but they are adjacent in come cases to fire areas, leaving me confused as to why they can return. Soon the return areas will include burned areas, making it worse.
     • 10/24, 05:00 am: Now they’ve added the active burn area colors, which makes it much more clear why certain areas are allowed to return. But the solid fire color won’t work once they let people back in.
     • 10/25, 2:30 am: Here we see the overlapping burn and repopulation areas, which is very confusing, as it shows many colors in overlap. But otherwise, the map is looking pretty good.
     • 10/26, 7:00 pm: In this update, we now see the fire perimiter separately. The evacuation and repopulation areas are clear, as well as the active fire.
   • Several days into the event, they (San Diego County Emergency) released a Google earth data file. I have no idea what this contained or looked like, as I don’t have Google earth on my computer.

So for some time I have been using Gallery as my picture site, and I’ve been quite happy with it overall (my prior post about it notwithstanding).

In recent versions, I have noted a reference to a “Google SiteMap” in the administration pages. Being ignorant of them, I ignored it. Yesterday, I decided to look a bit further into it to understand them. This was partly because lately I’ve felt like a large amount of my server bandwidth has been taken by search robots, and I wondered/hoped that the sitemap would make the crawler use less bandwidth.

When Googling for sitemap, most top links take you to the Google Webmaster Tools. Unfortunately, if you have a google account, this won’t really show you any information about what it is until after you’ve gone through a bit of extra registration, and even so, I don’t feel like there’s much information there.

The best link I’ve found about sitemaps is either the “Official” sitemaps page or the Wikipedia Sitemaps page. The first thing I wanted to know was if there was a way, like “robots.txt”, that web crawlers discover them. (In that case, the crawler just checks to see if the file is there, and IMO, it makes sense for the sitemap to be found in the same way. Yet I found no reference to a “standard” naming convention for sitemaps. Instead, most webpages I found went on and on about using RPC ‘pings’ to notify crawlers of sitemaps, or uploading your sitemap using the google webmaster tools. This is useful, I agree, but I was looking for a method which did not involve actively trying to get crawlers interested in my site, but rather would just be there when they came looking. What I eventually found (and is mentioned in both of the two links above about sitemaps), is that you can list the URL to the “Sitemap” in the robots.txt file, which can clue web crawlers into its presence.

(Incidentally, at this point, it is worth mentioning that there are two things called sitemaps. One is the “Google sitemap”, supported by practically all major search crawlers. The other is a generic term for a page which links to all other pages on your site, which has many of the same benefits, but is free form, rather than with semantic meaning. I don’t actually know if the robots.txt should specifically point to a “Google Sitemap” or if it can be any page which links to all other pages on your site.)

There is little information about what crawlers do with sitemaps, but I can at least share with you what information is in one. In addition to the URLs of each page you want included on your site, the sitemap tells the crawler (1) when the page last was modified, (2) how often the page is modified, and (3) relatively speaking, how important the page is to you. The sitemap seems to serve a few useful purposes. First, it may tell the crawler pages have not been modified, and need not be crawled. Next, it may point out pages to the crawler which are not easily linked on your page, and which the crawler might otherwise overlook. Also, it tells the crawler how often different pages change, which it may use as hints as to when to re-crawl them.

So back to my original question: “Does a sitemap reduce the crawler bandwidth?”. I think the answer is — no one knows, outside of the secret teams who implement the crawler. But in the case of Gallery (which it the largest content source on my site), the sitemap tells the crawler that the images themselves never change, but that new links may appear on gallery pages. This could reduce bandwidth as the crawler need not re-download all my pictures very often. But even better, the crawler could in fact reduce its crawling to non-listed pages if it trusts my last-modified timestamps, because if any of them are updated, the sitemap will contain the new timestamp.

Finding this somewhat satisfactory, I also went looking for a sitemap generator for WordPress, and found one by Arne Brachhold on the extension site. This extension even notifies search engines when the sitemap changes. This extension also had the added benefit of allowing me to specify other pages (which are not written in wordpress) to add to the Sitemap, since the blog is only part of my site.

Who knows if it will make any difference to my site. But I figure it cannot hurt.

So from a research perspective, I am curious— are we winning or losing the battle against spam? On the one hand, there are reports that upwards of 80% of all email is spam. But only a small fraction of that ever sees our inboxes. Sometimes I feel that the fraction is shrinking, but other times (like now), I feel that everything is just a stop-gap, and that this is really an arms race with a huge amount of wasted resources, both in bandwidth and in person-hours.

I was discussing this with a group of colleagues, and one made the claim that this is a “solved problem” for corporate America. That is to say that when you are working at a big corporation, you don’t get spam email in your inboxes. Is it true? If so, do you know why?

What about personal mail? Does the answer change depending on whether it’s an ISP mail or a webmail? What about for preventing spam from being received by young children? I’m just not convinced we’re anywhere close to a good solution on spam. Of course, there are others who argue that spam is fundamental. I hope not.

For some time I’ve used Gallery as my gallery software of choice. (And I still do–you can find the link to my gallery along the page header). But more recently, I’ve started to think that what I’d prefer to have is software which makes it easy to create scrapbooks online. I think the key difference here is the focus on narrative instead of the pictures themselves. Pictures are used to tell the story, not to be the primary point of the content. I’m happy to have a gallery at the same time which is highly integrated with the scrapbook, but I’d like the narrative (with collections of photos related to narrative parts) to allow presentation in a scrapbook format.

If anyone is aware of such software (particularly, but not strictly, free software), please let me know.