Just as a quick update, this afternoon I had another problem which isn’t “supposed to happen” in MacOS. When I arrived home from school, my laptop was burning hot in my bag. When I opened the screen, there was an error message telling me I needed to hard-power cycle my macbook by holding in the power button, because of an error. It was a kernel panic, but the laptop was left in some continuous processing loop, with the fan spinning, but doing no good in my bag, while an unhelpful message was telling me to power cycle it. So much for Mac’s “Just working”. At least with other OSes, I expect such errors, and am very careful about making sure they aren’t happening.

So I’ve been using the MBP for a while now, and it’s still taking some getting used to. It’s also still just “okay.” I wouldn’t say I’m overwhelmed with how much better it is than Windows.

Moreover, I installed updates today, and upon reboot, found that a new icon was present in the menu bar. And not just that, it gave me a nag message, asking me to sign up for MobileMe.

I have to agree with the vast number of users writing complaints that this new “feature” was unrequested, unauthorized, and undesired. (I won’t link to them here, but just a quick internet search will find them). Tactics like this will probably renew my desire to be using an opensource OS on my laptop, and there is a reasonable chance I won’t stick with Mac when it comes time to get another new laptop.

Granted, I’ve ruled out Windows too, so I’m not sure what will be next. Back to linux, and the headaches of not having powerpoint plus dealing with projectors? Perhaps opensolaris?

As an aside - also after installing the updates, my trackpad stopped working until I powered the machine off and back on — another fix which is supposedly only supposed to happen on Windows.

So after 28 years of never using a Mac, I now have one. I got it for work, in my continuing endeavor to try new things on work computers. My desktop will still be a linux PC, but my laptop is now a Macbook Pro. I’ve only had it a few days, so I don’t have a lot of thoughts about it—though I still have an open mind.

Pros:
- It looks shiny
- With a massive amount of RAM, it runs quite smoothly
- Better integration with X and native X open-source applications than achieved through Windows (and e.g. CygWin)

Cons:
- Whole new keyboard with different buttons which do different things (will take time to get used to)
- It hides what it’s doing even more than Windows, leaving the user to have no clue what’s happening behind the scenes
- New multi-touch pad is different and doesn’t have a second mouse button
- Did not come pre-installed with any games
- I really miss the home and end keys. What does exist for home and end are, first two-key combos, and second, they are more like a top-and-bottom. I don’t yet know how to go to the beginning and ending of a dialog box I’m typing in.

So anyway, there are things I like and don’t like, but there’s a lot which I won’t know until I get used to it more.

I have defended my dissertation, and filed it with the University. It’s formatting has been approved by the appropriate people, and the Dean of Graduate Studies has approved my final paperwork. I believe that officially, I have now completed my Ph.D. Graduation is June 22nd, and my parents, in-laws, and brother-in-law (and of course Kristina) will all be in attendance as my advisor hoods me.

Now it is time for a few brief moments of relaxation and a vacation, then it’s time to move on and get started on my career.

Well, to those of you following the job saga, it is now over (pending approval by the Provost). After 2.5 months of interviewing all over the country, we have decided to accept a position in the Computer Science department at Purdue University. None of the details are set yet, so don’t start asking questions about when we’re moving or anything. But sometime prior to August 18th (my start date at Purdue), we’ll be moving to the West Lafayette, Indiana area. So when you find yourselves visiting Purdue, Indiana, Indianapolis, or Chicago, let us know!

As an early user of gmail, I was able to select precisely the username I wanted, ckillian, which is a very common username for people whose first name starts with ‘C’ and whose last name is Killian. Unfortunately, as often goes for popular shared services, there are many gmail users who fit those parameters. By itself, that wouldn’t be a problem, except that on occasion, these other gmail users seem to forget that ckillian is not their email address. They will use it to buy tickets on ticketmaster, place beach house reservations, setup ipod accounts, request proprietary recipes from companies, purchase items from websites, and, most recently, even to use it to purchase online postage from the USPS.

The “best” part is when users are so convinced it is their address that they go through Google’s password recovery system to try to get my password. This has happened 3 times so far. (I know, because Google sends me a link to my email addresses to follow if I want to proceed in resetting my password.) One truly intelligent user, after going through the password change system and failing, actually sent me an email asking if I would forward the information to her, which I was happy to do (on a temporary basis).

I recognize that for the users, this is generally an honest mistake (I get these receipts for a CXXXX Killian, and what’s obvious to me is that ckillian is there username for some other things, and they just got mixed up while entering the email address). When I get such receipts, responses, etc., if there is a phone number listed for the user, I often make the attempt to phone them to let them know of their mistake. But more often than not, there is no method shown to contact them with. In these cases, I have two options: (1) ignore it, and hope I don’t get stuck on some mailing list, or (2) contact the seller/sender and let them know they are sending information to the wrong party. Some vendors handle this well — the Apple store took care of it without hassle. Others, like the USPS, take some convincing (at first they thought I was trying to commit fraud). And then there are those like Ticketmaster, who I have simply given up on, because I can’t seem to get them to stop sending me junk even when I did setup the account (and diligently unchecked the boxes so I would not receive the junk).

This represents a fairly significant issue though, because for many of these services and sites, by going through a forgotten password dialog, I could have the password reset and emailed to my account, giving me access to their information and account, and possibly other information such as credit cards, or perhaps just the ability to purchase things using their credit cards.

And what frustrates me the most is that most of these sites have set up a kind of account based on this email address, without validating that the new user actually has access to the email address. It’s one thing if you simply mistype an email and as a result a single receipt goes to the wrong email address. It’s yet another if you are saving state for the user under this email address without validating it. Most websites form of validation is just to have the user type it twice. But we should know by now that user data cannot be trusted, and if we are going to store that kind of information, we really should validate the email address.

And it’s not even that hard to do so—mailing lists do this all the time. When you subscribe, they send you an email for you to prove you have received before allowing your subscription to proceed. All sites creating accounts should do the same. I would much rather have gotten an email from the USPS telling me to validate someone else’s account (which I would not have) than gotten a receipt for a delivery confirmation postage for a particular person.

So to those of you developing websites which create accounts for email addresses — please, please, please validate these email addresses before storing them!

I’m starting a new category for programming tips. I jokingly referred to something as the C++ feature of the week (for Mace development) with one of our developers, and he responded that he needed to subscribe. And it seemed like a good idea, so now I think I’ll start trying to blog about these new features I learn about.

So to start it off, there are two C++ features of the week for this past week:

1. STL collection swap. The C++ STL Collections contain a swap() method, which takes another collection (of the same type) as a parameter. The method does what’s expected — to swap the one collection’s elements with the other. What makes this an interesting function is that it does it in constant time. It doesn’t require constructing, copying, or otherwise wasting time with the two collections. It just does pointer copies of internal collection state. To see how this is useful, consider these two cases I’ve applied this to:
   • Maps within maps. In one case, a Mace programmer had a map from an int to a vector. In this case, the int represented the number of things in the vector (admittedly, this is a bit of a simplification). So, when removing something from the vector, you would remove the vector from the map, then re-add it with the new key. (This is because for other good reasons, the key of a map entry cannot be changed). Because of the cost of this removal and re-addition, the programmer had originally implemented this using pointers, which, while correct and efficient, gave some of our other tools problems, and so we wanted to re-write it without using pointers. swap() made this possible. To do the update, use this code:
     
void removeElement(IntVectorMap& ivmap, int size) {
IntVectorMap::iterator i = ivmap.find(size);
assert(i != ivmap.end());
i->second.pop_front();
ivmap[size-1].swap(i->second);
ivmap.erase(i);
}

     This does involve a construction of a new collection, but moves the elements of the collection quite efficiently. Since we will erase the original map entry, causing the old vector to cease to exist, the fact that it now holds no elements does not matter.
   • The second case was one where we wanted to iterate through a set, but other code might be adding things to the set at the same time. To maintain code safety, we must not lose newly added things, so we can process them later. The original design involved making a copy of the set, then clearing the original set, and iterating over the copy. Once again, swap() is the right tool here too.
     
void processSet(IntSet& s) {
IntSet t;
t.swap(s);
for(IntSet::iterator i = t.begin(); ...) {
...
}
if (!s.empty()) { processSet(s); }
}

     As an added bonus, if you need to hold a lock to touch S, you can simply acquire the lock and do the switch, a very fast operation, the release the lock.
2. boost::concept_check. We were updating our serialization code, but found that the compiler error messages on template errors are hard-to-decipher (to be generous). These errors were caused by one of two problems in one of our cases. First, we had added a new template parameter, and inserted it before some existing ones. In code which hadn’t been updated though, if they provided the older optional template parameters, the compiler would get very confused, and report error messages which could not be deciphered, and pointed to lines of code which didn’t make any sense. In the other error case, the default template parameter might not work with other types passed. (Specifically, it was a parameter telling how to serialize a collection, and the collection elements might not have been serializable.) This message was a little easier to decipher, complaining about types which could not be serialized, but still didn’t point to the right lines of code.

   Using boost’s concept check, we were able to help both of these problems. In the first case, we wrote a base class for all valid parameters of the template, then used a concept check to make sure the template parameter was convertible to the base class. Passing in the older parameter now would generate a shorter, easier to understand message, and the concept check library makes sure that the line of code makes sense. In the second case, we had to write our own concept checker, which would essentially just write code that needed to be able to compile (in this case, instantiating the type, serializing it, and deserializing it). Again, the concept_check library would make sure the error message was pointed to in the right place.

That’s all for this edition. Watch the programming category if you want to see other programming tips.

I should start by explaining I regularly run my web browser with cookies disabled. The reason is that I decided websites are tracking you too closely, and especially websites which you didn’t even know you were visiting. For example, open up your cookie list. (In firefox, this is: Tools->Options (under Windows, Edit->Preferences under Linux), then Privacy->”Show Cookies”. The questions to ask yourself are:

1. How many of the sites listed do I even recognize?
2. Of the sites I do recognize, what do I want that site to remember about me the next time I visit?

Cookies, you see, are files that a server gives to a web browser, and asks it to present them whenever they visit a set of pages on a set of sites. Cookies have a number of legitimate uses, most notably to give the browser a “session” id. The “session” id is used so the browser user can, e.g., log in, and have the server remember keep track of information related to the login. (The other option, not using cookies, is to make the sessionid part of the URLs, which is both ugly, and more likely to be logged by third parties such as proxies and caches run by ISPs)

Then there are some arguably useful features of cookies. For example, many online retailers will set a cookie identifying you at your browser, and recognize you immediately when you visit again (not for purchasing, but for welcoming, tracking the products you look at, so to remind you of past products you’ve visited and to suggest new products based on your viewing history. I personally find that a little creepy, though I admit in some cases it can be valuable. A few years ago, there were even reports of sites using cookies to do Dynamic Pricing (story by CNN), a practice where sites change the prices based on information they keep about the customer. There were reports of users visiting Amazon from a new computer, finding an item they like, then logging in, and seeing it for a new price. In my opinion, these types of things outweigh the possible positive benefits from having a site remember me just for cause.

Next there are in my book some outright despicable practices. Advertisements placed on sites will add cookies which get reported back to these tracking sites anytime you visit any site with an advertisement from the same company. As a result, there are sites which simply compile vast amounts of information about where you go and what you do online, to use in any way they seem fit. These are commonly called “Tracking Cookies” by products such as Ad-aware and Spybot, which will remove the ones they recognize for you.

I have simply taken the approach (mostly as an experiment) that sites shall not store cookies without my express consent. To that end, I have installed CookieSafe, which makes it easier to manage cookie settings. I either give or reject cookies from specific sites. This occurs as a site preference, meaning if a site uses both kinds of cookies, and I want to use the site, I accept them both. Importantly, the third-party cookies are still rejected — I have to authorize them separately.

So my browsing works like this: I browse normally, then if a site isn’t working (and particularly if submitting a login doesn’t work), I realize it needed cookies to work. I then decide if I really want to use the site, and if I do, I enable cookies for that site only.

Now, when I view my list of cookies, I can identify most of the sites. (Some I must have authorized, but don’t quite recognize by site name, like the third party my bank uses to process online billpay.). I find this to be much more acceptable, and my browsing hasn’t been worse for the wear.

A few days ago, however, I saw something that really brought a smile to my face. On a site I visited while trying to figure out what it meant to buy fertile eggs, I saw this image, where an ad belongs:

I just had to laugh. If a site wants to not send me ads because I reject cookies — then great! I didn’t want them anyway. But somehow I think they’ve missed the point of advertising. If I were they, I would send SOMETHING back. But all the same - I hope other sites take this approach. It could be the end to all the annoying flash ads I get, if instead I got these images everywhere!

Last month I posted about the evolving maps during the San Diego Firestorm 2007. Yesterday as I was sitting in a waiting room, I was browsing the Union-Tribune, and found this article going into a bit of the detail of how those maps were created. It still doesn’t talk much about what advances were made, but does describe the players, basically San Diego State University, a team from Google, a prof from UCSD, and a collection of worldwide researchers who focus on imaging all got together. Form the U-T article, I mainly glean that the map images were the result of taking map images from a wide variety of sources (satellite, aerial footage, thermal imaging), and using “geo-referencing” to align them all onto the same map.

Flash crowds are something that I think about a lot. This is mainly because it’s one of the prime challenges of building distributed systems.

Consider what happened in 1999 when Victoria’s Secret ran a Super Bowl ad announcing an online webcast of its Spring Fashion Show. The result was a sudden large volume of traffic to their site to view the webcast, so much that many customers were unable to view the webcast because the server could not handle the flash crowd.

A similar problem occurred after 9/11/2001, when everyone went to their favorite online news outlets for the emerging story.

What separates the two, of course, is that Victoria’s Secret planned their webcast (but failed to forsee the limits of their servers), where crisis situations are unpredicted, and generally not provisioned for.

This was clear in handling the San Diego firestorm last week in several ways, two of which I’ll mention here. What I find fascinating is how the people involved here had to adapt their technologies to handle the Crisis. In general, unexpected situations may always lead to this, and the people involved should largely be applauded. But at the same time, this presents us an opportunity to look at what happened to try and prepare automated systems for next time. Specifically, we need to improve or GIS/Mapping techniques, and our transparent web-content scalability techniques.

1. Basic information: Those of us who did not want our TVs constantly glued to a local news source (or radio) turned to the internet for updates. Furthermore, people worldwide with ties to San Diego, who don’t get local San Diego stations, tried to get that information online. This overloaded local servers and required adaptations.
   • San Diego County Emergency: This website was overloaded early on. There was a similar “official” website and phone system (211) for getting updates, but the phones apparenty had a 5 minute wait time, and website took over 15 minutes to load. At least the county emergency site would eventually load. This website hasn’t changed much since the beginning, despite a painfully small news clip box and downloads box which don’t work very well. The main changes here are (1) maps, which I’ll talk about later, and (2) apparently at some point Qualcomm is rumored to have been donating resources (bandwidth) to this site. That’s probably why it started loading better. I can only speculate that they wanted to keep information on this site so people would know it was authentic.
   • KPBS fire blog. KPBS realized their site would not stand the load, so they switched to using Twitter for updates. I didn’t try their site before they switched to Twitter, so I don’t know much about this one. I also don’t know much about Twitter, but it seems to be something designed for very short messages — like the size of an SMS. Using Twitter they were able to withstand the load, and people were able to use Twitter to subscribe, get content pushed to them, etc. KPBS was also the one to create a decent Google Map, which I’ll cover later.
   • The San Diego Union-Tribune fire blog. The San Diego U-T, a.k.a. SignOnSanDiego, started, like other sources, using headlines on their site. They quickly moved to a blog format to support the frequency of updates, but hosted at a url under their main site. Next, they realized that was too loaded, and split the blog out into a separate hostname (firestorm.signonsandiego.com), which allowed them to host it on other servers. This too could not keep up with the load, so finally they moved it to Blogger, the Google blog service. I was fascinated (and pleased) that both news sources had gone to public infrastructure broader than they alone could provide, and saw this as a success for big content installations. But still, this suggests we should look at server technologies which automatically replicate and scale content availability to match its demand, to avoid reconfigurations that might lose users.
2. Maps/Evacuation info. Perhaps the most important piece of information everyone wanted was where the fire was and where its headed. Much talk was made of the reverse-911 system (where 911 calls you to tell you their’s an emergency and you need to evacuate), which was very successful in alerting residents when they needed to evacuate. Of course, didn’t work for all — those of us without a landline would not get one (though now we know you can go here to add our non-land-lines (BEWARE, ONLY WORKS IN INTERNET EXPLORER). It also didn’t work for residents near Ramona, after their phone service was out due to the fire. In these cases, the old fashioned publis-safety-officer with loadspeaker and banging on door was the remaining solution. I myself signed up for updates as SMS messags on my phone (though I would have preferred a less expensive email option). But what we all wanted a place where we could just go and look at a map of evacuated and fire ares, which would be updated as time progressed. Here were our options:
   • No map: headlines only. If you know your regions of the town, you could mostly figure out what was going on from TV and radio news descriptions of where the fire was. But if you didn’t know the neighborhood names, you then had to go look them up, very inefficient.
   • TV maps. These maps were generally very bad. There was this one map they kept putting up with the evacuation areas, which was like 7 city names, and a fire symbol next to their name. Totally useless. Eventually, these switched to the county emergency maps (below), though they didn’t do much for giving credit here.
   • Site hosted maps. The U-T briefly tried to host a map on their site, but the load was too great, and it was updated too infrequently to be of any use.
   • Google maps. These were what I had great hopes for, but apparently the technology wasn’t quite there yet. The LA Times site had a map with the burn areas both of San Diego and LA, but it wasn’t very precise, and the fire icons on the map just didn’t give very good details. The best Google map was the one KPBS maintained, which they updated when they got the chance. It was fairly decent, and showed evacuation and burned areas, though there were a number of glitches. For example, a couple of times markers were placed in Las Vegas or LA, when they were supposed to be within San Diego County. Also, the Google Map never got to the point where it showed the active burn areas. It did, however, get linked from the main google maps page.

     Most interestingly though, twice on the KPBS twitter, they indicated they couldn’t update the map because of limitations, and they were working with Google to get improvements made. I’m not sure what (if anything) changed, but I was pleased to see they didn’t feel limited by the features, and that Google was willing to get involved. The google maps to me really showed the greated promise, because they were able to show satelite overlaid with fire information, street-level details, traffic information and also news blurbs on map icons. Problems, however, were that it was apparently too hard to update, the news blurbs were too big and thus were displayed poorly, and that you couldn’t turn on and off layers of the map (e.g. maybe you didn’t want to see the cluttered evacuation center icons or home destruction icons right now). As an attempt though, this gets the best credit, and should be the starting piont for future efforts. The SD County Emergency site needed to be able to easily update one or more map layers which people could download (from a scalable hosting site), and transparently add to the mapping program of their choice (be it google maps, MS live, or an offline map program. Over the county’s own maps, it provided very important details such as streets. GIS (Geographic information services) should be a major research front in this for future crises.

   • PDF downloads from the SD County Emergency site. In my opinion, these were the best of the maps, because they were most accurate, timely, and detailed. Unlike the other maps, these were the only ones to show active fire areas (though that took a few days to be added). At one point I realized that these were essentially the root of all maps — they were the input that KPBS was using for their google map, and all other news sources were using them and redistributing them. In fact, NBC San Diego actually was using these maps on the air, actually interacting with them in adobe acrobat reader. Meanwhile, they were telling viewers they too could download them from the NBC San Diego site (on the one hand, giving viewers the impression they made them, ont he other hand, taking important load away from the county emergency site, which was strained). Within this series of maps (and I’ve downloaded them all so I can see the progression of both the fire and the mapping techniques), it’s fascinating to see the progress. Originally, it was just the burn and evacuation areas. It broke up mandatory and voluntary evacuation areas with colors, and showed the burned area in solid red. But as the burn area grew, and some areas (some of which were burned because reopened), there were problems. Lack of transparency made it hard to see, and there were getting to be too many colors. (Is it burned, or is it ok to return?). So they eventually progressed to making the burned area translucent, and showing the active fire areas so (1) you could understand why they were letting people back into certain areas, and (2) you could see that an area had been both burned and was reopened. But the overlapping made it appear as a separate color, and was confusing unless you zoomed in. So eventually they just showed the fire perimiter with the active fire areas, so they could clearly show which areas were evacuated, and which were re-opened.

     Some confusing things remained to me. For example, un-inhabited areas were never shown as evacuated, but it was never obvious to me whether they were not evacuated because they weren’t inhabited, or because the fire wasn’t expected to go there. This because I didn’t know the geography of the county that well. In the most recent map I noticed they’ve marked some national forests, which make it easier to see this. Also, the lack of street-level details made it hard to tell where on the map you were. It did at least break up the map into the “Thomas-brothers” map grid, though since I don’t have one it didn’t do me any good. Next, I thought it was strange that the only download was a ~1MB PDF file, which no-doubt was getting a lot of hits. Seems like they could have saved a lot of bandwidth by offering the map in a 300KB JPEG format as well, which would have sufficed for many of us. Furthermore, the PDF had many layers, was large, and slow to render, having to be redrawn every time you zoomed or moved in the map.

     Example maps (for all maps, go to http://www.sdcountyemergency.com/).

     • First Emergency Map, 10/22, 9:00 am: This map was the first county-wide map with both Witch and Harris fires. It shows burn zone, evacuation zone, and voluntary evacuation zone. It also shows the 2003 fire perimeters, which was very interesting, but not so useful for evacuation. It is however, the best indicator of how the fire burned a different area this time. Problems include the different layerings, lack of Thomas Guide grid, too many dots for municipal stuff.
     • 10/23, 12:15 pm Map: Now they added the Thomas Guide. But there are spots where all three of fire, mandatory, and voluntary evacuations are layered, leaving one confused.
     • 10/23, 10:30 pm: So I’m noticing the 10:30 trend here, which I’m wondering if its the news cycle release. This map has areas people can return to, which is good, but they are adjacent in come cases to fire areas, leaving me confused as to why they can return. Soon the return areas will include burned areas, making it worse.
     • 10/24, 05:00 am: Now they’ve added the active burn area colors, which makes it much more clear why certain areas are allowed to return. But the solid fire color won’t work once they let people back in.
     • 10/25, 2:30 am: Here we see the overlapping burn and repopulation areas, which is very confusing, as it shows many colors in overlap. But otherwise, the map is looking pretty good.
     • 10/26, 7:00 pm: In this update, we now see the fire perimiter separately. The evacuation and repopulation areas are clear, as well as the active fire.
   • Several days into the event, they (San Diego County Emergency) released a Google earth data file. I have no idea what this contained or looked like, as I don’t have Google earth on my computer.