Google App Engine

The first major change is the use of Google App Engine to power SacTraffic.org. Heretofore SacTraffic had a backend powered by Perl on a LAMP-based system and relied on pretty much whatever hosting I could scrounge off friends for its home on the web.

Now, sitting on the shoulders of App Engine, with the firepower of Google behind it, I don’t have to worry so much where it’s hosted or how much traffic it can handle before it melts down. I know it can handle whatever gets thrown at it.

Also by using App Engine there’s an honest to goodness data store underneath it so this allows for more flexible and detailed queries on the CHP data. Data is now available via arbitrary queries, down to the CHP “Area” level, like ‘Placerville’ or ‘North Sacramento’.

Maps API v3

On the client end, the map engine behind SacTraffic has been updated to the Google Maps API version 3. This means, among other things, that it’s a lot faster to load.

I’ve also tweaked the look of the map a bit, but mostly because I could.

Atom Feeds instead of RSS

Because the CHP incidents are more data than content, I found that Atom feeds are better suited to syndicating them that RSS was. Along with the PubSubHubbub support that’s part of the new setup, the new Atom feeds are ideally suited to be the foundation for other systems (like the @sactraffic Twitter feed, which they do in-fact power).

More Live Cameras

The CA Department of Transportation has added quite a few cameras — like 25 more — since I last checked so I added them this time around. Sadly, they still use that nasty Microsoft ASF format so they still play like crap on non-MS systems, but when they do play they’re still pretty neat.

Optimized code

Along with the update to Google Maps API v3, I’ve gone through the other JavaScript, CSS and even HTML files to make the site better, stronger, faster. From using CSS sprites for the graphics to obscure JavaScript optimizations, I’ve probably tweaked on it.

More to come

I’ve got other tricks up my sleeve, like more mobile development and faster data updates, all just waiting on the time to implement them, so check back often.

Sacramento Connect is a network of high-quality news providers and bloggers in the Sacramento region and is the brain-child of The Sacramento Bee*. Tying all the affiliated sites together is a toolbar you’ll see at the bottom of all the pages, including SacTraffic.

The toolbar will show traffic related stories from the Sacramento Connect network, allow users to search the affiliated sites and allow greater access to SacTraffic content from the other Sacramento Connect sites.

All it all, I think it’s a nice addition, but then I’m biased…

*Full disclosure: I’m a web developer with the Sacramento Bee and my group was instrumental in the development and deployment of Sacramento Connect and the toolbar.

Anyway, I just finished another set of updates, though this time around the changes are more subtle. Outwardly I added support for geotagging the incident location in the Twitter updates so if you use a Twitter client that supports geotagging, the location of the incidents will appear with the tweet.

Behind the scenes I put in two additional changes. First I converted to Twitter’s OAuth mechanism for authentication, which should provide some more security under the hood and second, I switched to Google’s Closure compiler which provides better javascript compression than YUI Compressor which I adopted back in October.

Last, but not remotely least — and this was something I did back in October — the sactraffic.org code is publicly available via github under an OpenBSD-style BSD license. You can check out the code there and suggest changes.

By far the most important update was finally figuring out how to convert the proprietary coordinates provided in the CHP incident feed into real latitude and longitude for mapping. I literally had an epiphany in the shower and ran out to finish the conversions on paper. It turns out that it’s just math, man. Anyway, I’m still tweaking the scale (the further you get from Sacramento the farther off the points are) but now almost all the incidents map correctly.

The most obvious change of course is that now the site has some style to it. It was looking pretty drab so I dressed it up a bit (and I matched the Twitter page to it too). This in, turn lead to a custom iPhone site that only shows the incident list because having the list and the map was just too much on the small screen. I also got a set of really nice icons for the various traffic incidents so the map itself looks pretty good too.

Along with the style updates I integrated Google AdSense ads into just about everything. Its good for a few pennies here and there so I figured what the heck. Even the live camera pages have a new look with ads that match the width of the video.

Oh and I added more live cameras off the DOT video site too. I also re-enabled the cameras on Macs because they kinda sorta work… sometimes. They’re still freakin’ cool on Windows.

Finally, I started playing with Google’s Webmaster Tools and created a sitemap and submitted it to the big search engines. Maybe that is the reason for the spike in traffic .

Anyway, more to come…

*tap* *tap* is this thing on…

You (all 4 of you) may have noticed a little outage over the last few days. For whatever reason AT&T shutdown all of my ISP’s phone lines and this included their DSL lines. Since www.lectroid.net and www.sactraffic.org sit on a server in my closet, this meant they were offline. Oopsie.

Now I know s*** happens in technology but my ISP effectively went dark with only a single little blurb on their web site about their phones being out. No updates, no nothing and not even any mention of their DSL customers. Add to this, the alternate numbers they offered didn’t work. I seriously thought they might have gone out of business. Not really the behavior you want from an ISP.

So I went hunting for a new broadband provider. I’m in a semi-rual area so my options were limited, I was left with Comcast (my cable provider) and AT&T (my phone provider). Comcast was never a serious option for personal reasons so after a single, and rather pleasant, phone call to what turned out to be a call center in Texas, I’ll be getting 6 megs of DSL goodness directly from AT&T and installed on Tuesday. I’m looking forward to 4 times the bandwidth for even less than I was paying before.

This isn’t without some cost however. For almost 15 years I’ve used local ISPs for my internet service. Yeah, sometimes I paid a little more, but I liked being able to call (or IM) an admin directly if something wasn’t working and get it looked at, in fact I was often friends with them so I had the inside track. Oftentimes I was friends with the ISP owners as well. But the last guy I knew at my ISP, or at any ISP, left about 6 months ago, so it was time to move on.

The other change will be for the first time in almost 10 years I won’t have a fixed IP address, so I won’t be hosting my own stuff on a server in my closet. I’m still working out where the web server will go so you can expect another outage while I work that out (email, however, will be fine).

But 6 megs of DSL goodness…

The jQuery library allowed me to simplify the code a lot and fix some minor annoyances (yay, opened incidents stay open through refreshes).

Of course I made some minor UI changes — this you can see — mostly to make things a little easier on odd size displays (looking at you, iPhone).

More notes in the About page.

It didn’t last long. My little Via C3 powered server I run on my little tiny DSL line almost immediately melted down. Interestingly it wasn’t the server load, as Sactraffic is all client-based code, and it wasn’t the bandwidth as my 1.5mbps/384kbps DSL line never peaked past 70% (though that in itself is well into “yellow-light” area)… it was the sheer number of requests and the rate they came in at that did the server in.

Suddenly there was a whole lot of:

[error] (35)Resource temporarily unavailable: fork: Unable to fork new process

in the logs. I was SacBee-dotted.

My home server was just not configured for that kind of traffic. I did some quick Googling and as I suspected the default kernel settings in OpenBSD are intentionally conservative (secure by default). I made a quick call, while I’m sure I could have tweaked it and gotten things humming, it would still be a ancient homebuilt server on a home DSL line.

Punt.

It took a bit for the firehose to turn off and things to calm down. What’s interesting is that due (I assume) to the massive number of incidents the CHP was experiencing the CHP XML feed that powers Sactraffic was also experiencing problems so at least I wasn’t alone .

The trick was to just have the feeds open in their own pop-up page. Not nearly as elegant, but effective nonetheless.

They’re all ASX/ASF format files which means you pretty much need Windows Media Player on, well Windows. Sorry about that. On Macs I’ve had some limited success with Flip4Mac to play the feeds. YMMV.

So go check it out.

I moved things around, primarily to look better on smaller browsers (am talking to you iPhone). I also added real estate for site-specific updates (like this post) to automagically populate when, well, there are updates.

I then added microformats, specifically the vevent and geo formats. Not a whole lot of value to the casual browser, but full geek points get awarded anyway.

Finally I pulled the Sac DOT video cameras, because basically they sucked. I have a huge list of CalTrans video cameras to add here soon, it’s much better quality video but the immediate problem is that it’s 640×480 and doesn’t scale down to fit.

After thinking it through a little bit I decided a while back to run with the Traffic on Twitter thing and do it up right. The result: sactraffic.org.

I used that same XML feed from the CHP and in addition to Twittering new incidents I created a local JSON feed of just Sacramento area incidents and “mashed up” some geocoding/mapping, local news, videos and even local weather.

Hopefully this makes a simple, yet useful, tool for folks as there are RSS feeds that can be tailored to a particular set of streets or freeways (think: check your commute in your RSS reader before you leave for the day) and what’s even slicker is using Twitter’s new “track” feature to get live updates on traffic incidents at the street or freeway level right to your phone.

Most of this was made possible by the Google Feed and Maps APIs.

It’s been a fun hobby project and was a good refresher on JavaScript which I was getting rusty at (and perhaps still am). Clearly my design skills still suck (hey, anyone have a line on decent traffic icons for Google Maps?).

I’m serious when I say “hobby project” as I’ve tried really hard to keep work on this limited to my spare time at home. I have plenty of work-type work to do at work, messing around with stuff like this at work would probably be frowned upon.

Anyway, I stuck a big ol’ Google AdSense ad on it so if it gets popular maybe I can subsidize my caffeine habit .