Building Neat Stuff with Dojo and YQL

I’m in love with YQL. Never heard of it? I’m not surprised. To borrow a term from the hip-hop community, YQL is probably the most slept-on technology in the field of web development. YQL is defined by Yahoo! as: ‘an expressive SQL-like language that lets you query, filter, and join data across Web services.’ In a nutshell, YQL lets you query for interesting data that can be easily consumed by a web app. The best way to understand what YQL is about is to play around with the YQL Console. Be sure to click the ‘Show Community Tables’ Link in the right sidebar to reveal all of the data sources. Look at this stuff! Want to find favorably reviewed sushi restaurants in San Francisco? Here ya go. (Click the ‘Test’ button to fetch the data). How about the most recent Tweets with the #dojo hashtag? Neat, yes?

Getting YQL Data Into Your Page

The coolest thing about YQL is how easy it is to get this great data into your app. Historically speaking it has been a pain to do mashup style apps with JavaScript due to the fact that browsers make it all-but-impossible to invoke AJAX requests across domain boundaries. YQL uses a clever technique known as JSON-P to work-around these cross-domain restrictions. JSON-P utilizes the fact that browsers do allow src attributes of script elements to be cross-domain. Thus it’s perfectly fine to include a script element with a Yahoo! domain in your page. Here is what that looks like for the Twitter query that I linked to earlier.

<script type="text/javascript"
src="http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20twitter.search%20where%20q%3D'%23dojo'%3B&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=cbfunc"></script>

The JavaScript that the above script element returns to us consists entirely of a block of JSON data wrapped in a function invocation:

cbfunc({
 ...
  "results": {
   "results": [
    {
     "to_user_id": "null",
     "text": "Understanding Widget - The dojo toolkit http://ow.ly/5kMmW #dojo",
     "id": "82046157043408896",
     "from_user_id": "2632678",
     "geo": "null",
     "iso_language_code": "en",
     "to_user_id_str": "null",
     "source": "<a href="http://www.hootsuite.com" rel="nofollow">HootSuite</a>"
    },
...

So, if we define a function in our page with the same name as the invocation supplied by YQL, that function will be called and passed the JSON ‘payload’:

// Somewhere in our page, above the YQL script element.
// This function must be called 'cbfunc' as that is the
// function that the YQL-returned JavaScript is calling
function cbfunc(yqlData) {

    // Do something cool with yqlData
    // ...

}

We now have a page that can consume data from Twitter. And we are doing it all client-side! As you can probably tell, access to great data with no server-side setup makes me a happy developer.

Using Dojo to Execute Dynamic YQL Queries

Using YQL as demonstrated above is cool but also has the limitation that your query to YQL is completely static. Sure, the #dojo hashtag is interesting, but perhaps we want to build a user interface that allows a user to choose a different hashtag to search. Since JSON-P requires that we include our query via the src attribute in a script element in the page, providing the ability to do dynamic queries requires us to dynamically inject script elements. This sounds tricky, but don’t be scared, we have Dojo. Dojo provides an API called dojo.io.script that makes dynamic script element injection easy. But an even slicker way of invoking YQL via Dojo is by using this module that I found on GitHub. It abstracts away the usage of dojo.io.script, and let’s us interact with YQL at the ‘query’ level. Using this module, we can write code like this:

function doTwitterSearch(searchTerm) {
  var query =
    "select * from twitter.search where q='" + searchTerm + "'";
  dojox.yql(query, {
    load: function(yqlData){
      // yqlData is the YQL JSON payload, do something neat with it
    ...
    }
 });
}

If that doesn’t strike you as cool, I’m sorry, but I don’t think that we can be friends. To use this module, download a Dojo Source Build and then plop the files from the GitHub repo into the dojox folder.

With those files in place, we can construct a page like this. This page loads Dojo, requires our dojox.yql module and demonstrates a simple YQL query to fetch the weather in Raleigh:

<!DOCTYPE html>
<html>
  <head>
  <meta charset="utf-8">
  <title>YQL and Dojo</title>
</head>
<body></body>
<!-- Include dojo.js -->
<script type="text/javascript" src="dojo/dojo/dojo.js"></script>

<!-- dojo.require the YQL module from GitHub -->
<script>
    dojo.require("dojox.yql");
</script>

<!-- Execute a query -->
<script type="text/javascript">
dojo.ready(function(){
  var query =
    "select * from google.igoogle.weather where weather='Raleigh'";
	dojox.yql(query, {
		load: function(weatherData){
		  console.log(weatherData);
	    }
	});
});
</script>
</html>

OK this is fun. But, I have a job.

You may be wondering if YQL has any real usage outside of Hack Nights. It does. The area of day-to-day software development that YQL really shines is the all-important product demo. Almost all demo’s involve the use of ‘dummy’ data. Demo’s are often done at the prototype or proof-of-concept stage of a project and at this point, there likely isn’t any real data to be had. When you find yourself needing some dummy data, you have two basic strategies to choose from:

• The Boring Strategy Dummy up some simple data and in-line it into your app. Don’t think too hard about this. Just create an array of Employees or something.
• The Fun Strategy Put some thought into what data might be interesting for your demo. Use YQL to fetch this data. This will only take slightly more time than The Boring Strategy.

The Fun Strategy is always better. I’ll show you what I mean. Click the button below to add a few dummy ‘Employee’ records to the table.

Add Employee

Yawn. OK, now click this button a few times. It queries Flickr for photographs of animals classified in the Order Rodentia, sorted by ‘Interesting-ness’.

Add Interesting Rodent

How many employees did you add? And how many rodents? I bet you’ve got about 3 employees, and about 3 dozen of God’s more fluffy creatures. The point that I’m trying to make here is that all data are not created equal. Your demo will be much more compelling if you give your audience something that makes them smile. Remember, people expect that your talk is going to be boring; it is a software demo after all. If you can beat the odds and grab people’s attention in the early going, they will stay engaged throughout the entirety of the demo, and you stand a much better chance of driving home whatever point it is that you are trying to make.

There’s a reason that most people use simple, hard-coded data in their demo’s. There was a time when using interesting data was simply not worth the effort. It usually involved janky server-side code that did something nefarious like scrape the HTML of a Yahoo! or Google property. YQL eliminates that hurdle and makes it dead-simple to pull fun data into your apps.

An Example of a YQL-Driven App

A few months ago, I used this exact technique for a personal project involving stock quote data. I’ve always loved quirky data visualizations, and for as long as I can remember I’ve had this idea for a visualization of stock quotes using fish. (Yes, I am crazy). I call it QuoteBowl. The basics of QuoteBowl involve a portfolio of stocks with each stock symbol in the portfolio represented by a goldfish in a fish bowl. If the stock is up on the day, the fish is rendered in a green color. If the stock is down, the fish is red. From there, the possibilities for fun visualizations are limitless. You could have different types of fish for different classes of stocks. You could draw the fish bigger or smaller depending on how much the stock was up or down. You could make the fish turn over and float upside down if the stock fell more than ten percent, etc, etc. I took a couple weekends and got a decent start. Check it out for yourself at quotebowl.com. Add some fish. It’s fun!

Here is the code that drives the data. I am using the Yahoo! Finance YQL service to fetch quotes on the symbols that a user has added to the fish bowl. The symbols are stored as an array of strings called userSymbols:

var query =
  'select symbol, LastTradePriceOnly, ChangeRealtime, ChangeinPercent from yahoo.finance.quotes where symbol in (' + userSymbols.join() + ')',

dojox.yql(query, {
  load: function( quoteData ) {
    // Add/Update the fish
  }

Conclusion

Software Demo’s are serious business. Everyday millions of dollars change hands, or don’t change hands, depending on how well a particular software demonstration went. That is not to say that every demo is mission critical, but regardless, the ability to give a great demo is a skill that every engineer should aspire to have. Given the right situation YQL may be the perfect tool for giving your demo the extra splash of flavor that it needs. Happy Demo’ing, and may all your fish be green.