So far, we've seen basic Cascalog predicates and queries. We saw queries that pull data from one source generator and maybe include one predicate test. In this recipe, we'll see several more complex queries.
For this recipe, we'll need the same project.clj file and dependencies from the Initializing Cascalog and Hadoop for distributed processing recipe. We'll also use the Doctor Who companion data that we defined in that recipe. The source code for this data is available in the code for the book, and you can also download just the code from http://www.ericrochester.com/clj-data-analysis/data/companions.clj to create this dataset.
We'll start with simple queries and build up from there:
- First, let's take a look at a simple join:
user=> (?<- (stdout) [?name ?dr] (full-name ?c ?name) (doctor ?c ?dr)) … RESULTS ----------------------- Ace 7 Adam Mitchell 9 Adelaide Brooke 10 Adric 4 Adric 5 Amy Pond 11 Astrid Peth 10 …This pulls each companion's full name from one table and the numbers of Doctors they accompanied from another table. The binding for the companion's key,
?c, stays the same across all the generators and predicates for each row of the output. Also, notice that the second predicate(doctor ?c ?dr)can generate more than one row of output for each value of?c, as it did for Adric. - Let's take a look at one that's slightly more complex:
user=> (?<- (stdout) [?name ?dr ?actor ?tenure] (full-name ?c ?name) (doctor ?c ?dr) (actor ?dr ?actor ?tenure)) … RESULTS ----------------------- Ace 7 Sylvester McCoy 1987–89, 1996 Adam Mitchell 9 Christopher Eccleston 2005 Adelaide Brooke 10 David Tennant 2005–10 Adric 4 Tom Baker 1974–81 Adric 5 Peter Davison 1981–84 Amy Pond 11 Matt Smith 2010–present Astrid Peth 10 David Tennant 2005–10 …This time, we've reused the same query as the preceding one, but we've also included the table of actors who've played Doctor Who.
- Even with more than one generator, we can still include one or more test predicates. Here's a query that only returns companions for the modern-era Doctors:
user=> (?<- (stdout) [?name] (full-name ?c ?name) (doctor ?c ?dr) (>= ?dr 9)) … RESULTS ----------------------- Adelaide Brooke Amy Pond Astrid Peth Christina de Souza Craig Owens Donna Noble … - Test predicates can be used to do more than just filter data. We can also use them to generate data. For example, in the next query, we add a
modernflag to each row in order to indicate whether they are from the modern era or not. We do this by assigning the test predicate's output to a variable binding using the:>operator:user=> (?<- (stdout) [?name ?modern] (full-name ?c ?name) (doctor ?c ?dr) (>= ?dr 9 :> ?modern)) … RESULTS ----------------------- Ace false Adam Mitchell true Adelaide Brooke true Adric false Adric false Amy Pond true Astrid Peth true …
Out of the simple building blocks of generator and predicates, Cascalog makes it easy to build complex queries in an intuitive way. We'll see even more complex queries in the upcoming recipes, but composing generators and predicates and sharing bindings are the basis of all of them.