Map operators transform data in the map phase, with one input row being mapped to one output row. A simple example of a custom map operator is an operator that triples all the numbers that pass through it:

(defmapfn triple-value [x] (* 3 x))

Something similar to this can be used to rescale all the values in a field.

Map concatenation operators transform data in the map phase, but each input row can map to one output row, many output rows, or none. These operators return a sequence, and each item in the sequence is a new output row. For example, this operator splits a string in whitespace, and each token is a new output row. We'll use this in the following predicate to count the number of names that each companion had:

(defmapcatfn split [string] (string/split string #"s+"))
(?<- (stdout) 
     [?name ?count]
     (full-name _ ?name) (split ?name :> ?token) (c/count ?count))

Filter operators remove rows from the output in the map phase. They take one item, and they return a Boolean. If true, the item should be included in the results; if false, then the item should be held back. For example, this filter returns true if the input is an even number, and we use it in a query to return only the companions who accompanied an even number of doctors:

(deffilterfn is-even? [x] (even? x))
(?<- (stdout) 
[?companion ?dr-count]
     (doctor ?companion _)
     (c/count ?dr-count)
     (is-even? ?dr-count))

Buffers operators work in the reduce phase. They process a group of rows as a single input. They take an entire list of input rows to process and return one or more items for the output. For example, this buffer operator takes rows of strings and returns the total number of characters in all the strings. We use it in the query to count the characters in the full names of the companions for each doctor:

(defbufferfn count-chars [strings]
  [(reduce + 0 (mapcat #(map count %) strings))])
(?<- (stdout) 
[?dr ?companion-chars]
     (doctor ?c ?dr)
     (full-name ?c ?name)
     (count-chars ?name :> ?companion-chars))

Aggregate functions work in the reduce phase to combine input rows into one value. Compared to buffer operators, aggregate operators are in some ways more flexible—they can be used with other aggregators—but are more restricted in other ways.

Each aggregator function has to be able to be called with no parameters, one parameter, or two parameters. The call with no parameters returns the initial value, while the call with one parameter takes the aggregator's state and returns the final value. Moreover, the call with two parameters takes the state and a new value and folds the two together into a new state.

For example, this returns the average length of the names of each doctor's companions:

(defaggregatefn mean-count
  ([] [0 0])
  ([[n total] string]
   [(inc n) (+ total (count string))])
  ([[n total]] [(float (/ total n))]))
(?<- (stdout) 
[?dr ?companion-chars]
     (doctor ?c ?dr)
     (full-name ?c ?name)
     (mean-count ?name :> ?companion-chars))

Parallel aggregate operators are the most restricted, but they also give the best performance. Unlike the rest, they can be run in the map phase of the computation. These aggregators are defined by two functions. One function is called on each row, and one is called to combine the results of calling the first function on two rows.

This example returns the average length of the name of each doctor's companions:

Having so many options to build operators provides us with a lot of flexibility and power in how we define and create queries and transformations in Cascalog. This allows you to create powerful, custom workflows.