11. Simplifying Datomic Syntax by Writing a DSL
The context here is that creating schema and writing inserts for Datomic are unnecessarily verbose, especially for people who think in Clojure. The idea is to take advantage of this “expressiveness gap” and use Clojure to write a DSL to make writing Datomic schema and inserts less verbose. We’ll reduce and map over Clojure maps to generate the Datomic syntax for schema creation and adding datoms. Instead of using the Datomic shell, we’ll use an in-memory version of Datomic that we access from Clojure.
Assumptions
In this chapter we assume the following:
You have Leiningen installed.
Using the Datomic API programmatically is insufficient for your environment, and you need to make your database changes for a release explicit in a file (common in large, heavily audited, and financial environments).
You’re aware that a Datomic Schema doesn’t define the set of attributes that can be associated with entities,1 and that this association is made by the application.
1. http://docs.datomic.com/schema.html
Benefits
In this chapter you’ll learn how to write a DSL reader and code generator in Clojure. This knowledge will be helpful when you write Datomic syntax, which can be cumbersome.
Imagine we wanted to create a new schema in Clojure for a library use case. A naive schema would have a schema for Book, with an Author field and a Title field. To represent this in Datomic, we’d write a schema like this:
[
;; book
{:db/id #db/id[:db.part/db]
:db/ident :book/title
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A book's title"
:db.install/_attribute :db.part/db}
{:db/id #db/id[:db.part/db]
:db/ident :book/author
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A book's author"
:db.install/_attribute :db.part/db}
]
We understand that representing a table row concept in a key-value store implementation drives some of the complexity here. But at the same time, for a reader who comes from a relational database background, there appears to be lots of repetition there.
Ideally, we just want to do something like this:
(create-schema :book {:title :string :author :string})
To use a relational database mindset, you can see that the essentials of the schema name, the ‘column name’, and ‘column type’ are there.
The Recipe—Code
The recipe involves these steps:
1. Create a new Leiningen project datomic-dsl in your projects directory, and change to that directory:
lein new app datomic-dsl
cd datomic-dsl
2. Modify the project.clj to look like the following:
(defproject datomic-dsl "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.6.0"]
[com.datomic/datomic-free "0.9.5130"]]
exclusions [joda-time]]]
:main ^:skip-aot datomic-dsl.core
:target-path "target/%s"
:profiles {:uberjar {:aot :all}})
3. Create the file src/datomic_dsl/create.clj with the following contents:
(ns datomic-dsl.create
(:require
[datomic.api :as d]))
(defn create-schema
"Given a schema name and some schema attribute key-value pairs,
create an insertable datomic schema."
[schema-name & attributes]
(let [fields (apply hash-map attributes)]
(mapcat (fn [[attribute-name attribute-type]]
[{:db/id (d/tempid :db.part/db)
:db/ident (keyword schema-name attribute-name)
:db/valueType (keyword "db.type" attribute-type)
:db/cardinality :db.cardinality/one
:db/doc (str "A " schema-name "'s " attribute-name)
:db.install/_attribute :db.part/db}])
fields)))
4. Create the file test/datomic_dsl/create_test.clj with the following contents:
;lein test :only datomic-dsl.create-test
(ns datomic-dsl.create-test
(:require [clojure.test :refer :all]
[datomic-dsl.create :refer :all]
[datomic-dsl.common-test :refer :all]
[clojure.pprint :refer :all]
[datomic.api :as d]))
(def book-schema
"Use the API we've created to create some Datomic schema syntax.
Add a pprint to see what this looks like."
(vec
(create-schema "book"
"title" "string"
"author" "string")))
;(pprint book-schema)
(def trans-result
"Add the schema to the database. Save the result in a symbol if we need to
look at it."
(d/transact conn book-schema))
(def book-schema-query-result
"Query the schema to see if it exists - schemas are entities too!"
(d/touch (d/entity (d/db conn) :book/title)))
(deftest schema-created-test
(testing "Is the schema created with a book title?"
(is book-schema-query-result)))
5. Create the file src/datomic_dsl/add.clj with the following contents:
(ns datomic-dsl.add
(:require
[datomic.api :as d]))
(defn convert-dsl-to-keywords-fn
"Pass this in as a parameter so we can use it later."
[schema-name]
(fn [[attribute-key attribute-value]]
;Build a vector for the datomic fact add, if the value is a map then
we'll set the ID
[(keyword schema-name attribute-key) attribute-value]))
(defn convert-to-map
"Format the input vectors for the Datomic map syntax."
[input]
[(apply hash-map (first (concat input)))])
(defn add-datom
"Given a schema name and some key-value pairs, generate a Datomic insert
string. Note nested values are out of scope."
[schema-name attributes]
(let [convert-dsl-to-keywords (convert-dsl-to-keywords-fn schema-name)]
(convert-to-map
[(concat [:db/id (d/tempid :db.part/user)]
(mapcat convert-dsl-to-keywords attributes))])))
6. Create the file test/datomic_dsl/add_test.clj with the following contents:
;lein test :only datomic-dsl.add-test
(ns datomic-dsl.add-test
(:require [clojure.test :refer :all]
[datomic-dsl.add :refer :all]
[datomic-dsl.common-test :refer :all]
[datomic-dsl.create-test :refer :all]
[clojure.pprint :refer :all]
[datomic.api :as d]))
(def book-add
"Define our data structure and convert it to Datomic syntax."
(add-datom "book"
{"title" "Mutiny on the Bounty"
"author" "Charles Nordhoff"}))
;(pprint book-add)
(def add-result
"Keep a reference to the transaction result if we need to look at it
later."
(d/transact conn book-add))
(def book-datom
"Query against the result. Note the full stop in the query to find the
first one."
(d/touch
(d/entity
(d/db conn)
(d/q '[:find ?e . :where [?e :book/author]]
(d/db conn)))))
;(pprint book-datom)
(deftest datom-added-test
(testing "Is the datom added in the database?"
(is (= 2 (count book-datom)))))
7. Create the file src/datomic_dsl/create_nested.clj with the following contents:
(ns datomic-dsl.create-nested
(:require
[clojure.zip :as zip]
[datomic.api :as d]))
(defn map-zip
"Define a custom zipper for maps similar to zip/vector-zip and zip/
seq-zip."
[m]
(zip/zipper
(fn [x] (or (map? x) (map? (nth x 1))))
(fn [x] (seq (if (map? x) x (nth x 1))))
(fn [x children]
(if (map? x)
(into {} children)
(assoc x 1 (into {} children))))
m))
(defn dsl-zipper
"Given a dsl, return a zipper so its nested nodes can be walked with a HOF
like map."
[dsl]
(map-zip dsl))
;http://josf.info/blog/2014/04/14/seqs-of-clojure-zippers/
(defn zip-nodes
"Returns all nodes in loc."
[loc]
(take-while (complement zip/end?) ;take until the :end
(iterate zip/next loc)))
(defn create-schema-nested-fn
"When mapped onto a zipped node of a nested schema dsl, create
corresponding dsl entity syntax."
[schema-name zipped-node]
(if (= (type (zip/node zipped-node)) clojure.lang.MapEntry)
(let [elem (zip/node zipped-node)
attr-key (key elem)
attr-val (val elem)
has-parent (not (nil? (zip/up (zip/up zipped-node))))
parent (if has-parent (first (zip/node (zip/up zipped-node)))
schema-name)
is-nested (map? attr-val)
first-val-val (if is-nested :db.type/ref (keyword "db.type"
attr-val))
doco (str "A " parent "'s " attr-key)]
{:db/id (d/tempid :db.part/db)
:db/ident (keyword parent attr-key)
:db/valueType first-val-val
:db/cardinality :db.cardinality/one
:db/doc doco
:db.install/_attribute :db.part/db})))
(defn create-schema-nested
"Given a schema name and a dsl input, return Datomic syntax for creating
nested schema entities."
[schema-name dsl]
(filter (comp not nil?)
(map #(create-schema-nested-fn schema-name %)
(zip-nodes (dsl-zipper dsl)))))
8. Create the file test/datomic_dsl/create_nested_test.clj with the following contents:
;lein test :only datomic-dsl.create-nested-test
(ns datomic-dsl.create-nested-test
(:require [clojure.test :refer :all]
[datomic-dsl.create-nested :refer :all]
[datomic-dsl.common-test :refer :all]
[clojure.pprint :refer :all]
[datomic.api :as d]))
(def borrowevent-schema-dsl
"Define our data structure."
{"book" {"title" "string"
"author" "string"}
"borrower" {"name" "string"}
"date" "string"})
(def nested-book-schema
"Use the API we've created to create some Datomic schema syntax.
Add a pprint to see what this looks like."
(create-schema-nested "borrowevent" borrowevent-schema-dsl))
(def trans-result
"Add the schema to the database. Save the result in a symbol if we need to
look at it."
(d/transact conn nested-book-schema))
;(pprint nested-book-schema)
(def book-schema-query-result
"Query the schema to see if it exists - schemas are entities too!"
(d/touch (d/entity (d/db conn) :borrower/name)))
(deftest schema-created-test
(testing "Is the schema created with a borrower name?"
(is book-schema-query-result)))
9. Create the file src/datomic_dsl/add_nested.clj with the following contents:
(ns datomic-dsl.add-nested
(:require
[datomic.api :as d]
[clojure.zip :as zip]
[datomic-dsl.create-nested :refer [map-zip dsl-zipper zip-nodes]]))
(defn up-stepper
"Step over the map-entries up."
[node]
(take-while (complement nil?)
(iterate zip/up node)))
(defn zip-node-map-entries
"If this is a map-entry - get the key out of the zipper."
[me]
(if (= (type (zip/node me)) clojure.lang.MapEntry)
(key (zip/node me))))
(defn steps-up
"Keys to the top for a given node."
[node]
(filter (comp not nil?)
(map zip-node-map-entries
(up-stepper node))))
(defn map-key-seqs
"For a given dsl, get a sequence of steps to each of the nested nodes."
[dsl]
(filter
(comp not empty?)
(map steps-up
(zip-nodes
(dsl-zipper dsl)))))
(defn get-key-sequences
"Get each of the nested nodes as a flat vector of nodes to map over."
[nested-dsl]
(let [nest-datom-dsl-zipper (map-zip nested-dsl)]
(map vec (map-key-seqs nested-dsl))))
(defn append-in
"Given a key sequence - append a value at the nested location."
[nested-coll nested-key-seq param]
(assoc-in nested-coll nested-key-seq
(conj (get-in nested-coll nested-key-seq) param)))
(defn replace-children
"For a map of maps - replace the children maps with their nested dbids."
[[k v]]
(if (= clojure.lang.PersistentHashMap (class v))
[k (:db/id v)]
[k v]))
(defn add-matching-dbids-to-nested-child
"Map this onto each of the nested children using a key-sequence to give
them a dbid."
[nest-datom-dsl-input current-seq]
(let [current-seq-rev (reverse current-seq)
parent-seq (drop-last current-seq)
last-two (take-last 2 current-seq-rev)
dbid (d/tempid :db.part/user)
child-dbid {:db/id dbid}
parent-keyword (case (count last-two)
2 (keyword (first last-two) (second last-two))
1 (keyword (first last-two)))
parent-dbid (if (> (count current-seq) 1) {parent-keyword dbid})
child-replace (append-in nest-datom-dsl-input current-seq-rev
child-dbid)
parent-replace (merge child-replace parent-dbid)]
parent-replace))
(defn key-sequences
"Get the key sequences from the dsl and filter out the top-level ones and
return a set of vectors."
[dsl]
(set
(map vec
(map rest
(filter #(> (count %) 1)
(get-key-sequences dsl))))))
(defn nested-insert-dsl
"Step through the existing dsl and add matching dbids to future parent and
children entities."
[schema-name dsl]
(reduce add-matching-dbids-to-nested-child
dsl (key-sequences dsl)))
(defn convert-key-sequence-fn
"For a given dsl entry with dbids - convert it to Datomic syntax - and
append it to the results"
[nested-dsl]
(fn [key-sequence]
(let [local-datom (get-in nested-dsl key-sequence)
db-id-ref (:db/id local-datom)
local-datom-minus-dbid (dissoc local-datom :db/id)
local-datom-minus-children (into {} (map replace-children
local-datom-minus-dbid))
parent-name (take-last 1 key-sequence)
last-two (take-last 2 key-sequence)
last-one (first (take-last 1 key-sequence))
parent-keyword (case (count last-two)
2 (keyword (first last-two) (second last-two))
1 (keyword (first last-two)))
new-keys (into {} (map (fn [[k v]] [k (keyword (str last-one "/"
k))]) local-datom-minus-children))
new-map (clojure.set/rename-keys local-datom-minus-children
new-keys)
new-map-with-dbid (merge new-map {:db/id db-id-ref})]
new-map-with-dbid)))
(defn key-sequences-rev
"Given a dsl - return the key-sequences to each node of the nested
structure in reverse order"
[dsl]
(map reverse (key-sequences dsl)))
(defn convert-dsl-to-datomic-syntax
"Map over the key-sequences to step through the dsl with dbids and return
datomic syntax."
[schema-name dsl]
(let [dsl-with-dbids (nested-insert-dsl schema-name dsl)]
(let [convert-key-sequence (convert-key-sequence-fn dsl-with-dbids)]
(map convert-key-sequence (key-sequences-rev dsl)))))
10. Create the file test/datomic_dsl/add_nested_test.clj with the following contents:
;lein test :only datomic-dsl.add-nested-test
(ns datomic-dsl.add-nested-test
(:require [clojure.test :refer :all]
[datomic-dsl.add-nested :refer :all]
[datomic-dsl.common-test :refer :all]
[datomic-dsl.create-nested-test :refer :all]
[clojure.pprint :refer :all]
[datomic.api :as d]))
(def nest-datom-dsl
"Define our data structure."
{"borrowevent"
{"book" {"title" "Mutiny on the Bounty"
"author" "Charles Nordhoff"}
"borrower" {"name" "John Smith"}
"date" "today"}})
(def nested-datomic-add-syntax
"Convert our data structure to datomic syntax."
(convert-dsl-to-datomic-syntax "borrowevent" nest-datom-dsl))
;(pprint nested-datomic-add-syntax)
(def add-result
"Keep a reference to the transaction result if we need to look at it
later."
(d/transact conn nested-datomic-add-syntax))
(def borrower-datom
"Query against the result. Note the full stop in the query to find the
first one."
(d/touch
(d/entity
(d/db conn)
(d/q '[:find ?e . :where [?e :borrower/name]]
(d/db conn)))))
;(pprint borrower-datom)
(deftest datom-added-test
(testing "Is the datom added in the database?"
(is (= 1 (count borrower-datom)))))
11. Create the file test/datomic_dsl/common_test.clj with the following contents:
(ns datomic-dsl.common-test
(:require [clojure.test :refer :all]
[datomic.api :as d]))
(def uri "datomic:mem://datomic-dsl") (d/create-database uri)
(def conn (d/connect uri))
12. Remove the file test/datomic_dsl/core_test.clj.
Testing the Solution
Several solutions are tested in this section: create schema, add datom, create nested schema, and add nested datom.
Testing Create Schema
Follow these steps to test create schema:
1. Run the following:
lein test :only datomic-dsl.create-test
You should get the following result:
lein test datomic-dsl.create-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
This means it was successful and that the schema was added to the in-memory Datomic database.
2. We can’t see the output. Modify the file test/datomic_dsl/create_test.clj and uncomment the line to make it look like this:
(pprint book-schema)
3. Now run the lein test :only datomic-dsl.create-test command from above once more. You should get the following result:
[{:db/id {:part :db.part/db, :idx -1000000},
:db/ident :book/author,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's author",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000001},
:db/ident :book/title,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's title",
:db.install/_attribute :db.part/db}]
lein test datomic-dsl.create-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
We can see the Datomic schema syntax being created before it is applied to the database and run. It is interesting that the Datomic syntax above came from our simple DSL:
(create-schema "book"
"title" "string"
"author" "string")
Testing Add Datom
Follow these steps to test add datom.
1. Run the following:
lein test :only datomic-dsl.add-test
2. You should get the following result:
lein test datomic-dsl.add-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
This means it was successful and that the datom was added to the in-memory Datomic database.
3. We can’t see the output. Modify the file test/datomic_dsl/add_test.clj and uncomment the lines, to make them look like this:
(pprint book-add)
...
(pprint book-datom)
4. Now run the lein test :only datomic-dsl.add-test command above once more. You should get the following result:
[{:db/id {:part :db.part/db, :idx -1000000},
:db/ident :book/author,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's author",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000001},
:db/ident :book/title,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's title",
:db.install/_attribute :db.part/db}]
[{:book/author "Charles Nordhoff",
:db/id {:part :db.part/user, :idx -1000002},
:book/title "Mutiny on the Bounty"}]
{:book/author "Charles Nordhoff", :book/title "Mutiny on the Bounty", :db/id
17592186045418}
lein test datomic-dsl.add-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
We can see the Datomic schema syntax being created before it is applied to the database and run. We also see the result of querying the database and getting our datom back:
{:book/author "Charles Nordhoff", :book/title "Mutiny on the Bounty", :db/
id 17592186045418}
It is interesting that the Datomic syntax above came from our simple DSL:
(add-datom "book"
{"title" "Mutiny on the Bounty"
"author" "Charles Nordhoff"})
Testing Create Nested Schema
Follow these steps to test created nested schema:
1. Run the following:
lein test :only datomic-dsl.create-nested-test
You should get the following result:
lein test datomic-dsl.create-nested-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
This means it was successful and that the schema was added to the in-memory Datomic database.
2. We can’t see the output. Modify the file test/datomic_dsl/create_nested_test.clj and uncomment line 25 to make it look like this:
(pprint nested-book-schema)
3. Now run the lein test :only datomic-dsl.create-nested-test command from above once more. You should get the following result:
({:db/id {:part :db.part/db, :idx -1000000},
:db/ident :borrowevent/book,
:db/valueType :db.type/ref,
:db/cardinality :db.cardinality/one,
:db/doc "A borrowevent's book",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000001},
:db/ident :book/author,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's author",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000002},
:db/ident :book/title,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's title",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000003},
:db/ident :borrowevent/date,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A borrowevent's date",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000004},
:db/ident :borrowevent/borrower,
:db/valueType :db.type/ref,
:db/cardinality :db.cardinality/one,
:db/doc "A borrowevent's borrower",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000005},
:db/ident :borrower/name,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A borrower's name",
:db.install/_attribute :db.part/db})
lein test datomic-dsl.create-nested-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
We can see the Datomic schema syntax being created before it is applied to the database and run. It is interesting that the Datomic syntax above came from our simple DSL:
(def borrowevent-schema-dsl
"define our data structure"
{"book" {"title" "string"
"author" "string"}
"borrower" {"name" "string"}
"date" "string"})
Testing Add Nested Datom
Finally, to test add nested datom, follow these steps:
1. Run the following:
lein test :only datomic-dsl.add-nested-test
You should get the following result:
lein test datomic-dsl.add-nested-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
This means it was successful and that the datom was added to the in-memory Datomic database.
2. We can’t see the output. Modify the file test/datomic_dsl/add_nested_test.clj and uncomment the lines to make them look like this:
(pprint nested-datomic-add-syntax)
...
(pprint borrower-datom)
3. Now run the lein test :only datomic-dsl.add-nested-test command from above once more. You should get the following result:
({:db/id {:part :db.part/db, :idx -1000000},
:db/ident :borrowevent/book,
:db/valueType :db.type/ref,
:db/cardinality :db.cardinality/one,
:db/doc "A borrowevent's book",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000001},
:db/ident :book/author,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's author",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000002},
:db/ident :book/title,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A book's title",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000003},
:db/ident :borrowevent/date,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A borrowevent's date",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000004},
:db/ident :borrowevent/borrower,
:db/valueType :db.type/ref,
:db/cardinality :db.cardinality/one,
:db/doc "A borrowevent's borrower",
:db.install/_attribute :db.part/db}
{:db/id {:part :db.part/db, :idx -1000005},
:db/ident :borrower/name,
:db/valueType :db.type/string,
:db/cardinality :db.cardinality/one,
:db/doc "A borrower's name",
:db.install/_attribute :db.part/db})
({:db/id {:part :db.part/user, :idx -1000006},
:borrowevent/borrower {:part :db.part/user, :idx -1000008},
:borrowevent/date "today",
:borrowevent/book {:part :db.part/user, :idx -1000007}}
{:db/id {:part :db.part/user, :idx -1000007},
:book/title "Mutiny on the Bounty",
:book/author "Charles Nordhoff"}
{:db/id {:part :db.part/user, :idx -1000008},
:borrower/name "John Smith"})
{:borrower/name "John Smith", :db/id 17592186045420}
lein test datomic-dsl.add-nested-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
We can see the Datomic schema syntax being created before it is applied to the database and run. We can also see the result of querying for part of our datom (the borrower) in the database:
{:borrower/name "John Smith", :db/id 17592186045420}
It is interesting that the Datomic syntax above came from our simple DSL below:
(def nest-datom-dsl
"define our data structure"
{"borrowevent"
{"book" {"title" "Mutiny on the Bounty"
"author" "Charles Nordhoff"}
"borrower" {"name" "John Smith"}
"date" "today"}})
Well done! Our DSL for Datomic nested data is working.
Notes on the Recipe
It should be noted that a Datomic schema does not define which attributes can be associated with which entities. This is made by the application. The design of this DSL assumes a particular application use case that can be evolved later and not a general usage pattern for Datomic.
Note that we used Clojure 1.6.0 for this Recipe because there were problems with Datomic and 1.7 at the time of writing.
create.clj
There is only one function to look at it here. Most of the work is done in these two lines:
(let [fields (apply hash-map attributes)]
(mapcat (fn [[attribute-name attribute-type]]
We take the DSL and stick it in a hash-map, and then we map the key-value pairs with a boilerplate text wrapper, which also adds some Datomic ids.
create_test.clj
First we convert the DSL to Datomic syntax using def book-schema. Then we transact against the database, and fail the test if there are any errors using def trans-result. Then we run a query against the database to see if we can see our schema change using def book-schema-query-result. Then we assert that we got a non-nil response to our query using deftest schema-created-test.
add.clj
In the function convert-dsl-to-keywords-fn, we create a two-stage function. The first call initializes the parameter schema-name and returns a function. The resulting function takes two parameters—a key value for being mapped over a hash-map structure. This function is used for taking simple key values in our DSL and turning them into a Datomic-style syntax.
In the function convert-to-map we take the result of the work in add-datom, convert the structure to a hash-map, and then wrap it in a vector to make it Datomic syntax.
In the function add-datom we map the convert-dsl-to-keywords function (returned from the parent function) over the DSL to convert it to Datomic style syntax. We then append a Datomic id.
add_test.clj
First we convert the DSL to Datomic syntax using def book-add. Then we transact against the database, and fail the test if there are any errors using def add-result. Then we run a query against the database to see if we can see our datom using def book-datom. Then we assert that we have two items in our result using deftest datom-added-test.
create_nested.clj
First we define a custom zipper for maps using the function map-zip.2 Then we convert a DSL map to a zipper using the function dsl-zipper. Then we return a flat sequence of all the zipper nodes in the nested map using the function zip-nodes. Then, given a zipped node, we wrap it using Datomic boilerplate syntax and give it a type reference to its child entities, if any, using the function create-schema-nested-fn. Then we map the create-schema-nested-fn over the zipped nodes to produce the Datomic syntax with child entities, and then filter out all the nil results using the function create-schema-nested.
2. This was first seen here: http://stackoverflow.com/questions/15013458/clojure-zipper-of-nested-maps-repressing-a-trie.
create_nested_test.clj
First we convert the DSL to Datomic syntax using def nested-book-schema. Then we transact against the database, and fail the test if there are any errors using def trans-result. Then we run a query against the database to see if we can see our schema change using def book-schema-query-result. Then we assert that we got a non-nil response to our query using deftest schema-created-test.
add_nested.clj
The first thing we do is retrieve the functions map-zip, dsl-zipper, and zip-nodes from the file create_nested.clj in the namespace declaration.
In the function up-stepper we return a sequence of nodes, given a node, from that node to the top of the nested structure.
In the function zip-node-map-entries we take a node and see if the underlying type is a MapEntry, then we return the first half of it (the key).
In the function steps-up we combine the previous two functions, so that given a node, we can get the keys of all the maps from the node to the top of the nested structure.
In the function map-key-seqs we take a DSL and return a set of step-sequences, each of which can traverse the DSL to a particular node. (This way you can map over this result, and use it to operate on every node by using the particular keys for that node to step through the tree.)
In the function get-key-sequences we get each of the nested nodes as a flat vector. This can be easily mapped over.
In the function append-in we take a nested data structure, a key sequence to step through it, and a value to add in that location. This function combines assoc-in and get-in functions.
In the function replace-children we walk a data structure to add dbids for matching parents and children datoms. This is used by the convert-key-sequence-fn function.
In the function add-matching-dbids-to-nested-child, we take the current DSL and a location sequence. At the location sequence, matching dbid references are added to parent and child datoms. The resulting modified DSL is returned.
In the function key-sequences we take the DSL and get the key sequences that refer to each of the nested nodes.
In the function nested-insert-dsl we start with our DSL and reduce the function add-matching-dbids-to-nested-child over the key sequences. This steps through the key sequences and one by one adds matching dbids to the parents and children. The result is returned as a modified DSL.
In the function convert-key-sequence-fn we take our nested DSL with added dbids and convert it to flattened Datomic syntax. For each key value we add a Datomic boilerplate text. So that this can be mapped using the key sequence, there is a wrapper function that takes an additional parameter compared to the original DSL.
In the function key-sequences-rev we return a version of the key sequences with the order of the keys reversed.
In the function convert-dsl-to-datomic-syntax we map the convert-key-sequence-fn function over the reversed key sequences. This takes the original DSL syntax with added dbids and flattens it into Datomic syntax.
add_nested_test.clj
First we convert the DSL to Datomic syntax using def nested-datomic-add-syntax. Then we transact against the database and fail the test if there are any errors using def add-result. Then we run a query against the database to see if we can see our datom using def borrower-datom. Then we assert that we have two items in our result using deftest datom-added-test.
Conclusion
In this chapter we have done the following:
Generated a basic Datomic schema using a DSL.
Transacted a generated schema against an in-memory instance of Datomic.
Generated a basic add datom statement using a DSL.
Transacted a generated add datom against a Datomic instance.
Generated schema for multiple associated schema with a DSL.
Transacted a generated schema for multiple nested entities against a Datomic instance.
Generated add datom syntax for multiple entities for using a DSL.
Transacted a generated add datom syntax for multiple entities against a Datomic instance.