In Chapter 19 we learned how to include external libraries in our Android projects. Kotlin doesn’t have dedicated XML and JSON processing classes included in its standard library, so to achieve XML- and JSON-related tasks we use appropriate external libraries and add some convenience functions in the form of extension functions.
This chapter assumes you have a sample app you can use to test the code snippets provided. Use whatever app you like or one of the apps we developed in this book. You could, for example, add some sample code providing Log output for testing inside the activity’s onCreate() function, or you could use a test class using one of Android’s test methodologies. Choose a method that best suits your needs.
XML Processing
XML files are at their simplest files of a form similar to this:
<?xml version="1.0" encoding="UTF-8"?>
<ProbeMsg>
<TimeStamp>2016-10-30T19:07:07Z</TimeStamp>
<ProbeId>1A6G</ProbeId>
<ProbeValue ScaleUnit="cm">37.4</ProbeValue>
<Meta>
<Generator>045</Generator>
<Priority>-3</Priority>
<Actor>P. Rosengaard</Actor>
</Meta>
</ProbeMsg>
For XML processing one or a combination of the following paradigms gets used.
DOM Model: Complete tree handling: In the Document Object Model (DOM) the XML data get treated as a whole represented by an in-memory, tree-like structure.
SAX: Event-based processing: Here an XML file gets parsed and with each element or attribute an appropriate event is fired. The events are received by callback functions that have to be registered with the SAX processor. Such a “tell me what you are doing” style of processing is commonly called push parsing.
StAX: Stream-based processing: Here you perform operations like “Give me the next XML element” and the like. In contrast to SAX, where we have a push parsing, for StAX we tell the parser what it has to do: “I tell you what you do.” This therefore is called pull parsing.
On Android you typically handle small to medium-size XML files. For this reason in this chapter we use the DOM. For reading, we first parse the complete XML file and store the data in a DOM tree in memory. Here operations on them like deleting, changing, or adding elements are easy to accomplish and happen in memory; thus they are very fast. For writing we take the complete DOM tree from memory and generate an XML character stream from it, perhaps writing the result back to a file.
For XML handling we add the Java reference implementation
Xerces
as an external library. Inside Android Studio, open the module’s
build.gradle file and inside the
dependencies section add:
implementation 'xerces:xercesImpl:2.12.0'
Reading XML Data
The DOM implementation we can use by virtue of the
Xerces implementation already contains everything needed to read XML elements. We will, however, add a couple of extension functions that greatly improve the DOM API’s usability. For this aim, create a package
com.example.domext, or you can also use any other suitable package name. Add a
Kotlin file dom.kt inside this package, and as its contents write:
package com.example.domext
import org.apache.xerces.parsers.DOMParser
import org.w3c.dom.Document
import org.w3c.dom.Node
import org.xml.sax.InputSource
import java.io.StringReader
import java.io.StringWriter
import javax.xml.transform.OutputKeys
import javax.xml.transform.TransformerFactory
import javax.xml.transform.dom.DOMSource
import javax.xml.transform.stream.StreamResult
fun parseXmlToDOM(s:String) : Document {
val parser: DOMParser = DOMParser()
return parser.let {
it.parse(InputSource(StringReader(s)))
it.document
}
}
fun Node.fetchChildren(withText:Boolean = false) =
(0..(this.childNodes.length - 1)).
map { this.childNodes.item(it) }.
filter { withText || it.nodeType != Node.TEXT_NODE }
fun Node.childCount() = fetchChildren().count()
fun Node.forEach(withText:Boolean = false,
f:(Node) -> Unit) {
fetchChildren(withText).forEach { f(it) }
}
operator fun Node.get(i:Int) = fetchChildren()[i]
operator fun Node.invoke(s:String): Node =
if(s.startsWith("@")) {
this.attributes.getNamedItem(s.substring(1))
}else{
this.childNodes.let { nl ->
val iter = object : Iterator<Node> {
var i: Int = 0
override fun next() = nl.item(i++)
override fun hasNext() = i < nl.length
}
iter.asSequence().find { it.nodeName == s }!!
}
}
operator fun Node.invoke(vararg s:String): Node =
s.fold(this, { acc, s1 -> acc(s1) })
fun Node.text() = this.firstChild.nodeValue
fun Node.name() = this.nodeName
fun Node.type() = this.nodeType
Those are all package-level functions and extension functions for
org.w3c.dom.Node, with the following characteristics:
In the DOM API, each element in the tree (e.g., ProbeValue in the XML data from the beginning of this section) gets represented by a Node instance.
We add a
parseXmlToDOM(s:String)
package-level function that converts an XML string to a Document.
We add a fetchChildren() function
to Node that returns all nontext children of a node that are disregarding text elements. If you add with- Text=true as a parameter, the text nodes of an element get included in the children list, even if they only contain spaces and line breaks. For example, in the XML data from the beginning of this section, the node Meta has three children: Generator, Priority, and Actor. With withText=true the spaces and line breaks between them would be returned as well.
We add a childCount() function
to Node that counts the number of children elements of a node, disregarding text elements. The official DOM API does not provide a function for that.
We add a forEach() function
to Node that allows us to iterate through a node’s children the Kotlin way. The original DOM API does not provide such an iterator, as it only has functions and properties hasChild- Nodes(), childNodes.length, and childNodes.item(index:Int) to iterate through children. If you add withText=true as a parameter, the text nodes of an element are included in the children list, even if they only contain spaces and line breaks.
We add a get(i:Int) function
to Node to get a certain child from an element, disregarding text nodes.
We overload the invoke operator of Node, which belongs to the parentheses (). The first variant with a String parameter navigates to a child by name: node("cn") = node → child with name “cn.” If the parameter starts with a @ the attribute gets addressed: node("@an") = node → attribute with name “an.” In the latter case, you still need to call text() to get the attribute value as a string.
The second variant of the overloaded invoke operator
allows us to specify several strings, which navigates to a child from a child from a child, and so on.
We add functions to Node: first, text() gets the text contents of an element, then name()gives us the node name, and then type()evaluates to the node type (for possible values see the constant properties of the Node class).
This snippet provides examples of how to use the API and the
extensions.
import ...
import com.example.domext.*
...
val xml = """<?xml version="1.0" encoding="UTF-8"?>
<ProbeMsg>
<TimeStamp>2016-10-30T19:07:07Z</TimeStamp>
<ProbeId>1A6G</ProbeId>
<ProbeValue ScaleUnit="cm">37.4</ProbeValue>
<Meta>
<Generator>045</Generator>
<Priority>-3</Priority>
<Actor>P. Rosengaard</Actor>
</Meta>
</ProbeMsg>"""
try {
// Parse the complete XML document
val dom = parseXmlToDOM(xml)
// Access an element
val ts = dom("ProbeMsg")("TimeStamp").text()
Log.d("LOG", ts) // 2001-11-30T09:08:07Z
// Access an attribute
val uni = dom("ProbeMsg")("ProbeValue")("@ScaleUnit")
Log.d("LOG", uni.text()) // cm
// Simplified XML tree navigation
val uni2 = dom("ProbeMsg","ProbeValue","@ScaleUnit")
Log.d("LOG", uni2.text()) // cm
// Iterate through an element's children
dom("ProbeMsg")("Meta").forEach { n ->
Log.d("LOG", n.name() + ": " + n.text())
// Generator: 045
// Priority: -3
// Actor: P. Rosengaard
}
}catch(e:Exception) {
Log.e("LOG", "Cannot parse XML", e)
}
...
Altering XML Data
Once we have a DOM representation of an XML tree in memory, we can add elements. Although we could use the functions already provided by the DOM API,
Kotlin allows us to improve the expressiveness. For this purpose, add the following code to our extension file
dom.kt (I don’t add new imports; press Alt+Enter to let Android Studio help you add necessary imports):
fun prettyFormatXml(document:Document): String {
val format = OutputFormat(document).apply { lineWidth = 65
indenting = true
indent = 2
}
val out = StringWriter()
val serializer = XMLSerializer(out, format)
serializer.serialize(document)
return out.toString()
}
fun prettyFormatXml(unformattedXml: String) =
prettyFormatXml(parseXmlToDOM(unformattedXml))
fun Node.toXmlString():String {
val transformerFact = TransformerFactory.newInstance()
val transformer = transformerFact.newTransformer()
transformer.setOutputProperty(OutputKeys.INDENT, "yes")
val source = DOMSource(this)
val writer = StringWriter()
val result = StreamResult(writer)
transformer.transform(source, result)
return writer.toString()
}
operator fun Node.plusAssign(child:Node) {
this.appendChild(child)
}
fun Node.addText(s:String): Node {
val doc = ownerDocument
val txt = doc.createTextNode(s)
appendChild(txt)
return this
}
fun Node.removeText() {
if(hasChildNodes() && firstChild.nodeType == Node.TEXT_NODE)
removeChild(firstChild)
}
fun Node.updateText(s:String) : Node { removeText()
return addText(s)
}
fun Node.addAttribute(name:String, value:String): Node {
(this as Element).setAttribute(name, value)
return this
}
fun Node.removeAttribute(name:String) {
this.attributes.removeNamedItem(name)
}
Here is a description of what we have in this case
Functions prettyFormatXml( document: Document )
and prettyFormatXml( unformattedXml: String ) are utility functions mainly for diagnostic purposes. They create a pretty string given a Document or an unformatted XML string.
Extension function
Node.toXmlString()
creates a string representation of the XML subtree starting from the current node. If you do this for the Document, the whole DOM structure will be converted.
We overload the plusAssign operator (corresponding to +=) of Node to add a child node.
We add an addText() extension to Node for adding text content to a node.
We add a removeText() extension to Node for removing text content from a node.
We add an updateText() extension to Node for altering the text content of a node.
We add an addAttribute() extension to Node for adding an attribute to a node.
We add a removeAttribute() extension to Node for removing an attribute from a node.
We add an updateAttribute() extension to Node for altering an attribute of a node.
For example, use cases of these functions include the following code snippets. First we add an
element plus attribute to a given node:
val xml = """<?xml version="1.0" encoding="UTF-8"?>
<ProbeMsg>
<TimeStamp>2016-10-30T19:07:07Z</TimeStamp>
<ProbeId>1A6G</ProbeId>
<ProbeValue ScaleUnit="cm">37.4</ProbeValue>
<Meta>
<Generator>045</Generator>
<Priority>-3</Priority>
<Actor>P. Rosengaard</Actor>
</Meta>
</ProbeMsg>"""
try {
val dom = parseXmlToDOM(xml)
val msg = dom("ProbeMsg")
val meta = msg("Meta")
// Add a new element to "meta".
meta += dom.createElement("NewMeta").
addText("NewValue").
addAttribute("SomeAttr", "AttrVal")
Log.d("LOG", "
" + prettyFormatXml(dom))
}catch(e:Exception) { Log.e("LOG", "XML Error", e)
}
For this to work we also use the createElement() function
from the Document class. At the end this code writes the altered XML to the logging console.
The following code samples explain how we can change and remove attributes and
elements:
val xml = """<?xml version="1.0" encoding="UTF-8"?>
<ProbeMsg>
<TimeStamp>2016-10-30T19:07:07Z</TimeStamp>
<ProbeId>1A6G</ProbeId>
<ProbeValue ScaleUnit="cm">37.4</ProbeValue>
<Meta>
<Generator>045</Generator>
<Priority>-3</Priority>
<Actor>P. Rosengaard</Actor>
</Meta>
</ProbeMsg>"""
try {
val dom = parseXmlToDOM(xml)
val msg = dom("ProbeMsg")
val ts = msg("TimeStamp")
val probeValue = msg("ProbeValue")
// Update an attribute and the text contents of
// an element.
probeValue.updateAttribute("ScaleUnit", "dm")
ts.updateText("1970-01-01T00:00:00Z")
Log.d("LOG", "
" + prettyFormatXml(dom))
// Remove an attribute
probeValue.removeAttribute("ScaleUnit")
Log.d("LOG", "
" + prettyFormatXml(dom))
// Removing a node means removing it from
// its parent node.
msg.removeChild(probeValue)
Log.d("LOG", "
" + prettyFormatXml(dom))
}catch(e:Exception) {
Log.e("LOG", "XML Error", e)
}
Creating New DOMs
If you need to write a
DOM representation of an XML document from scratch, first create a
Document instance. This one does not have a public constructor; instead you write:
val doc = DocumentBuilderFactory.
newInstance().newDocumentBuilder().newDocument()
From there you can add elements as described previously. Note that to see any output from our prettyFormatXml()
utility function you must add at least one child element to doc.
Exercise 1
Add a createXmlDocument() function
to the dom.kt file to simplify document creation.
JSON Processing
JavaScript Object Notation (JSON) is the little sister of XML. Data written in the JSON format require less space compared those same data using the XML format. In addition, JSON data almost naturally map to JavaScript objects in a browser environment and JSON therefore has gained considerable attention during recent years.
Kotlin’s standard library doesn’t know how to handle JSON data, so, similar to XML processing, we add a suitable external library. From several possibilities we use the widely adopted
Jackson library. To add it to an Android project, inside the module’s
build.gradle file in the
dependencies section add
implementation
'com.fasterxml.jackson.core:jackson-core:2.9.8'
implementation
'com.fasterxml.jackson.core:jackson-databind:2.9.8'
(on two lines, remove the line breaks).
Several paradigms exist for JSON processing. The most commonly used are a tree-like structure with JSON-specific objects, and a mapping between Kotlin and JSON objects with various semiautomatic conversion mechanisms. We leave the mapping methodology for your further research; it contains a couple of highly involved peculiarities, mostly for JSON collection mapping. The Jackson home page gives you more information about it. We instead describe mechanisms to handle an in-memory tree representation of JSON data.
For the rest of this section we use the following JSON data to explain the functions used inside the examples:
val json = """{
"id":27,
"name":"Roger Rabbit",
"permanent":true,
"address":{
"street":"El Camino Real",
"city":"New York",
"zipcode":95014
},
"phoneNumbers":[9945678, 123456781],
"role":"President"
}"""
JSON Helper Functions
The Jackson library for JSON processing contains all that is needed to write, update, and delete JSON elements. The library is quite extensive and contains an enormous amount of classes and functions. To simplify the development and to include Kotlin goodies we use a couple of package-level functions and extension functions to improve the JSON code readability. These are best located in a Kotlin file json.kt inside some package com.whatever.ext.
We start with the imports, add an
invoke operator
so we can easily fetch a child from a node, and add a
remove and a
forEach function for removing a node and traversing through the children of a node:
import com.fasterxml.jackson.core.JsonFactory
import com.fasterxml.jackson.core.util.DefaultPrettyPrinter
import com.fasterxml.jackson.databind.JsonNode
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.node.*
import java.io.ByteArrayOutputStream
import java.math.BigInteger
operator fun JsonNode.invoke(s:String) = this.get(s)
operator fun JsonNode.invoke(vararg s:String) =
s.fold(this, { acc, s -> acc(s) })
fun JsonNode.remove(name:String) {
val on = (this as? ObjectNode)?:
throw Exception("This is not an object node")
on.remove(name) }
fun JsonNode.forEach(iter: (JsonNode) -> Unit ) {
when(this) {
is ArrayNode -> this.forEach(iter)
is ObjectNode -> this.forEach(iter)
else -> throw Exception("Cannot iterate over " +
this::class)
}
}
Next we add simple alias
text() for
asText()to
streamline text extraction:
fun JsonNode.text() = this.asText()
Another iterator traverses through the children of an object node. This time we take care of the children’s names as well:
fun JsonNode.forEach(iter: (String, JsonNode) -> Unit ) {
if(this !is ObjectNode)
throw Exception(
"Cannot iterate (key,val) over " + this::class)
this.fields().forEach{
(name, value) -> iter(name, value) }
}
To write a child of an object node we define a
put() function
, so we can write
node.put( "childName", 42 ):
// Works only if the node is an ObjectNode!
fun JsonNode.put(name:String, value:Any?) : JsonNode {
if(this !is ObjectNode)
throw Exception("Cannot put() on none-object node")
when(value) {
null -> this.putNull(name)
is Int -> this.put(name, value)
is Long -> this.put(name, value)
is Short -> this.put(name, value)
is Float -> this.put(name, value)
is Double -> this.put(name, value)
is Boolean -> this.put(name, value)
is String -> this.put(name, value)
is JsonNode -> this.put(name, value)
else -> throw Exception(
"Illegal value type: ${value::class}")
}
return this
}
For appending a value to an array object we define an
add() function
, which works for various types:
// Add a value to an array, works only if this is an
// ArrayNode
fun JsonNode.add(value:Any?) : JsonNode {
if(this !is ArrayNode)
throw Exception("Cannot add() on none-array node")
when(value) {
null -> this.addNull()
is Int -> this.add(value)
is Long -> this.add(value)
is Float -> this.add(value)
is Double -> this.add(value)
is Boolean -> this.add(value)
is String -> this.add(value)
is JsonNode -> this.add(value)
else -> throw Exception(
"Illegal value type: ${value::class}")
}
return this
}
For JSON object creation we define various
createSomething() style
functions, and we also add a couple of Kotlin-like builder functions:
// Node creators
fun createJsonTextNode(text:String) = TextNode.valueOf(text)
fun createJsonIntNode(i:Int) = IntNode.valueOf(i)
fun createJsonLongNode(l:Long) = LongNode.valueOf(l)
fun createJsonShortNode(s:Short) = ShortNode.valueOf(s)
fun createJsonFloatNode(f:Float) = FloatNode.valueOf(f)
fun createJsonDoubleNode(d:Double) = DoubleNode.valueOf(d)
fun createJsonBooleanNode(b:Boolean) = BooleanNode.valueOf(b)
fun createJsonBigIntegerNode(b: BigInteger) = BigIntegerNode.valueOf(b)
fun createJsonNullNode() = NullNode.instance
fun jsonObjectNodeOf(
children: Map<String,JsonNode> = HashMap()) :
ObjectNode {
return ObjectNode(JsonNodeFactory.instance, children)
}
fun jsonObjectNodeOf(
vararg children: Pair<String,Any?>) :
ObjectNode {
return children.fold(
ObjectNode(JsonNodeFactory.instance), { acc, v ->
acc.put(v.first, v.second)
acc
})
}
fun jsonArrayNodeOf(elements: Array<JsonNode> =
emptyArray()) : ArrayNode {
return ArrayNode(JsonNodeFactory.instance,
elements.asList())
}
fun jsonArrayNodeOf(elements: List<JsonNode> =
emptyList()) : ArrayNode {
return ArrayNode(JsonNodeFactory.instance,
elements)
}
fun jsonEmptyArrayNode() : ArrayNode {
return ArrayNode(JsonNodeFactory.instance)
}
fun jsonArrayNodeOf(vararg elements: Any?) : ArrayNode {
return elements.fold(
ArrayNode(JsonNodeFactory.instance), { acc, v ->
acc.add(v)
acc
})
}
Extension functions
toPrettyString()
and
toJsonString()
can be used to generate a string representation of any JSON node:
// JSON output as pretty string
fun JsonNode.toPrettyString(
prettyPrinter:PrettyPrinter? =
DefaultPrettyPrinter()) : String {
var res:String? = null
ByteArrayOutputStream().use { os ->
val gen = JsonFactory().createGenerator(os).apply {
if(prettyPrinter != null) this.prettyPrinter = prettyPrinter
}
val mapper = ObjectMapper()
mapper.writeTree(gen, this)
res = String(os.toByteArray())
}
return res!!
}
// JSON output as simple string
fun JsonNode.toJsonString() : String =
toPrettyString(prettyPrinter = null)
The main idea of all these extension functions is to improve conciseness by adding JSON object-related and JSON array-related functions to the base node class JsonNode and perform class casts during runtime. Although it makes the JSON code smaller and more expressive, the risk of getting exceptions during runtime is increased.
Reading and Writing JSON Data
To read in JSON
data, all you have to do is to write:
val json = ... // see section beginning
val mapper = ObjectMapper()
val root = mapper.readTree(json)
From here we can investigate JSON elements, iterate through and fetch JSON object members, and extract JSON array elements:
try {
val json = ... // see section beginning
val mapper = ObjectMapper()
val root = mapper.readTree(json)
// see what we got
Log.d("LOG", root.toPrettyString())
// type of the node
Log.d("LOG", root.nodeType.toString())
// <- OBJECT
// is it a container?
Log.d("LOG", root.isContainerNode.toString())
// <- true
root.forEach { k,v ->
Log.d("LOG",
"Key:${k} -> val:${v} (${v.nodeType})")
Log.d("LOG",
" <- " + v::class.toString())
}
val phones = root("phoneNumbers")
phones.forEach { ph ->
Log.d("LOG", "Phone: " + ph.text())
}
Log.d("LOG", "Phone[0]: " + phones[0].text())
val street = root("address")("street").text()
Log.d("LOG", "Street: " + street)
Log.d("LOG", "Zip: " + root(“address”, “zipcode”).asInt())
}catch(e:Exception) {
Log.e("LOG", "JSON error", e)
}
The following code snippet shows how to alter a JSON
tree by adding, changing, or deleting nodes or JSON object members.
// add it to the "try" statements from the
// last listing
// remove an entry
root("address").remove("zipcode")
Log.d("LOG", root.toPrettyString())
// update an entry
root("address").put("street", "Fake Street 42")
Log.d("LOG", root.toPrettyString())
root("address").put("country", createJsonTextNode("Argentina"))
Log.d("LOG", root.toPrettyString())
// create a new object node
root.put("obj", jsonObjectNodeOf(
"abc1" to 23,
"abc2" to "Hallo",
"someNull" to null
))
Log.d("LOG", root.toPrettyString())
// create a new array node
root.put("arr", jsonArrayNodeOf(
23,
null,
"Hallo"
))
Log.d("LOG", root.toPrettyString())
// write without spaces or line breaks
Log.d("LOG", root.toJsonString())
Creating New JSON Trees
To create a new JSON
tree in memory you use:
val root = jsonObjectNodeOf()
From there you can add JSON elements as described previously.
Exercise 2
Create a JSON document corresponding to:
{
"firstName": "Arthur",
"lastName": "Doyle",
"dateOfBirth": "03/04/1997",
"address": {
"streetAddress": "21 3rd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-1234"
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "mobile",
"number": "123 456-7890"
}
],
"children": [],
"spouse": null
}
