Code never lies, comments sometimes do.
—Ron Jeffries
This chapter is a little bit different than the others. While all the other chapters dealt with practical examples and approaches, this chapter is more theoretical.
Even though this chapter contains many code examples, its primary goal is to reveal how Core Data stores work underneath.
What it means “creating a custom store”
What are the possible use cases of creating your own store
Differences between atomic and incremental stores
How to register a new custom store
Building an atomic store
Building an incremental store
Custom Store: What Exactly Does It Mean?
Okay, so a quick reminder of what a Core Data stack looks like.
We have three main components – the data model, a data store, and the context.
A diagram of the Core data stack. Inside the box, Context, Store, and Data model are placed vertically and interlinked.
A reminder of the Core Data stack
Drilling down to the “Store” component , we know that we have four types of such a store – three atomic (XML, binary, and in-memory) and one incremental (SQLite).
Now, it’s time to reveal that we have the option to create a custom store by subclassing either NSIncrementalStore or NSAtomicStore .
These two classes let you write your own store type and basically control how Core Data saves, fetches, and optimizes local data.
For example, you can write a store that can work with CSV files instead of SQLite or even a store that syncs independently with a web service. In fact, almost anything is possible once you have control.
Why Do We Need That?
In most cases, you don’t. The four store types that come with Core Data are suitable for most of your needs.
But there are a couple of reasons you need to know how to create your own data store.
One thing to understand is that using your own custom data store doesn’t mean additional modifications for the rest of your stack. The data model and the context don’t know they are working with a custom data store, and the rest of your codebase stays the same.
It means that your store is easily replaceable, so there is no need to worry!
Rely on Your Technology
SQLite is a very efficient way of storing data, and it is no coincidence that it was chosen as the go-to DB in many mobile applications.
But it doesn’t mean that it’s the only way of managing data on a mobile device.
For example, there are great solutions for implementing a NoSQL datastore on iOS, and writing a custom store is a way of connecting your Core Data to a different type of store.
Cross-Platform Support
If you work with Android developers in the same team, there are cases where there is a requirement for both platforms to work on the same data file format.
In many situations, aligning all platforms to work on the same formats and technology is essential to team success.
Lucky us, by using a custom store type, we can work on the same format and still use the excellent Core Data features.
Migrating from an Old Persistent Store
If you are working on an existing app that doesn’t have Core Data implemented yet, but the app already has a data store, creating a custom store may be a good starting point.
Migrating from an old persistent layer can always be cumbersome, so this method can ease the migration pain.
Connect the Persistent Store Directly to a Web Service
Because we have complete control of how the store behaves, an excellent idea would be to bind it directly to your backend service.
That may seem like a weird idea, especially when you think of the “separation of concerns” principle.
But there are ways you can connect your store directly to a web service without making it too coupled, like dependency injection.
This is an exciting approach we can explore later.
To Explore Another Stack Component
Learning how to create a custom store can reveal some Core Data secrets of how it works underneath.
It basically means that you take responsibility for fetching, saving, and faulting.
It’s a process that gives another overview of Core Data from a different angle and can only make you a better iOS developer.
Perhaps, that’s the best reason to try and learn it.
NSIncrementalStore vs. NSAtomicStore
As mentioned, Core Data stores are divided into two types – atomic and incremental.
How to choose which type of store to create?
It depends on your needs – each store type has its drawbacks and advantages.
Atomic stores favor simplicity over performance. In atomic stores , we load all the data to memory, and each time we need to perform changes (insert/update/delete), we have to save all the data.
Therefore, it’s a straightforward store to manage, but it also consumes maximum memory.
Atomic stores are relevant when we need to base our data on JSON, XML, or CSV files.
Also, notice that these files cannot be too big – in that case, consider using the incremental store.
Compared to atomic stores , incremental stores are more complicated to implement. They require us to handle all the faulting and optimizations ourselves and are meant for those who favor performance over simplicity.
Incremental stores are used where we have big data files or other special async data fetching, such as HTTP requests .
Both store types allow us to create basically anything we want as a persistent store. We just need to understand how they work.
How Do They Work?
Before we move on, I want to explain the basic way stores work in Core Data, and it’s even simpler than you think.
First, let’s talk about Core Data’s responsibility and what is yours.
Initializing our store
Handling the fetching, including predicates
Managing the different contexts for us
Create objects based on your data.
Generate object IDs.
Define the store metadata information.
Provide additional data information when requested.
In other words, what we need to do as developers is handle the connection between the Core Data framework and our backing store, whatever it might be.
Before we start setting up an atomic store, let’s look at its superclass – NSPersistentStore.
NSPersistentStore
NSPersistentStore is the base class for all Core Data persistent stores. NSPersistentStore is also the superclass of NSIncrementalStore and NSAtomicStore , and if we want to create our own store, we need to subclass one of these classes.
As I said earlier, we don’t initialize the stores ourselves – that’s the Core Data framework’s job.
Our job is to register them and tell Core Data what type of store we want them to load.
Every store has metadata that contains both a type and UUID.
The metadata information helps Core Data initialize the new store, manage its migration, share it between extensions, and, in general, take care of our store overtime.
We started with declaring a type and a UUID based on these values, and we also created a storeType and a store description. Both will be used when we set up our Core Data container.
Notice that the overridden methods and variables are mandatory for creating a custom store. Don’t worry – we have plenty of methods to override for the store to work, and that’s a good start.
Now that we have our metadata, we can register our store to Core Data.
Registering the New Store
To ensure Core Data loads our store, we need to perform two steps: register and then load it.
If everything goes well, our container will be loaded without any issues.
NSAtomicStore
Now that our store is loaded, let’s dive into the store itself.
Atomic stores are different from incremental stores because they hold all the data in memory.
That makes them simple and fast, but not that efficient in terms of memory, especially when dealing with big stores.
It’s no surprise that the primary task will be loading all the data to memory.
Mapping the Data
Notice we are not mapping the objects themselves but instead their object IDs. Do you remember why?
The reason is that a managed object is not something that the store holds or creates – this is part of the managed object context’s job.
That’s why it’s so enriched to learn how to create a data store of your own – you finally get to know how things are working behind the scenes, and we are only getting started.
Loading All the Data
As I said earlier (and probably even a few times), atomic stores hold all their data from the start.
The load() method is part of the NSAtomicStore class , and you must override it when creating your own atomic store .
If our store works with some sort of a CSV file, that’s the time for us to go to the file and load all its records.
Load the CSV (or any other persistent file or data) file into the memory.
Loop all its rows.
- For every row
Generate a new referenceID.
Generate a new objectID based on the referenceID.
Create a new cache node (that’s our “object”).
Fill the cache node with data.
Map the objectID to its referenceID.
Add the cache node to the store.
It looks like a lot of work, ha? Well, it’s not that much considering this is most of the job you are going to do when creating your own store.
Please go line by line and try to understand what is happening according to the step I listed before.
More Insights
Don’t worry. I won’t leave you in the dark with the code I just showed you.
Here are a couple of things you should know.
For every row in the CSV, our goal is to create something called a cache node. A cache node represents a record in our store, which is where we hold our data once we load it from the local file.
When we insert a new record to our store, we are basically inserting a new cache node.
To create a cache node, we need to provide an objectID. Up until now, we only read objectID values but never generated one.
This is the place where we fill our store with information from the CSV. A good tip would be to get the names of the key attributes out from the entity description instead of making them hard-coded.
It may look like an over-engineering task, but it pays off in the long run.
What About Relationships?
Core Data isn’t just a persistent store but also an object graph. It means that our new store needs to manage relationships between different objects.
Don’t forget that the album node is just a standard cache node – we need to generate an objectID (based on the entity description), map it, and insert it to the store, exactly as we did with the song node.
Adding a New Object
Our store probably needs to support adding new objects.
After we know how to load all the data from our CSV file and convert it to nodes, adding new information should be quite easy, but it requires us to perform additional work.
When Core Data inserts and manipulates objects, our store doesn’t do anything – remember, the context is the application sandbox. Our store enters the picture only when the context save() method is called.
In this case, we need to do three things: generate a new reference ID, create a new cache node, and save the data to the CSV file.
Generate a New Reference ID
It looks like we have already been there, no?
I told you things will be more and more familiar from now on.
When the Core Data context (now it is your client) asks to insert a new object, the first thing we need to do is to return a reference object.
The reference object must be unique, and it should be derived from the object values.
If not, we need to keep mapping it.
Add a New Cache Node
Adding a new cache node is similar to what we did earlier when we loaded data from the file, but in the opposite direction – from the context down to the store.
Now, even though the implementation looks obvious, there is a small catch: relationships.
If we insert a new object with relationships, we need to make sure that when we create a new cache node, we take care of all the wiring and linking.
One of the things that can help us is the mapping that we did between the reference objects and the objectIDs.
NSAtomicStore has a function called cacheNode(for objectID: NSManagedObjectID) that can help us get a cache node by objectID, and we need to use it to connect everything we need.
Saving
The final step would be to update our local file with all the new changes.
But what is the right way to do that?
There are several ways to implement that, and it depends on how you choose to track your data.
One way would be to take the brute-force approach. We have a map with all the objectIDs, so we can generate cache nodes for all of them and, from the cache nodes, create the data that we can save for our CSV file.
That may be the most reliable and simple way of saving our data back to the file, but it’s not the most elegant approach.
Another way would be to keep the data created when we loaded the CSV file and update it with the changes every time we update or insert a new cache node.
Once we need to save the data back to the file, we have it already updated with all the changes.
Remember that you don’t need to worry about handling multiple contexts – this is the Core Data container’s job. Saving to the store and inserting new cache nodes only happens when the app requests to save the data locally.
Updating and Deleting
Just like adding a new cache node, updating and deleting require implementing additional methods.
Notice that NSAtomicStore makes life easier for you – it already gives you the node and the managed object.
Now that is a good time to reuse the code from the newCacheNode function and consolidate it in one place.
Removing cache nodes is done similarly – Core Data calls the willRemoveCacheNodes method just before the cache nodes are removed from the store.
This is where we need to remove the corresponding data and relationships.
NSIncrementalStore
Unlike NSAtomicStore , NSIncrementalStore aims to solve other situations where you can’t hold the store in memory and need to load the data only when required.
There are two primary use cases for that: when the data store is too big to hold in memory and when the data is not saved on the device and you can fetch it upon request only, like a web service .
But first, let’s talk about how we build an NSIncrementalStore of our own.
I think that the most important thing to understand is the fact that we have much more control and therefore much more responsibility.
Implementing NSAtomicStore was simple. All we had to do was fetch all our records, connect them to a reference ID, and fill cache nodes.
We need to handle all the store operations ourselves, such as fetching, saving, and deleting.
We need to manage faulting . Remember, it’s incremental, and we load only what we need.
Let’s get to work.
Loading the Store
Incremental stores usually work with a local file (in many cases, it’s a SQLite file) and need to make sure the file is in the correct location (and if not, create it).
Check if the file is corrupted and throw an error.
Load basic data from the store (optional).
Let’s elaborate on what “loading basic data” means.
Indeed, incremental stores are not like atomic stores – we don’t load all the data into memory.
But it doesn’t mean we can’t load any data at all.
For example, if we have an enormous store of songs, we can load their IDs into memory, making it easier for us to fetch additional data when needed.
Fetching something like “headers” can ease our implementation, and as a bonus, it’s a great way to make sure our file is not corrupted and in the correct format.
Execute Store Requests
To make our incremental store work properly, the first thing we need to handle is store requests from our context – remember, the context is our “client” now that we are a store.
The execute() function has two parameters: the store request itself (we’ll talk about it in a minute) and the relevant context.
This function handles fetch requests, saving, and even batch actions.
NSPersistentStoreRequest
The NSPersistentStoreRequest instance in the execute() method signature encapsulates all the necessary information to perform what the context requests us to do.
Fetching
When the requestType property equals fetchRequestType, we know the context is trying to perform a fetch request.
Let’s talk about the preceding code.
First, we have a particular function called getSongsIds(byRequest:). We need to write this function, and it needs to retrieve song IDs from our backing store according to the fetch request it receives.
The fetch request contains all the information we need, including a predicate and sort descriptors.
This is the most complicated work you need to do here – how to use the predicate and sort descriptor you’ve got to perform a request to your CSV or SQLite file?
My tip here is that you don’t need to cover all the possible use cases of predicates and sorting.
While you are building your app and adding more and more fetch requests, analyze the fetch request according to your needs and add the relevant code as you go. Don’t try to replicate SQLite-based incremental stores entirely – this is not the goal of building an incremental store.
Once we have all the song IDs, we can perform a for loop and create a managed object for each one of them.
Notice we don’t just initialize a new managed object in every loop iteration – we first generate an objectID derived from the entity description and the songID. Only then do we retrieve a managed object from the context.
If the context already has a managed object with this objectID, it will return the existing one.
The songID is the reference object and unique identifier for our objects. Attaching it to the objectID is our store’s way to register our records from the backing store.
Another thing you might have noticed is that we don’t fill our songs with data at all – that’s because we initialize only faulted objects. Remember that Core Data feature?
Faulting lets us optimize our requests and load additional data only on requests. We’ll cover faulting soon!
Saving
Going back to the start of the execute() function, as I mentioned, this function handles not just fetching but also saving.
But what exactly is a “saving” store request?
Well, “saving” means inserting, updating, and deleting objects.
To handle saving requests, we first need to check the request type and compare it to saveRequestType.
The next step would be casting the fetch request to NSSaveChangesRequest.
Yes, the execute() function uses polymorphism here – it sometimes can be a fetch request and sometimes a save request.
Like the fetch request, the save request also encapsulates all the information needed to save the new data to your store.
insertedObjects
updatedObjects
deletedObjects
Each of them is an array containing the list of objects being requested to save.
What are these objects? Well, in this case, the objects are actually NSManagedObject. If you cast them to Song (for example), you will be able to retrieve all the information needed for saving it in your backing store.
ObjectID
Now, take a moment, read the preceding code, and think if something is missing.
Hint: Look at the code we wrote earlier when we loaded all the data from the backing store and registered it to the incremental store.
Are you there yet?
We need to map the objectID to its reference value (“ songID ” in this case). Remember that?
The reason it doesn’t happen in the execute() function is that, as you probably know, new objects added to Core Data have a temporary objectID. They get their permanent objectID only after the saving operation is completed.
The obtainPermanentIDs function passes a list of new objects and requires returning a list of corresponding object IDs.
I think that the implementation should already be familiar to you by now.
obtainPermanentIDs is another excellent example of how learning about NSIncrementalStore reveals how Core Data works under the hood. Suddenly, many things become more evident.
But there is one more piece missing to the puzzle, and that’s faulting .
Faulting
Remember when we fetched objects but didn’t fetch their data and I told you we’d talk about faulting soon? These objects were like “ghosts,” empty objects.
When the “user” (the user is actually the app itself) calls song.name, we need to go and fetch the name if required.
Fortunately, NSIncrementalStore helps us with managing that area very well.
We only need to implement one more function, and that’s newValuesForObject.
This function gets called when the store needs to fulfill an object with information from the store.
The code is straightforward except for one thing – the version (I marked that in bold).
When we create the NSIncrementalStoreNode , we need to provide a version that should be incremented every time the node is created.
The “version“ helps merge conflicts and should be saved persistently – a specific version for a row. In the preceding example, I read the version from the disk and saved it back – to maintain it for future fetches.
NSIncrementalStore keeps track of faulted objects for us – and this is basically the hard work that needs to be done.
What about relationships?
Even though it looks scary, relationship faulting is superficial. All we need to do is to analyze what the destination entity is and return its NSManagedObjectID (or IDs in case of a to-many relationship).
I handled two relationships in the preceding example – one for Album (to-one) and one for Composer (to-many).
And if you ask yourself where the object data is, you should know the answer by now. If Core Data needs it, it will ask the data by calling the newValuesForObject() function. Your job is just to implement it.
Web Services
Here’s a section that I mentioned early, and it probably still sounds weird to you. Because the incremental store is incremental, a possible use case for implementing such a store is to connect the store directly to your backend API.
Think of it for a second – we have a perfectly excellent object graph framework with predicates, sorting, and caching mechanisms.
You are using that mechanism for your app anyway. Why do you “care” if the data is received from your local or remote storage?
That’s part of the beauty of Core Data. As I said more than once, Core Data is not a SQLite wrapper. It’s much more than that.
This is where you manage your entities, and the option of binding it directly to your server may be a great approach.
The primary problem with connecting our store to a server is that calling an HTTP request is a time-consuming operation and may take a few seconds to return.
All the steps inside your store must be synchronized – Core Data stores don’t support async operations.
Executing a Core Data request from the app should conclude that fact and perform on a background thread, considering all the Core Data background operation constraints that we are already aware of.
Summary
Implementing incremental and atomic stores is one of the fascinating subjects to learn when diving into Core Data, not because of what you can do with it but rather what you can learn from it.
It’s an excellent opportunity to switch your position in iOS development and think as a framework maker for a second.
We’ve learned how to create atomic and incremental stores and how they can serve us by connecting our store directly to our back end.
Now is the time to see how Core Data can connect to more iOS features.