What we've done so far is basically useless. But, from now on, we will start to add features and make it capable of fitting in practical needs, including the capability of synchronizing multiple data items with multiple clients, and merging conflicts.
Ideally, the data we need to synchronize will have a lot of items contained. Directly changing the type of data to an array would work if there were only very limited number of these items.
Now let's change the type of the data property of DataStore and DataSyncingInfo interfaces to string[]. With the help of TypeScript, you will get errors for unmatched types this change would cause. Fix them by annotating the correct types.
But obviously, this is far from an efficient solution.
If the data store contains a lot of data, the ideal approach would be only updating items that are not up-to-date.
For example, we can create a timestamp for every single item and send these timestamps to the server, then let the server decide whether a specific data item is up-to-date. This is a viable approach for certain scenarios, such as checking updates for software extensions. It is okay to occasionally send even hundreds of timestamps with item IDs on a fast network, but we are going to use another approach for different scenarios, or I won't have much to write.
User data synchronization of offline apps on a mobile phone is what we are going to deal with, which means we need to try our best to avoid wasting network resources.
The reason why we thought about sending timestamps of all items is for the server to determine whether certain items need to be updated. However, is it necessary to have the timestamps of all data items stored on the client side?
What if we choose not to store the timestamp of data changing, but of data being synchronized with the server? Then we can get everything up-to-date by only sending the timestamp of the last successful synchronization. The server will then compare this timestamp with the last modified timestamps of all data items and decide how to respond.
As the title of this part suggests, the process is server-centered and relies on the server to generate the timestamps (though it does not have to, and practically should not, be the stamp of the actual time).
Note
If you are getting confused about how these timestamps work, let's try again. The server will store the timestamps of the last time items were synchronized, and the client will store the timestamp of the last successful synchronization with the server. Thus, if no item on the server has a later timestamp than the client, then there's no change to the server data store after that timestamp. But if there are some changes, by comparing the timestamp of the client with the timestamps of server items, we'll know which items are newer.
Now there seems to be quite a lot to change. Firstly, let's handle synchronizing data from server to client.
This is what's expected to happen on the server side:
- Add a timestamp and identity to every data item on the server
- Compare the client timestamp with every data item on the server
- Respond with items newer than what the client has as well as a new timestamp.
And here's what's expected to happen on the client side:
- Synchronize with the last timestamp sent to the server
- Update the local store with new data responded by the server
- Update the local timestamp of the last synchronization if it completes without error
Updating interfaces
First of all, we have now an updated data store on both sides. Starting with the server, the data store now contains an array of data items. So let's define the ServerDataItem interface and update ServerDataStore as well:
export interface ServerDataItem {
id: string;
timestamp: number;
value: string;
}
export interface ServerDataStore {
items: {
[id: string]: ServerDataItem;
};
}
And for the client, we now have different data items and a different store. The response contains only a subset of all data items, so we need IDs and a map with ID as the index to store the data:
export interface ClientDataItem {
id: string;
value: string;
}
export interface ClientDataStore {
timestamp: number;
items: {
[id: string]: ClientDataItem;
};
}
Previously, the client and server were sharing the same DataSyncingInfo, but that's going to change. As we'll deal with server-to-client synchronizing first, we care only about the timestamp in a synchronizing request for now:
export interface SyncingRequest {
timestamp: number;
}
As for the response from the server, it is expected to have an updated timestamp with data items that have changed compared to the request timestamp:
export interface SyncingResponse {
timestamp: number;
changes: {
[id: string]: string;
};
}
I prefixed those interfaces with Server and Client for better differentiation. But it's not necessary if you are not exporting everything from server.ts and client.ts (in index.ts).
Updating the server side
With well-defined data structures, it should be pretty easy to achieve what we expected. To begin with, we have the synchronize method, which accepts a SyncingRequest and returns a SyncingResponse; and we need to have the updated timestamp as well:
synchronize(request: SyncingRequest): SyncingResponse {
let lastTimestamp = request.timestamp;
let now = Date.now();
let serverChanges: ServerChangeMap = Object.create(null);
return {
timestamp: now,
changes: serverChanges
};
}
Now we are going to add items that are newer than the client to serverChanges:
let items = this.store.items;
for (let id of Object.keys(items)) {
let item = items[id];
if (item.timestamp > lastTimestamp) {
serverChanges[id] = item.value;
}
}
Updating the client side
As we've changed the type of items under ClientDataStore to a map, we need to fix the initial value:
store: ClientDataStore = {
timestamp: 0,
items: Object.create(null)
};
Now let's update the synchronize method. Firstly, the client is going to send a request with a timestamp and get a response from the server:
synchronize(): void {
let store = this.store;
let response = this.server.synchronize({
timestamp: store.timestamp
});
}
Then we'll save the newer data items to the store:
let clientItems = store.items;
let serverChanges = response.changes;
for (let id of Object.keys(serverChanges)) {
clientItems[id] = {
id,
value: serverChanges[id]
};
}
Finally, update the timestamp of the last successful synchronization:
clientStore.timestamp = response.timestamp;
For a server-centered synchronizing process, most of the changes are made through clients. Consequently, we need to figure out how to organize these changes before sending them to the server.
One single client only cares about its own copy of data. What difference would this make when comparing to the process of synchronizing data from the server to clients? Well, think about why we need the timestamp of every data item on the server in the first place. We need them because we want to know which items are new compared to a specific client.
Now, for changes on a client: if they ever happen, they need to be synchronized to the server without requiring specific timestamps for comparison.
However, we might have more than one client with changes that need to be synchronized, which means that changes made later in time might actually get synchronized earlier, and thus we'll have to resolve conflicts. To achieve that, we need to add the last modified time back to every data item on the server and the changed items on the client.
I've mentioned that the timestamps stored on the server for finding out what needs to be synchronized to a client do not need to be (and better not be) an actual stamp of a physical time point. For example, it could be the count of synchronizations that happened between all clients and the server.
Updating the client side
To handle this efficiently, we may create a separated map with the IDs of the data items that have changed as keys and the last modified time as the value in ClientDataStore:
export interface ClientDataStore {
timestamp: number;
items: {
[id: string]: ClientDataItem;
};
changed: {
[id: string]: number;
};
}
You may also want to initialize its value as Object.create(null).
Now when we update an item in the client store, we add the last modified time to the changed map as well:
update(id: string, value: string): void {
let store = this.store;
store.items[id] = {
id,
value
};
store.changed[id] = Date.now();
}
A single timestamp in SyncingRequest certainly won't do the job any more; we need to add a place for the changed data, a map with data item ID as the index, and the changed information as the value:
export interface ClientChange {
lastModifiedTime: number;
value: string;
}
export interface SyncingRequest {
timestamp: number;
changes: {
[id: string]: ClientChange;
};
}
Here comes another problem. What if a change made to a client data item is done offline, with the system clock being at the wrong time? Obviously, we need some time calibration mechanisms. However, there's no way to make perfect calibration. We'll make some assumptions so we don't need to start another chapter for time calibration:
- The system clock of a client may be late or early compared to the server. But it ticks at a normal speed and won't jump between times.
- The request sent from a client reaches the server in a relatively short time.
With those assumptions, we can add those building blocks to the client-side synchronize method:
- Add client-side changes to the synchronizing request (of course, before sending it to the server):
let clientItems = store.items; let clientChanges: ClientChangeMap = Object.create(null); let changedTimes = store.changed; for (let id of Object.keys(changedTimes)) { clientChanges[id] = { lastModifiedTime: changedTimes[id], value: clientItems[id].value }; } - Synchronize changes to the server with the current time of the client's clock:
let response = this.server.synchronize({ timestamp: store.timestamp, clientTime: Date.now(), changes: clientChanges }); - Clean the changes after a successful synchronization:
store.changed = Object.create(null);
Updating the server side
If the client is working as expected, it should send synchronizing requests with changes. It's time to enable the server to handling those changes from the client.
There are going to be two steps for the server-side synchronization process:
- Apply the client changes to server data store.
- Prepare the changes that need to be synchronized to the client.
First, we need to add lastModifiedTime to server-side data items, as we mentioned before:
export interface ServerDataItem {
id: string;
timestamp: number;
lastModifiedTime: number;
value: string;
}
And we need to update the synchronize method:
let clientChanges = request.changes;
let now = Date.now();
for (let id of Object.keys(clientChanges)) {
let clientChange = clientChanges[id];
if (
hasOwnProperty.call(items, id) &&
items[id].lastModifiedTime > clientChange.lastModifiedTime
) {
continue;
}
items[id] = {
id,
timestamp: now,
lastModifiedTime,
value: clientChange.value
};
}
We already talked about resolving conflicts by comparing the last modified times. At the same time, we've made assumptions so we can calibrate the last modified times from the client easily by passing the client time to the server while synchronizing.
What we are going to do for calibration is to calculate the offset of the client time compared to the server time. And that's why we made the second assumption: the request needs to easily reach the server in a relatively short time. To calculate the offset, we can simply subtract the client time from the server time:
let clientTimeOffset = now - request.clientTime;
Note
To make the time calibration more accurate, we would want the earliest timestamp after the request hits the server to be recorded as "now". So in practice, you might want to record the timestamp of the request hitting the server before start processing everything. For example, for HTTP request, you may record the timestamp once the TCP connection gets established.
And now, the calibrated time of a client change is the sum of the original time and the offset. We can now decide whether to keep or ignore a change from the client by comparing the calibrated last modified time. It is possible for the calibrated time to be greater than the server time; you can choose either to use the server time as the maximum value or accept a small inaccuracy. Here, we will go the simple way:
let lastModifiedTime = Math.min(
clientChange.lastModifiedTime + clientTimeOffset,
now
);
if (
hasOwnProperty.call(items, id) &&
items[id].lastModifiedTime > lastModifiedTime
) {
continue;
}
To make this actually work, we need to also exclude changes from the server that conflict with client changes in SyncingResponse. To do so, we need to know what the changes are that survive the conflict resolving process. A simple way is to exclude items with timestamp that equals now:
for (let id of Object.keys(items)) {
let item = items[id];
if (
item.timestamp > lastTimestamp &&
item.timestamp !== now
) {
serverChanges[id] = item.value;
}
}
So now we have implemented a complete synchronization logic with the ability to handle simple conflicts in practice.
At this point, we've hard coded the data with the string type. But usually we will need to store varieties of data, such as numbers, booleans, objects, and so on.
If we were writing JavaScript, we would not actually need to change anything, as the implementation does not have anything to do with certain data types. In TypeScript, we don't need to do much either: just change the type of every related value to any. But that means you are losing type safety, which would definitely be okay if you are happy with that.
But taking my own preferences, I would like every variable, parameter, and property to be typed if it's possible. So we may still have a data item with value of type any:
export interface ClientDataItem {
id: string;
value: any;
}
We can also have derived interfaces for specific data types:
export interface ClientStringDataItem extends ClientDataItem {
value: string;
}
export interface ClientNumberDataItem extends ClientDataItem {
value: number;
}
But this does not seem to be good enough. Fortunately, TypeScript provides generics, so we can rewrite the preceding code as follows:
export interface ClientDataItem<T> {
id: string;
value: T;
}
Assuming we have a store that accepts multiple types of data items - for example, number and string - we can declare it as follows with the help of the union type:
export interface ClientDataStore {
items: {
[id: string]: ClientDataItem<number | string>;
};
}
If you remember that we are doing something for offline mobile apps, you might be questioning the long property names in changes such as lastModifiedTime. This is a fair question, and an easy fix is to use tuple types, maybe along with enums:
const enum ClientChangeIndex {
lastModifiedType,
value
}
type ClientChange<T> = [number, T];
let change: ClientChange<string> = [0, 'foo'];
let value = change[ClientChangeIndex.value];
You can apply less or more of the typing things we are talking about depending on your preferences. If you are not familiar with them yet, you can read more here: http://www.typescriptlang.org/handbook.
Making the typing system happy with multiple data types is easy. But in the real world, we don't resolve conflicts of all data types by simply comparing the last modified times. An example is counting the daily active time of a user cross devices.
It's quite clear that we need to have every piece of active time in a day on multiple devices summed up. And this is how we are going to achieve that:
- Accumulate active durations between synchronizations on the client.
- Add a UID (unique identifier) to every piece of time before synchronizing with the server.
- Increase the server-side value if the UID does not exist yet, and then add the UID to that data item.
But before we actually get our hands on those steps, we need a way to distinguish incremental data items from normal ones, for example, by adding a type property.
As our synchronizing strategy is server-centered, related information is only required for synchronizing requests and conflict merging. Synchronizing responses does not need to include the details of changes, but just merged values.
First of all, we need the client to support incremental changes. And if you've thought about this, you might already be confused about where to put the extra information, such as UIDs.
This is because we were mixing up the concept change (noun) with value. It was not a problem before because, besides the last modified time, the value is what a change is about. We used a simple map to store the last modified times and kept the store clean from redundancy, which balanced well under that scenario.
But now we need to distinguish between these two concepts:
- Value: a value describes what a data item is in a static way
- Change: a change describes the information that may transform the value of a data item from one to another
We need to have a general type of changes as well as a new data structure for incremental changes with a numeric value:
type DataType = 'value' | 'increment';
interface ClientChange {
type: DataType;
}
interface ClientValueChange<T> extends ClientChange {
type: 'value';
lastModifiedTime: number;
value: T;
}
interface ClientIncrementChange extends ClientChange {
type: 'increment';
uid: string;
increment: number;
}
Similar changes to the data store structure should be made. And when we update an item on the client side, we need to apply the correct operations based on different data types:
update(id: string, type: 'increment', increment: number): void;
update<T>(id: string, type: 'value', value: T): void;
update<T>(id: string, type: DataType, value: T): void;
update<T>(id: string, type: DataType, value: T): void {
let store = this.store;
let items = store.items;
let storedChanges = store.changes;
if (type === 'value') {
// ...
} else if (type === 'increment') {
// ...
} else {
throw new TypeError('Invalid data type');
}
}
Use the following code for normal changes (while type equals 'value'):
let change: ClientValueChange<T> = {
type: 'value',
lastModifiedTime: Date.now(),
value
};
storedChanges[id] = change;
if (hasOwnProperty.call(items, id)) {
items[id].value = value;
} else {
items[id] = {
id,
type,
value
};
}
For incremental changes, it takes a few more lines:
let storedChange = storedChanges[id] as ClientIncrementChange;
if (storedChange) {
storedChange.increment += <any>value as number;
} else {
storedChange = {
type: 'increment',
uid: Date.now().toString(),
increment: <any>value as number
};
storedChanges[id] = storedChange;
}
Note
It's my personal preference to use <T> for any casting and as T for non-any castings. Though it has been used in languages like C#, the as operator in TypeScript was originally introduced for compatibilities with XML tags in JSX. You can also write <number><any>value or value as any as number here if you like.
Don't forget to update the stored value. Just change = to += comparing to updating normal data items:
if (hasOwnProperty.call(items, id)) {
items[id].value += value;
} else {
items[id] = {
id,
type,
value
};
}
That's not hard at all. But hey, we see branches.
We are writing branches all the time, but what are the differences between branches such as if (type === 'foo') { ... } and branches such as if (item.timestamp > lastTimestamp) { ... }? Let's keep this question in mind and move on.
With necessary information added by the update method, we can now update the synchronize method of the client. But there is a flaw in practical scenarios: a synchronizing request is sent to the server successfully, but the client failed to receive the response from the server. In this situation, when update is called after a failed synchronization, the increment is added to the might-be-synchronized change (identified by its UID), which will be ignored by the server in future synchronizations. To fix this, we'll need to add a mark to all incremental changes that have started a synchronizing process, and avoid accumulating these changes. Thus, we need to create another change for the same data item.
This is actually a nice hint: as a change is about information that transforms a value from one to another, several changes pending synchronization might eventually be applied to one single data item:
interface ClientChangeList<T extends ClientChange> {
type: DataType;
changes: T[];
}
interface SyncingRequest {
timestamp: number;
changeLists: {
[id: string]: ClientChangeList<ClientChange>;
};
}
interface ClientIncrementChange extends ClientChange {
type: 'increment';
synced: boolean;
uid: string;
increment: number;
}
Now when we are trying to update an incremental data item, we need to get its last change from the change list (if any) and see whether it has ever been synchronized. If it has ever been involved in a synchronization, we create a new change instance. Otherwise, we'll just accumulate the increment property value of the last change on the client side:
let changeList = storedChangeLists[id];
let changes = changeList.changes;
let lastChange =
changes[changes.length - 1] as ClientIncrementChange;
if (lastChange.synced) {
changes.push({
synced: false,
uid: Date.now().toString(),
increment: <any>value as number
} as ClientIncrementChange);
} else {
lastChange.increment += <any>value as number;
}
Or, if the change list does not exist yet, we'll need to set it up:
let changeList = {
type: 'increment',
changes: [
{
synced: false,
uid: Date.now().toString(),
increment: <any>value as number
} as ClientIncrementChange
]
};
store.changeLists[id] = changeList;
We also need to update synchronize method to mark an incremental change as synced before starting the synchronization with the server. But the implementation is for you to do on your own.
Before we add the logic for handling incremental changes, we need to make server-side code adapt to the new data structure:
for (let id of Object.keys(clientChangeLists)) {
let clientChangeList = clientChangeLists[id];
let type = clientChangeList.type;
let clientChanges = clientChangeList.changes;
if (type === 'value') {
// ...
} else if (type === 'increment') {
// ...
} else {
throw new TypeError('Invalid data type');
}
}
The change list of a normal data item will always contain one and only one change. Thus we can easily migrate what we've written:
let clientChange = changes[0] as ClientValueChange<any>;
Now for incremental changes, we need to cumulatively apply possibly multiple changes in a single change list to a data item:
let item = items[id];
for (
let clientChange
of clientChanges as ClientIncrementChange[]
) {
let {
uid,
increment
} = clientChange;
if (item.uids.indexOf(uid) < 0) {
item.value += increment;
item.uids.push(uid);
}
}
But remember to take care of the timestamp or cases in which no item with a specified ID exists:
let item: ServerDataItem<any>;
if (hasOwnProperty.call(items, id)) {
item = items[id];
item.timestamp = now;
} else {
item = items[id] = {
id,
type,
timestamp: now,
uids: [],
value: 0
};
}
Without knowing the current value of an incremental data item on the client, we cannot assure that the value is up to date. Previously, we decided whether to respond with a new value by comparing the timestamp with the timestamp of the current synchronization, but that does not work anymore for incremental changes.
A simple way to make this work is by deleting keys from clientChangeLists that still need to be synchronized to the client. And when preparing responses, it can skip IDs that are still in clientChangeLists:
if (
item.timestamp > lastTimestamp &&
!hasOwnProperty.call(clientChangeLists, id)
) {
serverChanges[id] = item.value;
}
Remember to add delete clientChangeLists[id]; for normal data items that did not survive conflicts resolving as well.
Now we have implemented a synchronizing logic that can do quite a lot jobs for offline applications. Earlier, I raised a question about increasing branches that do not look good. But if you know your features are going to end there, or at least with limited changes, it's not a bad implementation, although we'll soon cross the balance point, as meeting 80% of the needs won't make us happy enough.
Though we have met the needs of 80%, there is still a big chance that we might want some extra features. For example, we want the ratio of the days marked as available by the user in the current month, and the user should be able to add or remove days from the list. We can achieve that in different ways, and we'll choose a simple way, as usual.
We are going to support synchronizing a set with operations such as add and remove, and calculate the ratio on the client.
To describe set changes, we need a new ClientChange type. When we are adding or removing an element from a set, we only care about the last operation to the same element. This means that the following:
- If multiple operations are made to the same element, we only need to keep the last one.
- A
timeproperty is required for resolving conflicts.
So here are the new types:
enum SetOperation {
add,
remove
}
interface ClientSetChange extends ClientChange {
element: number;
time: number;
operation: SetOperation;
}
The set data stored on the server side is going to be a little different. We'll have a map with the element (in the form of a string) as key, and a structure with operation and time properties as the values:
interface ServerSetElementOperationInfo {
operation: SetOperation;
time: number;
}
Now we have enough information to resolve conflicts from multiple clients. And we can generate the set by keys with a little help from the last operations done to the elements.
And now, the client-side update method gets a new part-time job: saving set changes just like value and incremental changes. We need to update the method signature for this new job (do not forget to add 'set' to DataType):
update( id: string, type: 'set', element: number, operation: SetOperation ): void; update<T>( id: string, type: DataType, value: T, operation?: SetOperation ): void;
We also need to add another else if:
else if (type === 'set') {
let element = <any>value as number;
if (hasOwnProperty.call(storedChangeLists, id)) {
// ...
} else {
// ...
}
}
If there are already operations made to this set, we need to find and remove that last operation to the target element (if any). Then append a new change with the latest operation:
let changeList = storedChangeLists[id];
let changes = changeList.changes as ClientSetChange[];
for (let i = 0; i < changes.length; i++) {
let change = changes[i];
if (change.element === element) {
changes.splice(i, 1);
break;
}
}
changes.push({
element,
time: Date.now(),
operation
});
If no change has been made since last successful synchronization, we'll need to create a new change list for the latest operation:
let changeList: ClientChangeList<ClientSetChange> = {
type: 'set',
changes: [
{
element,
time: Date.now(),
operation
}
]
};
storedChangeLists[id] = changeList;
And again, do not forget to update the stored value. This is a little bit more than just assigning or accumulating the value, but it should still be quite easy to implement.
Just like we've done with the client, we need to add a corresponding else if branch to merge changes of type 'set'. We are also deleting the ID from clientChangeLists regardless of whether there are newer changes for a simpler implementation:
else if (type === 'set') {
let item: ServerDataItem<{
[element: string]: ServerSetElementOperationInfo;
}>;
delete clientChangeLists[id];
}
The conflict resolving logic is quite similar to what we do to the conflicts of normal values. We just need to make comparisons to each element, and only keep the last operation.
And when preparing the response that will be synchronized to the client, we can generate the set by putting together elements with add as their last operations:
if (item.type === 'set') {
let operationInfos: {
[element: string]: ServerSetElementOperationInfo;
} = item.value;
serverChanges[id] = Object
.keys(operationInfos)
.filter(element =>
operationInfos[element].operation ===
SetOperation.add
)
.map(element => Number(element));
} else {
serverChanges[id] = item.value;
}
Finally, we have a working mess (if it actually works). Cheers!
When we started to add features, things were actually fine, if you are not obsessive about pursuing the feeling of design. Then we sensed the code being a little awkward as we saw more and more nested branches.
So now it's time to answer the question, what are the differences between the two kinds of branch we wrote? My understanding of why I am feeling awkward about the if (type === 'foo') { ... } branch is that it's not strongly related to the context. Comparing timestamps, on the other hand, is a more natural part of a certain synchronizing process.
Again, I am not saying this is bad. But this gives us a hint about where we might start our surgery from when we start to lose control (due to our limited brain capacity, it's just a matter of complexity).
Most of the code in this chapter is to handle the process of synchronizing data between a client and a server. To get adapted to new features, we just kept adding new things into methods, such as update and synchronize.
You might have already found that most outlines of the logic can be, and should be, shared across multiple data types. But we didn't do that.
If we look into what's written, the duplication is actually minor judging from the aspect of code texts. Taking the update method of the client, for example, the logic of every branch seems to differ. If finding abstractions has not become your built-in reaction, you might just stop there. Or if you are not a fan of long functions, you might refactor the code by splitting it into small ones of the same class. That could make things a little better, but far from enough.
In the implementation, we were playing heavily and directly with ideal in-memory stores. It would be nice if we could have a wrapper for it, and make the real store interchangeable.
This might not be the case for this implementation as it is based on extremely ideal and simplified assumptions and requirements. But adding a wrapper could be a way to provide useful helpers.