Chapter 11: Local Storage and Data Synchronization

When we talk about the concept of "mobile," we must always consider particular aspects of the devices used – things such as processing power, passive safety, and a very common case: offline operation.

We often find ourselves without network connectivity on our devices, either because we don't have a Wi-Fi network within our reach or because the network signal does not allow us to take advantage of data packets.

For that, there is an interesting concept in mobile development, Local Storage since we can have the necessary data for our applications to run on our device without needing an internet connection to obtain it. Of course, in most cases synchronization will be necessary later to keep the data up to date, both on our device and on the database server.

In this chapter, we will learn how Local Storage works, how to use it, and how to synchronize our data from Local Storage with existing data on the server side (in cases where there is such a need).

The chapter covers the following topics:

• Creating and using Local Storage entities
• Fetching data from Local Storage
• Analyzing data synchronization patterns

By the end of the chapter, you will know how to create local entities, fetch and manipulate data from them, and synchronize the data existing in those entities with entities existing on the server side.

Let's get to it!

Info

For now, Local Storage is a feature that only exists in mobile, but it is expected that something similar, even with a different technology, will emerge for reactive web applications.

Creating and using Local Storage entities

It may seem like we are facing a completely new concept, but we're not. The implementation of local entities in Service Studio is extremely similar to the implementation of entities that we have already seen (existing in a database server).

To create these entities, click on the Data tab, open the Entities folder, and go to the Local Storage section:

Figure. 11.1 – Local Storage section in Service Studio

Info

The Local Storage section is only available in the Phone App and Tablet App modules.

We can create our entities manually, or we can create them directly based on an existing entity on the server side (we'll look at this in more detail later):

Figure 11.2 – Local Storage options

These entities, when created, have the following properties:

• Attributes – Attributes are the fields where we store local data. They can be of several simple types. We can define some properties, varying according to the configured type, and one of the transversal properties is mandatory configuration.
• Primary key– The primary key is the entity identifier. This is created automatically at entity creation time and cannot be a composite key. It is a mandatory field.
• Foreign Keys – Foreign keys are attributes that define the relationships between entities. A foreign key in an entity refers to a primary key of the table to be related.

A difference we found regarding database entities is the fact that there is no possibility to create indexes. However, everything else is quite similar, as we can see in the local entity's properties window:

Figure 11.3 – Local Storage properties window

Also, in order to facilitate some synchronization techniques, there is one more CRUD operation, DeleteAll:

Figure 11.4 – Local entity representation with the DeleteAll operation

If we want to create a replica of an entity from the database as a local entity, we click the right mouse button on Local Storage, select the Add entity from database option, and, in the popup that opens, we choose the entity of the database that we want to replicate:

Figure 11.5 – Database replica selection popup

Then we can select the fields we want to replicate. We must always bear in mind that for mobile applications, databases should always be as light as possible, as we are dealing with devices that will probably have less capacity than usual computers:

Figure 11.6 – Database replica selected attributes

Tip

You can see more details about the construction of local data models, taking into account the weight of entities, here: https://success.outsystems.com/Documentation/Best_Practices/Development/OutSystems_Mobile_Best_Practices?utm_source=ost-outsystems+tools&utm_medium=ost-servicestudio&utm_campaign=ost-docrouter&utm_content=ost-helpid-30155&utm_term=ost-contextualhelp#Design_a_Lightweight_Local_Storage

And it's ready! Just click on the Add button and the entity will exist in Local Storage with the selected attributes:

Figure 11.7 – Local Storage entity generated from a database entity

In the Analysing data syncronization patterns section, we will see that the entities generated based on database entities allow the execution of accelerators to perform the synchronization between data in the Local Storage and in the server.

A peculiarity regarding Local Storage entities is that, unlike database entities, there is no concept of static entities (but they can still be synchronized – they're transformed to regular entities).

Local Storage entities shouldn't completely mirror the data in a server entity. Keep them to the absolute minimum required (performance advice). Local Storage entities can be used entirely on their own too (without any kind of server replica scenario, as standalone).

Now that we understand how we can create our entities in Local Storage we have to understand how we can get the data from these entities to use them in our applications. And we'll see that it's not that different from the way we get data from the database in the next section!

Fetching data from Local Storage

Since we're going to talk about aggregates, let's start by reviewing the definition.

An aggregate is a visual element of the OutSystems language that allows querying entity data. In an aggregate, you can define source entities, filter data, or sort data as needed.

We can visually fetch data from Local Storage using aggregates, just as we do when fetching data from the database. In fact, they are pretty much defined the same way.

The three tabs at the top of the editor allow us to add different data sources, create filters, and define the sorting:

Figure 11.8 – Aggregate with entity from Local Storage view

The first major difference about database aggregates that we can see is that they don't have test values or a data visualization. Since the data is not on the server, aggregates cannot view data directly from the device or use test values, even if the aggregate uses any variables. Keep in mind that there is Local Storage on each of your mobile devices.

The Sources section determines the source from which data is retrieved. We can add one or more Local Storage entities as a source and joins between source entities can also be added normally (we need to take into consideration that the "cost" of doing joins in local entities is very high in terms of performance).

Note

An aggregate does not allow mixing Local Storage entities and database entities in its source section.

The Filter section allows us to define one or more conditions to filter the aggregate output. To define the filters, we can use the attributes of the Local Storage entities defined as sources, as well as the logical operators or built-in functions that the platform provides:

Figure 11.9 – Filter functionality in aggregates with entities from Local Storage

The Sorting section allows you to define one or more attributes to sort the query results, in ascending or descending order.

The order of attributes in this section influences the output, with the first being the main sorting criteria and the others being used as tiebreakers:

Figure 11.10 – Sorting functionality in aggregates with entities from Local Storage

Local storage aggregates also allow calculated attributes. As with any other aggregate, these calculated attributes can be created through expressions, which have access to all attributes in the source Local Storage entities, as well as OutSystems built-in functions and variables accessible through the aggregates. A calculated attribute creates a new column in the aggregate output:

Figure 11.11 – Calculated attribute in aggregates with entities from Local Storage

There is also support for aggregating records, such as grouping multiple rows, or using aggregation functions such as sum or average. When we are aggregating records, only the aggregated columns will be part of the output.

On a mobile application screen, we can also define aggregates to fetch data from Local Storage and make the data available in the screen's scope.

Note

These aggregates cannot be used within data actions, since data actions are executed as server-side code.

Local Storage aggregates can also be used in client actions, either in the scope of a screen or in a global client action. They are defined in the same way as any other aggregate after being added to the action stream:

Figure 11.12 – Aggregates with entities from Local Storage used in Client Action

Widgets can be linked to data fetched from Local Storage. This can be done using the widget's source property so that when the widget is rendered, it will know what data will be displayed:

Figure 11.13 – Table widget linked with aggregate fetched from Local Storage

As we can see, fetching data from Local Storage is similar to fetching data from a database, with aggregates being defined in the same way.

In many cases, data present in Local Storage entities must conform to existing data in database entities. That is, there is a need to synchronize all this data. The platform offers a set of patterns for us to take full advantage of the data synchronization functionality, based on a set of criteria. Curious? Let's analyze this topic in the next section!

Analyzing data synchronization patterns

One of the capabilities that allow mobile applications to be quite functional is the fact that they allow data synchronization between the database and Local Storage These scenarios are very useful in contexts where applications work offline and data needs to be synchronized on both sides.

For this, OutSystems designed five synchronization patterns:

• Read-Only data – This pattern is useful for applications that only need to read data offline and the volume of data to transact between the database and Local Storage is low. This pattern is based on the principle of deleting all existing data on the Local Storage side, retrieving the data from the database, and creating it again on the Local Storage. An action for synchronizing Local Storage entities generated from database entities can be created automatically:

Figure 11.14 – Sync data (Read-Only) action autogeneration

• Read-Only data optimized – This pattern is useful when the mobile application needs to work offline and requires synchronization just to get data from the database to update Local Storage, but this time with a large volume of data. For this, the method is to save the data that was referenced when the last synchronization was done (we need to create an entity in Local Storage to manage this) and only carry out the processing of records modified, created, or deleted after the last synchronization. Here is an example:

This sample defines a database entity, Company, and its Local Storage counterpart, LocalCompany. Additionally, the SyncProperties Local Storage entity keeps the date and time of the last synchronization:

Figure 11.15 – Sync data (Read-Only) data model example

These are the steps we must follow to successfully carry out this synchronization model:

1. Track changed records by storing the timestamp of when the record was last updated or created.
2. Track deleted records.
3. Timestamp of the last synchronization. Note that this timestamp is established by the server to avoid problems due to clock differences between the client and server.
4. The application logic must keep the ModifiedOn and IsActive entity attributesupdated (marked as 1 and 2).

The following is a description of the logic of the OfflineDataSync client action:

Figure 11.16 – Sync data (Read-Only) OfflineDataSync example

These are the steps we must follow to successfully carry out this synchronization model:

1. Obtain the timestamp of the last synchronization.
2. Call the ServerDataSync server action to retrieve data from the database that changed since the last synchronization. The server returns a list of changed or added Company records, a list of deleted (inactive) Company records, and the timestamp of this synchronization.
3. Update the Company records in the Local Storage using the list of changed or added records returned by the server.
4. Iterate the list of deleted (inactive) Company records returned by the server and delete the corresponding records in the Local Storage.
5. Update the entity attribute SyncProperties.LastSync with the timestamp of this synchronization returned by the server.

The following is a description of the logic of the ServerDataSync server action:

Figure 11.17 – Sync data (Read-Only) server DataSync example

These are the steps we must follow to successfully carry out this synchronization model:

6. Assign the timestamp of this synchronization to an output parameter.
7. Obtains the list of changed or added Company records since the last synchronization. The aggregate uses the following filter:
8. Company.IsActive = True and (Company.ModifiedOn = NullDate() or Company.ModifiedOn >= LastSync)
9. Obtain the list of all deleted (inactive) Company records since the last synchronization. The aggregate uses the following filter:
10. Company.IsActive = False and (Company.ModifiedOn = NullDate() or Company.ModifiedOn >= LastSync)
11. Assign the timestamp and the two lists of Company records to the output parameters of the action.
    • Read/Write Data Last Write Wins – This pattern is typically used where it is unlikely that multiple end users will change the same data while applications are offline. The database contains the main data that can change over time and the Local Storage contains a subset of the main data and can be modified. Synchronization sends the modified data on Local Storage entities to the database and vice versa. On the server, data is updated in a "last write wins" strategy. That is, the latest data update replaces previous updates. An action for synchronizing Local Storage entities generated from database entities can be created automatically:

Figure 11.18 – Sync data (Read/Write) action autogeneration

Here is an example:

To automatically generate the logic needed to implement this pattern for an entity, follow these steps:

1. In Service Studio, open the Data tab.
2. Under Local Storage, select the local entity of the entity you want to synchronize with the server.
3. Right-click on the local entity and choose the Create Action to Sync Data (Read/Write) option (as in Figure 11.18).
4. This option is only available if the local entity is linked to the database entity (with the ID as a foreign key to the database entity). That happens if you create local entities with a right-click on Local Storage and choose Add Entity from Database.

This creates the actions needed to implement the Read/Write synchronization pattern:

• SyncLocal<Entity> – A client action that starts the synchronization between the local entity and the entity in the server database. It sends the added, changed, and deleted local records to the Sync<Entity> server action that handles the synchronization of the entity on the server side.
• Sync<Entity> – A server action called by the SyncLocal<Entity> action, which synchronizes the received local entity records with the server database records. It returns the current records of the entity in the database to be updated in the client's Local Storage.

Along with these actions, the new IsFromServer, IsModified, and IsActive attributes are added to the local entity to track changes and store meta-information needed by the synchronization process. To keep these new attributes updated and coherent for the synchronization process, the accelerator creates new client actions that must replace the use of the default local entity actions of the local entity:

• CreateOrUpdateLocal<entity>ForSync – Replaces the CreateOrUpdateLocal<entity> client action
• DeleteLocal<entity>ForSync – Replaces the DeleteLocal<entity> client action
• UpdateLocal<entity>ForSync – Replaces the UpdateLocal<entity> client action

These client actions are created in the Logic tab, under Client Actions, SyncActions_Local<entity>.

To guarantee the success of the synchronization process when using this accelerator, you must replace the use of all entity actions of the local entity with the corresponding new actions created by the accelerator.

If you want this pattern to run in the synchronization template mechanism, add a call to the SyncLocal<entity> client action in the OfflineDataSync client action.

This example defines a database entity, Company, and its Local Storage counterpart, LocalCompany. Additionally, the LocalCompany entity defines three metadata attributes to keep track of the synchronization status of the records.

Figure 11.19 – Sync data (Read/Write) data model example

The application logic must update the IsFromServer, IsModified, and IsActive metadata attributes of the local entity according to the following:

• IsFromServer: If True, the record exists on the server.
• IsModified: If True, the record has been modified locally.
• IsActive: If False, the record was deleted locally but may not yet have been removed from the server.

The following is a description of the logic of the OfflineDataSync client action:

Figure 11.20 – Sync data (Read/Write) offline DataSync example

These are the steps we must follow to successfully carry out this synchronization model:

1. Obtain the list of locally added Company records. The aggregate uses the following filter:
2. LocalCompany.IsFromServer = False and LocalCompany.IsActive = True
3. Obtain the list of locally updated Company records. The aggregate uses the following filter:
4. LocalCompany.IsModified = True and LocalCompany.IsFromServer = True and LocalCompany.IsActive = True
5. Obtain the list of locally deleted (inactive) Company records. The aggregate uses the following filter:
6. LocalCompany.IsActive = False and LocalCompany.IsFromServer = True
7. Call the ServerDataSync server action with the lists of locally added, updated, and deleted Company records as inputs. The server updates the data in the database and returns the list of updated Company records.
8. Delete all Company records in the Local Storage.
9. Recreate the Company records in the Local Storage using the list of records returned by the server.

The following is a description of the logic of the ServerDataSync server action:

Figure 11.21 – Sync data (Read/Write) server DataSync example

• Read/Write Data with Conflict Detection – This pattern is typically used where there is a likelihood that multiple end users will change the same data while applications are offline. The database contains the main data that can change over time and the Local Storage contains a subset of the main data and can be modified. Synchronization sends the modified data on Local Storage entities to the database and vice versa. On the server, the data is updated and if conflicts are detected, they are marked for future resolution.
  

Figure 11.22 – Sync data (Read/Write with conflict detection) server DataSync example

• Read/Write Data One-to-Many – This pattern is typically used with data models where "one-to-many" relationships exist and where multiple end users are unlikely to change the same data while applications are offline. The database contains the core data that can change over time and the Local Storage contains a subset of the core data and can be modified. Synchronization sends the modified data in Local Storage entities to the database and vice versa. On the server, data is updated in a last write wins strategy. That is, the latest data update replaces previous updates:

Figure 11.23 – Sync data (Read/Write One-to-Many) server DataSync example

Tip

You can see more details about data synchronization patterns at the following link: https://success.outsystems.com/Documentation/11/Developing_an_Application/Use_Data/Offline/Offline_Data_Sync_Patterns.

The best way to internalize these concepts is to see how they are actually used. OutSystems makes available on the Forge examples of the implementation of all patterns identified in this section. You can download the application here: https://www.outsystems.com/forge/component-overview/1638/offline-data-sync-patterns.

In a general context, patterns can be customized to suit business and user needs. We can see these patterns as accelerating templates and as being the best algorithms for most real cases, but at any time, we can create our own patterns if warranted. This functionality guarantees much more cohesive and satisfactory behavior in offline scenarios, which are frequent on mobile devices.

Summary

In this chapter, we understood what Local Storage entities are, and saw that the way they are created and manipulated is not that different from database entities. Furthermore, we realized that we can create them as replicas of existing ones in the database, thus also allowing us to speed up the creation of synchronization mechanisms between them.

We also learned how to obtain data from these entities for our applications and that we cannot mix Local Storage entities with database entities.

Finally, we understood the fundamentals of the most common data synchronization patterns between Local Storage and the database, and that OfflineDataSync ensures the functioning of offline mobile applications in a much more cohesive and satisfying way.

In the next chapter, we're going to shift our focus a bit. Let's think more about the frontend at the UX/UI level and how we can make our applications more attractive by using style guides! Curious? So, let's turn the page!