The final part of this book will provide deeper coverage of some technical aspects of the Funambol platform.
This section is more informative than the first part of the book, and Maria will probably skip it. However, her development team will find this additional information particularly useful to expand the possibilities of the Funambol platform beyond the immediate needs of personal data synchronization and mobile e-mail. The horizon of possible applications that can be mobilized is wide and becomes wider every day with the adoption of more powerful personal devices such as smartphones and netbooks.
A key aspect of the Funambol platform is that it is based on an open standard. SyncML is the standard protocol defined and maintained by the Open Mobile Alliance that is, OMA, (http://www.openmobilealliance.org) that addresses the problem of data synchronization between different devices and systems.
The full set of specification documents can be found at http://www.openmobilealliance.org/Technical/current_releases.aspx, under the Data Synchronization section as shown in the following screenshot:
It is worth noting that SyncML not only groups the specifications for data synchronization as seen until now, but also covers the so called SyncML Device Management, which will be briefly introduced in this chapter.
In December 2000, Ericsson, IBM, Lotus, Motorola, Nokia, Palm, Psion, and Starfish Software formed the SyncML initiative to accelerate the market's vision of ubiquitous data access from any networked device. Their goal was to create a common synchronization protocol that could be used industry-wide. They worked with end users, device manufacturers, data providers, infrastructure developers, application developers, and service providers to define a common mobile data synchronization protocol that would satisfy the resource constraints of mobile devices and wireless networks and still provide the extensibility to support a range of applications and data types. The result was SyncML.
In 2002, the original SyncML initiative was consolidated into the Open Mobile Alliance under the Data Synchronization Working Group (OMA DS) and Device Management Working Group (OMA DMs). OMA is the leading industry forum for developing market-driven, interoperable mobile service enablers. OMA was formed in June 2002 by nearly 200 companies including the world's leading mobile operators, device and network suppliers, information technology companies, and content and service providers.
The mission of the Open Mobile Alliance is to facilitate global user adoption of mobile data services by specifying market-driven mobile service enablers that ensure service interoperability across devices, geographies, service providers, operators, and networks, while allowing businesses to compete through innovation and differentiation.
SyncML Data Synchronization is defined as the process of making two sets of data look identical. To achieve this goal, starting from a coherent initial state (for example, after synchronization) the two sets of data need to be modified accordingly and a conflict resolution policy must be followed. A conflict arises when the same data element is modified in both sets in an inconsistent way.
What is a synchronization protocol? First of all, it is a way to communicate the modifications between different datasets over a period of time.
The primary characteristics of a synchronization protocol are as follows:
The addressing of the datasets (or database) and single records (in order to identify a particular database or a single record in a database)
The definition of the standard commands needed to communicate modifications
The definition of the ways these modifications can be exchanged (required features, transport protocols, and so on)
SyncML is an initiative to develop and promote a single, common data synchronization protocol that can be used industry-wide. Moreover, SyncML is a specification for a common data synchronization framework based on the exchange of XML-based messages between networked devices. SyncML is sponsored by companies such as Motorola, Nokia, Ericsson, IBM, and many others.
The SyncML specifications include:
The following figure shows a simple case where a mobile phone acts as a client connecting to a server.
The phone transmits a SyncML message to the server, including the most recent modifications. The server synchronizes its dataset (based on the changes received by the client, in terms of commands such as Add, Delete, and Replace) and responds with a SyncML message including its own modifications. This is a trivial example, but is useful to show the role assumed by the two devices during synchronization.
The SyncML client contains a synchronization agent and typically sends its modifications first. The SyncML server is the device implementing both the server-side synchronization agent and the synchronization logic, including the procedures to interpret the modifications, to discover and manage conflicts, and to generate return messages.
SyncML defines seven synchronization modes:
Two-way sync: The most common synchronization mode, where client and server exchange modifications on their data. The client first sends the modifications and the server then returns its own.
Slow sync/full sync: This is a special case of two-way sync where all the client database records are compared, field by field, with the ones on the server. The client sends the full database to the server, which in turn analyzes all records detecting missing items that need to be sent back to the client. It is typically used to recover from a situation where it is impossible to resolve synchronization conflicts and to reset to a consistent state between the client and the server. Another case when slow sync could be used is when client and server detect that they are out of sync so that a correct state must be recreated from scratch.
One-way sync from the client only: The client sends its modifications to the server, but no records from the server are expected nor returned.
Refresh sync from client only: The client sends its entire database to the server, which replaces its database with the data sent from the client.
One-way sync from the server only: The server sends its modifications to the client, but no client modifications are expected in return.
Refresh sync from server only: The server sends its entire database to the client. The client replaces the local set of data with the received one.
Server alerted sync: The server sends a synchronization alert to the client notifying it of the synchronization mode to be used.
It is not necessary for all clients to implement and support all these types of synchronization. For instance, the Funambol Mozilla Sync Client (described in Chapter 1) implements fast sync, full sync, refresh from client, and refresh from server. A server, however, is required to implement all synchronization types.
Note
With the Funambol Mozilla Sync Client, fast and slow syncs are selected automatically by the client, based on the status of the synchronization between the client and the server. Refresh syncs can instead be launched from the application menu selecting Tools | Recover. The following screenshot will be displayed and the user can choose whether to restore the server or the client database.
In the following sections, we will consider the basic elements needed to understand data synchronization in general and SyncML synchronization in particular. The following concepts will be presented:
In order to allow a sanity check on the synchronization state between two devices (was the last synchronization successfully completed?), SyncML makes use of two synchronization anchors, Next and Last. Each database owns the anchors which are sent to the other device during the synchronization initialization.
Last represents the last synchronization successfully performed. Usually it is a timestamp, but it could be a different identifier such as a session identifier. Next is a tag representing the current synchronization. Client and server exchange their anchors at the beginning of the synchronization process and are required to store the next anchor of the counterpart at the end of the sync process. Using this piece of information the server can detect if the last synchronization encountered problems and did not end correctly. For example, if the given Last is equal to the stored anchor, then the last synchronization was successfully terminated and, if the anchors do not match, client and server are out of sync and have to take actions to solve the issue. For this to work, it is important that the Next anchors are saved into the local repository only at the end of a successful synchronization.
In a multidevice synchronization scenario, the client and server must be able to create and deal with local item IDs independently. As a result, the server and client may have a different identifier for the same item while the server has to keep a mapping table between the client's Locally Unique IDs (LUID) and the server's Globally Unique IDs (GUID) as shown in the following figure:
Note that LUIDs are always created by the client device and only later communicated to the server with the SyncML command map.
A modification conflict arises when the same item has been modified on both server and client. When this happens, the synchronization engine has the responsibility of resolving the conflict. The client application is notified of the error condition by means of a status code.
For example, the following message represents a case where the server is notifying the client about a conflict resolution:
<Status> <CmdID>1</CmdID> <MsgRef>1</MsgRef> <CmdRef>2</CmdRef> <Cmd>Replace</Cmd> <SourceRef>1212</SourceRef> <Data>208</Data> <!-- Conflict, originator wins --> </Status>
Addressing defines the way in which entities involved in the synchronization (databases, items, and so on) can be addressed. This is achieved by a naming convention which is defined on the basis of URIs.
For example, a typical server addressing URI might be:
<Source> <LocURI>http://sync.syncml.org/sync-server</LocURI> </Source>
A client could use a different addressing identifier such as its IMEI number:
<Source> <LocURI>IMEI:493005100592800</LocURI> </Source>
Also, the dataset to be synchronized is addressed with a URI. Note that the URI can be either absolute or relative.
<Sync> <Target> <LocURI>./calendar</LocURI> </Target> </Sync> <Sync> <Target> <LocURI>http://www.syncml.org/sync/calendar</LocURI> </Target> </Sync>
Finally, even a single item of a database can be identified with the same method.
<Item> <Source> <LocURI>101</LocURI> </Source> </Item>
SyncML defines how the devices can exchange their capabilities during the initialization phase. Note that the synchronizing devices are not required to send their capabilities unless requested by the counterpart. The exchange of device capabilities is very useful at the server side, as the server can apply proper optimization according to the remote device resources and features (database types, available memory, and transmission speed).
One thing to consider in this context is that capabilities information can be quite large. For this reason, the device's capabilities should be sent only if requested. In addition, the information should conform to the capabilities of the counterpart. For example, if the client notifies the server that it does not support the data format vCard3.0, the server should not insert in its response, the vCard3.0 properties it supports.
The complete SyncML synchronization process can be represented as a sequence of phases, as shown in the following figure:
This is a three-step sequence—initialization, modifications exchange, and ID mapping.
Each step of the synchronization process is a packet, which is composed of multiple messages. This is necessary in order to meet particular requirements of the transport protocol or to support connection and device limitations.
SyncML specifies that before sending data modifications, the synchronizing counterparts exchange an initialization packet specifying the device characteristics and the synchronization requirements (for example, which databases need to be synchronized, which type of sync is needed, or which protocol features are supported). Note that the initialization packet can be merged with the modification package thereby reducing the chattiness of the communication. The drawback is that the synchronization process cannot be optimized as it could be, if the information capabilities were available before performing the synchronization analysis. The initialization package may also contain authentication credentials.
Each SyncML message is a well formed XML document with the element<SyncML> as root element and container for all the other tags. A single message is represented by a header within the element<SyncHdr> and a body enclosed in a<SyncBody> element as shown in the following image:
Note that as the XML message is not validated, the XML declaration and prolog can be omitted. The header contains routing information, database addressing, and protocol versioning, while the body contains one or more synchronization commands, as illustrated in the following image:
Each command is represented by a specific XML element, which can be a container for subelements, data items, or meta information.
SyncML commands can be divided into Request commands and Response commands.
Request commands include Add, Alert, Atomic, Copy, Delete, Exec, Get, Map, Put, Replace, Search, Sequence, and Sync. Response commands include Status and Results.
Note that the protocol does not specify any semantics for these operations and, as a result, the Add operation could have a different meaning on a different device.
The following table gives a brief description of each command:
|
Command |
Description |
|---|---|
|
|
Adds items to a database |
|
|
Notifies the counterpart of the intention to synchronize a particular database |
|
|
All items included within an |
|
|
Copies the items into the destination database |
|
|
Deletes the included items from the destination database |
|
|
Executes a program on the destination device |
|
|
Requests information from the remote device |
|
|
Updates LUID-GUID mapping |
|
|
Sends information (such as capabilities) to the device |
|
|
Replaces the included items into the destination database |
|
|
Specifies a search on the other device |
|
|
The included commands must be executed sequentially |
|
|
The container for the modification commands |
|
|
Status notification for the execution and interpretation of any other command |
|
|
Contains |
A two-way sync looks like the following sequence of messages:
<SyncML> <SyncHdr> <VerDTD>1.1</VerDTD> <VerProto>SyncML/1.1</VerProto> <SessionID>1</SessionID> <MsgID>2</MsgID> <Target> <LocURI>http://www.syncml.org/sync-server</LocURI> </Target> <Source><LocURI>IMEI:493005100592800</LocURI> </Source> </SyncHdr> <SyncBody> <Status> <CmdID>1</CmdID> <MsgRef>1</MsgRef><CmdRef>0</CmdRef> <Cmd>SyncHdr</Cmd> <TargetRef>IMEI:493005100592800</TargetRef> <SourceRef> http://www.syncml.org/ sync-server </SourceRef> <Data>212</Data> <!--Statuscode for OK, authenticated for session--> </Status> <Status> <CmdID>2</CmdID> <MsgRef>1</MsgRef><CmdRef>5</CmdRef> <Cmd>Alert</Cmd> <TargetRef>./dev-contacts</TargetRef> <SourceRef>./contacts/james_bond</SourceRef> <Data>200</Data> <!--Statuscode for Success--> <Item> <Data> <Anchor xmlns='syncml:metinf'> <Next>200005022T093223Z </Next> </Anchor> </Data> </Item> </Status> <Sync> <CmdID>3</CmdID> <Target><LocURI>./contacts/james_bond </LocURI></Target> <Source><LocURI>./dev-contacts</LocURI></Source> <Meta> <Mem xmlns='syncml:metinf'> <FreeMem>8100</FreeMem> <!--Free memory (bytes) in Calendar database on a device --> <FreeId>81</FreeId> <!--Number of free records in Calendar database--> </Mem> </Meta> <Replace> <CmdID>4</CmdID> <Meta><Type xmlns='syncml:metinf'>text/x-vcard </Type></Meta> <Item> <Source><LocURI>1012</LocURI></Source> <Data> <!--The vCard data would be placed here.--> </Data> </Item> </Replace> </Sync> <Final/> </SyncBody> </SyncML>
The request header contains the version number of the protocol, the DTD utilized, the session and message IDs, and the addressing of the target server (Target indicates the destination server and Source the client identifier).
The body starts with a<Status> command (note that the reported message is a modification message, we are assuming that the initialization already took place).
The next<Status> command responds to an<Alert> command requesting the synchronization of the server database ./contact/james_bond. Note the use of the next anchor as described in the earlier section Sync anchors.
The next command in the message is a<Sync> command containing the synchronization operations to perform.<Target> and<Source> specify the databases, while the<Meta> command includes information regarding the client status.
Finally, the command<Replace> represents the required operation (replace the inner items with the data contained in<Data>). Note that the<Meta> element specifies the data type of the data to be stored.
The following code is an example of a response:
<SyncML> <SyncHdr> <VerDTD>1.1</VerDTD> <VerProto>SyncML/1.1</VerProto> <SessionID>1</SessionID> <MsgID>2</MsgID> <Target><LocURI>IMEI:493005100592800</LocURI> </Target> <Source><LocURI>http://www.syncml.org/sync-server </LocURI> </Source> </SyncHdr> <SyncBody> <Status> <CmdID>1</CmdID> <MsgRef>2</MsgRef><CmdRef>0</CmdRef><Cmd>SyncHdr </Cmd> <TargetRef>http://www.syncml.org/sync-server </TargetRef> <SourceRef>IMEI:493005100592800</SourceRef> <Data>200</Data> </Status> <Status> <!--This is a status for the client modifications to the server.--> <CmdID>2</CmdID> <MsgRef>2</MsgRef><CmdRef>3</CmdRef><Cmd>Sync</Cmd> <TargetRef>./contacts/james_bond</TargetRef> <SourceRef>./dev-contacts</SourceRef> <Data>200</Data> <!--Statuscode for Success--> </Status> <Status> <CmdID>3</CmdID> <MsgRef>2</MsgRef> <CmdRef>4</CmdRef><Cmd>Replace</Cmd> <SourceRef>1012</SourceRef> <Data>200</Data> <!--Statuscode for Success--> </Status> <Sync> <CmdID>4</CmdID> <Target><LocURI>./dev-contacts</LocURI></Target> <Source><LocURI>./contacts/james_bond</LocURI> </Source> <Replace> <CmdID>5</CmdID> <Meta> <Type xmlns='syncml:metinf'>text/x-vcard</type> </Meta> <Item> <Target><LocURI>1023</LocURI></Target> <Data> <!--The vCard data would be placed here.--> </Data> </Item> </Replace> <Add> <CmdID>6</CmdID> <Meta> <Type xmlns='syncml:metinf'>text/x-vcard</type> </Meta> <Item> <Source><LocURI>10536681</LocURI></Source> <Data> <!--The vCard data would be placed here.--> </Data> </Item> </Add> </Sync> <Final/> </SyncBody> </SyncML>
The structure of the message is similar to the request message. Note the<Status> command with<CmdId> equals to 2: It is the status of the synchronization process; it has a value 200 in our case, which indicates that the synchronization completed successfully. The server modifications (one<Replace> and one<Add>) are embedded in the<Sync> command addressed to the database:./dev-contacts.
The<Final> command indicates that this is the last message in the packet. The server therefore expects no further messages from the client.
Configuring a mobile device is a difficult task, even for an expert user. Network parameters change from phone to phone, visual interfaces are difficult to use and understand, and all mobile operators have different configurations. Moreover, mobile devices are becoming smarter every day. New features are prepackaged on the device and new applications are downloaded and installed by the user. As a consequence, the task of configuring a mobile device is getting even more complex. Consumers, Corporate Information Management (IM/IT) departments, and operators are looking for a platform that allows remote management of devices in a convenient and effective way.
To answer this pressing need, some companies have started offering device management implementations for the mobile world. However, the available solutions are all based on different proprietary implementations. A proprietary platform exposes end users, mobile operators, device manufacturers, and Enterprise/IT departments to a serious risk of insufficient interoperability and associated cost implications. In the long run, it means being locked in with a vendor and a technology, while the market is moving towards adopting new standards.
The OMA Device Management protocol is an open, universal industry standard for remote device management of networked devices. It is a widely adopted protocol that is nowadays available on majority of the mobile phones in the market (even if the service can be hidden to the user or not implemented by carriers).
Device management is a generic term used for a technology that allows third parties to carry out the difficult procedure of configuring mobile devices on behalf of the end user. Third parties would typically be wireless operators, service providers, or corporate information management departments.
Through device management, an external party can remotely set parameters, troubleshoot service problems, and install or upgrade software.
In broad terms, device management consists of three parts:
Protocol and Mechanism: The protocol used between a management server and a mobile device.
Data model: The data made available for remote manipulation, for example, browser and mail settings.
Policy: The policy specifying who can manipulate a particular parameter or update a particular object in the device.
In a wireless environment, the crucial element for a device management protocol is the need to efficiently and effectively address the characteristics of mobile devices, including low bandwidth and high latency. The OMA DM protocol has been built keeping these requirements in mind.
OMA DM device management scope includes device configuration (modifying or reading operating parameters), software maintenance, inventory (reading from a device its current operating parameters, listing installed or running software, determining hardware configurations), and diagnostics (listening for alerts sent from a device, invoking local diagnostics on a device).
The OMA DM protocol is relatively simple from the standpoint of messaging sequence. The message sequence is essentially broken into three parts, as illustrated in the following diagram:
This phase is optional and data flows only from server to client. The server sends a notification package to the client requesting it to start a new management session. This is usually done with a WAP push message sent by a Push Proxy Gateway on behalf of the DM server that acts as the WAP Push initiator agent.
This phase is always required and data flows from client to server. It consists of a request from the client and the response from the server. The initial client request contains three primary pieces of information:
The server responds to the initial client request with server credentials with the goal of identifying the server to the client for authentication and identification purposes. The server also sends user interaction and the first management command with the response.
These two transactions represent the core of the management session. If the server sends a management command to the client in Transaction 3, then the client responds with data and status. Afterwards, the server can issue new management commands or simply return status information. The management session ends when the server sends neither additional management nor user interaction commands.
Device configuration data is organized in a hierarchical structure called the device management tree. Subtrees of the device management tree are called device management nodes, and a leaf, usually a single configuration parameter, is called a manageable object. Device objects can be anything from a single parameter, to a splash screen GIF file, to an entire application.
The device management tree is essentially mapped to permanent or dynamic objects as an addressing schema to manipulate these objects. Permanent objects can be thought of as objects that are built into the device at the time of manufacturing and typically cannot be deleted. Dynamic objects are objects that can be added or deleted (for example, network parameters, ring tones, or wallpapers).
As previously mentioned, the initial request from the client always contains device information whereby the data is retrieved from the ./DevInfo subtree. The ./DevInfo node is only part of the overall device management tree structure, and it maps to basic device parameters that will allow initial operations and inspection of the device by the customer support specialists. The ./DevInfo object is a standardized object so that any compliant device exposes the same basic information following the structure illustrated in the following figure:
The management subtree in the figure is a good example of how management objects are organized. ./DevInfo is the main node that includes subtrees (other management nodes) and leaves (manageable objects).The preceding figure shows the following subtrees:
In addition to the manageable object value, each manageable object has a set of properties associated with it. These properties define metadata information about an object to allow things such as access control. These properties are listed in the following table:
|
Property |
Description |
|---|---|
|
|
Defines who can manipulate the underlying object (required) |
|
|
Specifies how an object should be interpreted. For example, if the underlying object is a URL for a particular management server then the |
|
|
The name of the object in the tree (required) |
|
|
The size of the object in bytes (required for leaf objects, not applicable for interior nodes) |
|
|
User-friendly name of the object (optional) |
|
|
The timestamp of the last modification (optional) |
|
|
The MIME type of the object (required for leaf objects, optional for Interior Nodes) |
|
|
The version number of the object (optional) |
Funambol has an implementation of a SyncML Device Management server. It is a community project so that Funambol staff is not directly involved in the maintenance and development of the project. The project can be found on the Funambol forge at the URL https://dm.forge.funambol.org.
In this chapter, the SyncML protocol was described in detail. SyncML specifications cover two main areas of mobile applications—data synchronization and device management. The book is focused on the former protocol and applications, but for the sake of completeness it is worth learning more about SyncML DM, for which Funambol hosts a community implementation of the server.