OPERATING SYSTEMS AND SOFTWARE FOR MOBILE TV AND MULTIMEDIA PHONES
It isn’t that they can’t see the solution. It is that they can’t see the problem.
—Grover Cleveland, 22nd and 24th president of the United States
12.1 INTRODUCTION—SOFTWARE STRUCTURE ON MOBILE PHONES
Mobile phones have been with us for well over 2 decades. Do we need to look at the operating systems and software structure just because we are moving toward multimedia and mobile TV type applications? What makes these phones different and the software structure important? We will attempt to piece together this information in this chapter.
As we know, operating systems (OSs) were initially designed for the management of the “machine,” i.e., to handle all input/output, memory, disk drive, display, printers, keyboard, and serial and parallel interfaces on the “computer.” This was supposed to isolate the application software writers from the low-level functions of controlling to the machine peripherals. As the functions that had to be supported on the machines became virtual standards they began to be supported as part of the operating systems. In the not-so-long-ago past, the Windows 3.1 operating system had neither a TCP/IP stack nor a browser. These had to be installed separately if one wished to connect to the Internet. Having any type of media player natively as a part of the operating system was unheard of. As the use of the Internet became universal, it became necessary to incorporate these features into the operating system and, after the well-reported controversy on “monopolistic” practices, the Internet Explorer became part of the Microsoft operating system. So did a series of other features such as Infrared, Bluetooth, and Media Players in successive versions of the software releases. The Microsoft Windows operating system today reportedly has over 50 million lines of source code. Ever-ready hardware vendors have come forth with machines with 256+ Mbytes of RAM and sufficient hard disk, and new processor releases keep the machines running.
Mobile phones have always presented a unique operating environment. Mobile devices are constrained in terms of resources (processor capabilities, memory, battery power) and yet are required to deliver virtually all the functions a multimedia PC may be expected to deliver, but on smaller screens. Mobile phones are required to provide all types of connectivities such as Bluetooth Wireless, Infrared, Wi-Fi, Wi-LAN, WiMAX, USB, multimedia capabilities for audio and video, I/O capabilities for camera, keyboard, display, touch screen, external devices, external memory, USB, printers, and scanners as well as support for office applications such as POP3 mail, Microsoft Office, Adobe PDF document readers, image processing, and more. This is in addition to the mobile network functions such as GSM or CDMA network connectivity and broadcast TV receivers, functions that are absent in PCs. The functionalities of connection to the network are quite onerous, with support of multiple types of protocols and transmission conditions that lead to loss of connectivity and errored transmissions, handover, and roaming. The operating environment of mobiles is such that connectivity cannot be guaranteed due to transmission conditions. Packet losses can be expected to occur. This has implications on the client-server type applications.
The older versions of mobile phones limited to voice applications and SMS generally had no visible operating systems. The code written was proprietary to the hardware and the specific phone. All applications were directly coded. However, as more advanced phones and applications started coming in, this was no longer practical. The phones needed an operating system that could be ported on different processors and applications that could be migrated to new environments without having to be rewritten.
The hallmark of any mobile application has to be reliability, as any software hang-up would be unacceptable, as the phone would become useless in the hands of users who are not expected to be familiar with computers or software. Hence it is clear that a mobile phone operating system can-not be a subset of the desktop operating systems, at least in terms of functionality and performance. Mobile devices need a new operating system with a new way to interface with the applications on the one hand and the hardware and embedded software systems on the other.
What is special about multimedia phones? Multimedia phones take the OS requirements to a new level of functionality. The user applications require simultaneous voice, data, video, and audio usage, needing a high level of multitasking functionalities. For example, a phone may be used for a video call involving the use of the 3G network and local camera while simultaneously recording a TV clip or audio file and retrieving files from memory. In addition there are requirements for high-speed games, animations, and graphics, which in turn require native support of their building blocks. The functional modules needed to support multimedia phones are much higher than a standard mobile phone OS kernel.
12.2 SOFTWARE ORGANIZATION IN MOBILE PHONES
In order to understand the major differences, let us look at the functionalities and organization of software in a mobile phone. The software can be classified broadly in the following layers:
• operating system,
• middleware,
• application software, and
• access control.
FIGURE 12-1 Software Structure on Mobile Phones
At the basic layer is the silicon, which has the control processor and the applications processors embedded on it along with their drivers. The next layer is the OS kernel, which has an interface to the hardware drivers and devices on the silicon such as memory. The operating system layer can be seen as consisting of several stacks supporting services such as “Bluetooth Stack”, “Infrared Stack”, etc. (Fig. 12-1).
The OS kernel is followed by a “middleware layer,” which comprises programs that make use of the operating system services and function-specific “high level” application program interfaces (APIs) to the application software. Middleware software examples can include multimedia-specific software, mobile communications (3G, CDMA2000, GPRS), or specific software for animation, games, etc. It can also contain operator-specific features that help in easy porting of standards-compliant applications such as push-to-talk and video calling from multiple vendors.
The applications programs can constitute a wide range, such as MMS, browser, games, position location, instant messaging, video calling, mobile TV, and makes of OS functionalities and middleware functionalities provided as APIs.
12.3 WHY IS THE OPERATING SYSTEM IMPORTANT IN MOBILE PHONES?
12.3.1 Functional Requirements
In the multimedia world, the scenario has changed considerably since the original conception of a mobile as a communication device. Mobile handsets now provide:
• full IP-based communication, including IPv4, IPv6, SIP, VoIP, etc.;
• high memory capacity of 2–8 Gbytes or more;
• larger screens, higher resolution, more colors;
• improved cameras up to 8 Mpixels and advanced optics;
• streaming and broadcast TV and radio reception capabilities;
• richer media consumption—MP3/AAC music and XHTML browsing;
• mobile office tools, including e-mail and Microsoft Office capabilities (Word, PowerPoint, Excel);
• multi-band multi-standard communications support;
• navigation and position location information; and
• downloadable third-party applications.
Owing to the feature-rich nature of the phones it is evident that the applications now dominate the mobile phone scene rather than the basic function of communications. The nature of applications changes very fast as the hardware capabilities grow and to cater to the preferences of the market. This requires that a large number of developers be able to work ceaselessly and deliver applications that conform fully to the operator network and adhere to the protocols and standards and guidelines such as OMA, 3GPP, and W3C. The operating systems need to provide an environment not only meeting the capabilities of a multimedia device but also providing fast turnaround of new applications.
12.3.2 Support of Device Drivers and Protocol Stacks
Figure 12-2, which exhibits the software structure on mobile phones, also exhibits an important aspect of an operating system, i.e., support of device drivers as well as protocol stacks for important services commonly provided on mobile phones, such as Bluetooth, audio and video encoders, Infrared, Internet protocols (TCP/IP, RTP, SIP, WAP), audio and video support for recording, and layback. Other functionalities that are common across networks include support for messaging (SMTP, SMS, and MMS), graphics, Java support, communications network support (GSM-, GPRS-, EDGE-, WCDMA-, and CDMA-based networks), application suites (for personal information management, calendar, phone book, etc.), and access security functions. If an operating system supports these functions, then any application software (or middleware) can use the functionalities provided in the form of protocol stacks or ready services. This makes it possible to write applications that are network-specific variations of common services and provide them independent of underlying hardware. Other important functions of a mobile OS are power management, over-the-air (OTA) synchronization, and real-time operating system functionality.
FIGURE 12-2 Operating System and Middleware on Multimedia Phones
12.3.3 Mobile Phone Functionality—OS vs Application Software Modules
All the above functions are common to the mobile phone world irrespective of the operator or the phone manufacturer and it is expected that an operating system should support these functions to reduce development time of applications. Support of a majority of mobile phone functionalities as a part of the OS has been a hallmark of the Symbian operating system. The alternative approach is to have the majority of functions supported by OS software module add-ons, middleware, or applications software while having an OS with limited functionality but open source. Linux OS fits well in the latter philosophy. However, it is still finding strong support in view of the fact that it is an open source and development work on the platform can be much more broadly based.
However, it should be clear that the users do not always select the handsets based on the OS but on the user interface, applications, and branding. The Windows Mobile 5.0 OS is now changing this trend to the extent that smart phones with Windows Mobile OS and Windows Office mobile support have a niche with users.
12.3.4 Real-Time Operating Systems
A real-time operating system is desirable in a mobile phone in an environment characterized by multiple resource-intensive applications. Multimedia applications with high CPU requirements for video encoding, display, transcoding, and other functions present an environment that benefits from a real-time operating system kernel.
12.4 COMMON OPERATING SYSTEMS FOR MOBILE DEVICES
There are a number of operating systems that have come to be used on mobile phones over time. These include Symbian, Linux, Windows Mobile, Palm OS, and BREW as the prominent ones.
12.4.1 Symbian
Symbian was formed by Motorola, Nokia, Ericsson, and Psion in 1998 for developing operating systems for mobile devices. The first mobile phone based on Symbian was released in 2000 (Ericsson R380). The Symbian OS undoubtedly merits attention as it is supported by major mobile manufacturers and claimed over 85% of the smart-phone world market in 2006.
The Symbian operating system was designed specifically for mobile phones as opposed to some of the other systems derived from desktop systems. The Symbian OS therefore has APIs for messaging, browsing, communications, Bluetooth IrDA, keyboard, and touchscreen management and supports Java Virtual Machine. This enables the applications to be written in Java for better portability.
Symbian platforms quickly began to be associated with providing rich interworking applications and interfaces for mobile phones after their launch. Features such as full HTML browser, video telephony, streaming, messaging, presence, “push-to-talk,” Java support, branded keys, default wallpapers, operator menus, etc., were well appreciated. The Symbian platforms also provided support to Java, 3GPP, 3GPP2, OMA, and BREW.
Japan’s FOMA network has been the biggest user of the Symbian OS, which it had used exclusively until the recent past, when it began support of Linux as well. The Symbian OS is used extensively in Nokia phones. It is also used extensively in Europe and Asia. Some of the latest phones announced based on Symbian include the Sony Ericsson W950 Walkman, Nokia N93, and Samsung SH-902iS.
The key features of Symbian OS 9.3 are given in Table 12-1. The feature-rich support by the OS to the applications is evident.
Architecturally the Symbian OS provides a user interface (UI) framework through which the applications can access the OS-provided services (Fig. 12-3).
The architecture of Symbian is somewhat unique as it as a user interface layer separate from the operating system and hence needs to be selected individually by the phone manufacturer. The UIs are developed by Nokia, UIQ, and network operators such as NTT DoCoMo for its FOMA network.
Symbian has come out with a functional series for its phones that is characterized by screen size and support of various features. They range from S20, which was introduced at the initial release of the phones, to S60, used for smart phones. There are additional releases, e.g., S90, with varying sizes of displays and feature support. Figure 12-4 depicts the progression of Symbian operating systems.
Among the Symbian phones, S60 from Nokia has proved to be a very popular interface, and a number of phones are based on S60.
The Symbian operating system has had limited use in CDMA networks in the United States owing to its nonsupport of the frequency bands.
Symbian Operating System Features
FIGURE 12-3 Symbian OS Architecture (Version 9.3)
FIGURE 12-4 Symbian Series of Phones
12.4.2 Linux
Linux is a comparatively new entrant in the mobile OS arena. Linux is an open source software, the code of which is freely downloadable from the net. Linux is distributed under a GPL (general public license), implying that its source code must be made publicly available whenever a compiled version is distributed. The kernel has been adapted by phone manufacturers owing to its open source software and future availability of open source software and applications. China, for example, has adapted the use of Linux as OS (Linux embedded version “mLinux”) for use in its 3G mobile networks, a fact that will influence the mobile market considerably. Also, major manufacturers and operators, including NTT DoCoMo, Vodafone, NEC, Panasonic, Motorola, and Samsung, have announced support of a global platform for open Linux adoption. Most phone vendors are now working on Linux owing to the availability of third party software, which is especially important at the high end, such as for multimedia applications.
FIGURE 12-5 Typical Linux Environment for Multimedia Phones
The Linux core is, however, limited in functionality, and a majority of mobile phone functions such as multimedia support, communication functions, connectivity services, and platform management services need to be supported by middleware. Figure 12-5 depicts how the Linux OS kernel is located vis-à-vis the applications. A large part of the functions of the phone need to be supported by software modules that are beyond the OS. This has, however, not deterred many vendors in giving preference to Linux because it is open source. The need for uniform and bug-free implementation of the software modules has, however, raised concern as to the stability of the phone as its operation is now dependent on external modules. To alleviate these concerns, the CE Linux Forum is now working on a global reference architecture and common API for various components of software and middleware. The common architecture will include the videophone framework, telephony framework, and multimedia framework.
With regard to the actual implementation of Linux on the mobile phones, there are at present many variants being used.
mLinux is an embedded version of Linux adapted for use in 3G networks in China. Embedded Linux implies a Linux kernel, which has been ported to a particular CPU and is available as code in memory on the system-on-chip or board of the CPU.
mLinux supports all major CPUs such as ARM9, Intel Xscale, MIPS, Motorola Dragonball, i.Mx, and Texas Instrument OMAP processors (710, 730, 1510, and 1610), among others. mLinux features an enhanced bootloader and short boot times of less than 2 sec. It has multiprocessing and multithreading capability and memory management and provides support for pipelining. It also supports many protocols and network cards.
Many vendors have opted for commercial versions of Linux to have better functionality, which is natively supported by the OS while retaining its open source nature. An example is NTT DoCoMo, which has selected Monta Vista Linux to be an OS for the FOMA phones in addition to Symbian. Monta Vista Linux provides for easy integration of advanced multimedia applications and a standard development platform for wireless handset designs. The NEC Corporation has developed phones based on Monta Vista Linux for FOMA (N900iL and N901iC).
12.4.3 Palm OS
Palm has virtually been the symbol for PDAs since the days of the Palm-Pilot. It is only recently that the market has diversified through other PDAs and Pocket PCs.
The Palm OS is a multitasking operating system that provides protective memory management and Hot Sync capabilities. Palm supports the ARM and Motorola 68000 processors. Palm, Inc., the company owning the Palm OS, was actually split in two parts in 2003, i.e., palmOne and PalmSource. Subsequently palmOne became known as Palm, Inc. PalmSource is the company that licenses operating systems. The Palm OS has two versions, i.e., Palm OS Cobalt and Palm OS Garnet. The current release of Palm OS Cobalt has support for Bluetooth and Wi-Fi. The Palm OS Garnet is based on the Palm 5.2 and 5.3 operating systems.
System development kits (SDKs) are available for the Palm OS, and C, Visual Basic, C ++, and Java can be used for development of applications. A number of commercial development suites are also available, which provide processor-specific applications and development tools.
PalmSource has announced support of a Palm OS platform on Linux.
12.4.4 Windows Mobile
Windows Mobile is based on the Windows CE operating system. This is a 32-bit operating system with 4 Gbytes of directly addressable memory space. The environment is complete with the .“NET Compact Framework” (.NET CF) run-time and programming framework. Many users and developers have a clear preference for Windows Mobile owing to its commonality with the desktop Windows system and support for desktop applications and file formats as a mobile extension to the office applications. In particular smart phones and PDAs with Windows Mobile OS that are used as “mobile offices” are prime candidates for the Windows Mobile operating system, as they can use the Windows Mobile Office application, which integrates seamlessly with the desktop and office applications (including Microsoft Exchange server and SQL server). Typically these have a QWERTY keyboard (or a touchscreen input) and a large display of QVGA or sometimes VGA resolution. The phones support e-mail and Microsoft Office applications among others. Windows Mobile comes in two versions—Windows Mobile for Pocket PC and Windows Mobile for Smartphones. Microsoft has released Windows Mobile 5.0 for smart phones. Windows Mobile Pocket PC version is designed for devices with QWERTY keyboards or touchscreen for input, while the Smartphone version is for normal phones with numeric keypads.
Apart from Microsoft Mobile Office there are many third party applications available for Windows Mobile-based devices (in particular the pocket PC devices). Windows Mobile is available on Palm Treos, Motorola Q, Samsung i730, or Samsung SGH-300 as well as a host of other devices.
The development environment for Windows mobile devices is provided by the .NET CF, which consists of:
• common language run-time (CLR) designed for mobile devices that are memory constrained as well as resource constrained and
• .NET CF classes for support of managed applications.
The CLR is much smaller than a typical desktop CLR. The development tool for the platform is Visual Studio .NET and Microsoft-provided SDK. The programming is usually done in Visual Basic or C++.
A number of operators have come out with Windows 5.0-based mobile phones with live TV and multimedia capabilities. Modeo has come out with a mobile phone for its L-band DVB-H network, which is based on Windows Mobile 5.0. The phone is quadband GSM (850/900/1800/1900). It is based on an OMAP 850 applications processor, Microtune MT2260 tuner, DiBcom DVB-H demodulator, and NVIDIA GoForce-5500 graphics processor. Multimedia capabilities are supported by Windows Media Mobile, Windows Media Audio Mobile, MP3, and AAC player. Third party applications such as RealVideo and RealAudio are also supported. The OMAP applications and graphics processors give the phones a high level of multimedia capability that can be used by the applications. This includes live TV, applications with animation, and 3D graphics and games (Fig. 12-6).
Cingular in the United States has introduced the Cingular 2125 smart phone, which features a Windows Mobile 5.0 operating system. The phone is based on TI’s OMAP 850 processor and has 64 Mbytes of RAM and ROM, which can be extended with an extension card slot. The phone has Windows Media Player 10 Mobile and in addition supports MP3, AAC, and .wav music and MPEG-4 video. T-Mobile and Sprint in the United States have also unveiled Windows Mobile phones for use with their networks.
FIGURE 12-6 Visualization of a Windows Mobile 5.0 Phone from Modeo for DVB-H
The outlook for Windows Mobile 5.0 continues to be bright despite its “non-real-time” OS and its nonphone origin. This is owing to the extremely large base of installed Microsoft Windows machines and servers and wide base of applications.
12.4.5 Other Operating Systems
BREW (Binary Runtime Environment for Wireless) is a proprietary platform developed by Qualcomm.
BREW is used extensively in the United States, Korea, and India as well as a host of other countries and is available for the GSM, 3G-GSM, and CDMA networks (BREW is air-interface independent). The BREW environment, though proprietary, is very rich in the form of downloadable applications and the ease with which the mobile phone client can request them from the server for OTA download. It has rich support for multimedia and office applications such as mail. It has powerful user interface tools (UiOne), which enable developers and phone manufacturers to modify or personalize the interfaces. The applications can be put into various categories:
• BREW applications that have been developed and are available to any manufacturer as fully developed mobile phone services,
• user applications that can be developed by using the BREW environment and development tools, and
• applications that can be downloaded.
BREW applications include push-to-chat, e-mail, photo sharing, etc. (Fig. 12-7).
BREW can be hosted on various processors such as ARM or MSM6100 (Qualcomm Mobile Station Modem) and provides powerful support to multimedia applications. The following features are supported, for example, by BREW 2.1:
• MPEG-4 encoding and playback;
• camera interface, JPEG compression, and recording of video;
• 3D graphics engine;
• mobile-assisted position location;
• connection and content security through HTTPS, SSL, and encryption;
• media support for various video and audio formats;
• streaming and playback of PCM and QCP media formats;
• serial, USB interface, and removable storage;
• battery management; and
• messaging services.
Major Operators Using BREW Worldwide
A number of extensions to the BREW operating platform are available, for example, Microsoft games (MSN games) on the BREW platform, J2ME on BREW, Microsoft “Live Anywhere,” Microsoft MSN messenger, and Microsoft Mobile Office (Table 12-2).
12.5 MIDDLEWARE IN MOBILE PHONES
The term middleware in the mobile software terminology denotes software that carries out a specific well-defined function and is built on the OS stacks (Fig. 12-2). Commonly used functions provided by middleware include implementations of various types of codecs, communications, and protocols used in 3G and broadcast networks. The services that a middleware provides can be further used by the application software. As an example, a multimedia phone can be defined to have phone service as an application and a multimedia service as another application.
An example of mobile middleware is Packetvideo’s pvTV solution for DVB-H. pvTV provides a complete platform for video, enabling mobile phones based on various operating systems (Linux, Symbian, Windows Mobile). The pvTV middleware provides complete stacks and codecs for mobile TV, including H.264, MPEG-4, WMV, AAC+, and WMA. It also provides support for Microsoft DRM and OMA conformance. The pvTV client can enable various phone manufacturers with multiple OS types to have a quickly configurable mobile TV solution with full compliance with standards.
Another example of middleware can be had from the middleware for NTT DoCoMo’s FOMA network. FOMA is a 3G network in Japan and provides services such as i-mode (Internet access), i-appli (Internet applications), Deco-Mail (HTML e-mail), Chara-Den (videophone with cartoon type characters), and Chaku-motion (combining video and AAC audio to signal incoming calls). The services are network specific. The operators are naturally interested in the service operating identically on all phones irrespective of the operating system or the software structure. There are two methods to achieve this. The first is that the applications be written in a language, such as Java (Java Mobile Information Device Profile (MIDP) 2.0 and J2MEE), that is independent of the operating platform. The second is to have specific middleware to support all the services that are network specific (e.g., all FOMA services). Initially the FOMA phones were released with Java-based applications and having Symbian OS, which provides very strong native support toward realizing application stacks such as those for H.264 coding, H.263 video calls, AAC audio coding, various players and display drivers of Java-based applications, and games.
Now NTT DoCoMo has announced the use of middleware for its services such as Renesas Technologies’ “Mobile Videophone Middleware Package”. The middleware will provide the entire 3G-324M video call package, including encoding and decoding of video and audio, echo cancellation, and videophone protocols. The middleware is ported on the SH-Mobile applications processors. The new architecture is available for the FOMA series of handsets. Using the middleware makes it easier to port the applications to other platforms such as Linux. Similarly, NTT DoCoMo licensed embedded “push-to-talk over cellular” software from Ecrio, Inc., for use on mobile phones. The new service is available on the FOMA902i series of handsets, which also have Linux implementations. When Vodafone launched its services in Greece in 2003, is service suite included the Vodafone Live! messaging service, a high-volume MMS service. The service was launched with middleware from First Hop Wireless Broker suite, a company that provides middleware suites for 3GPP-compliant IMS services among other products.
12.5.1 How Does Mobile Middleware Provide Revenue Enhancement Opportunities?
Mobile middleware is the key to the delivery of customizable service to users, which can help in taking the use of the phone much beyond voice and provide revenue enhancement.
The success of services such as SMS and MMS is not hidden from anyone. In fact it has taken the world of mobile to the next generation of kids and teens. We are now at the dawn of new era in multimedia in which video, audio, animation and games, information, and services can be delivered at an attractive price to the user. Mobile phones have broken the barrier of pricing for multimedia handsets to a level at which critical mass has already been breached in many countries. The success of FOMA has exhibited the importance of rich content being delivered over 3G and other networks. The availability of digital rights management solutions means that piracy and the undesirable distribution of content can be restricted to a greater extent. Mobile networks with their built-in payment mechanisms are enablers for realization of revenues from the customers.
Middleware enables the networks to deliver new services and download and use applications that may not have been envisaged when a mobile phone or a network commenced services. The use of middleware makes it possible to transport applications to new networks with a variety of phones already in use without major efforts in reporting to different operating systems, phones, or networks.
12.5.2 Examples of Mobile Middleware Platforms
In addition to the MobiTV service Cingular has launched an on-demand video service. The on-demand service permits the users to view or download premium content. Premium channels (such as HBO Mobile) can be subscribed to on a monthly rental basis, while music and video clips can be viewed on a pay-per-view basis.
Cingular has used RealNetwork’s Helix Online TV platform to deliver the services. However, the unique thing about the network is that the platform is both a content aggregation and a distribution platform, both for retail and for wholesale customers. Thousands of content providers can thus ingest content and indicate prices per view, and the users can have access to the content. This is done by using the Qpass M-commerce solution. Qpass also gives Cingular control over the way the content is priced, displayed to customers, distributed, and billed. As mobile networks present a big opportunity for revenues from video content, the RealHelix Online TV platform together with Qpass commerce solution is set to provide a major advantage to users as well as content providers and operators. Cingular operates in multiple countries through 3G and EDGE networks and customers can roam and still have seamless access.
The service configuration and delivery of Cingular video has been possible through the use of the Helix mobile server, which interacts with the Helix DNA client. Digital rights management is assured through Helix digital rights management and the gateway to mobile networks is Helix Mobile Gateway. Content ingest and delivery are handled by the Helix mobile producer and Helix service delivery platform.
The Helix DNA client performs a number of functions such as:
• auto bandwidth detection,
• Playnow (near TV-like playback experience),
• Truelive—live playback of TV stream (if necessary without error protection),
• Trickplay (playback at various speeds, fast forward, reverse, DVR-type functions),
• visual progressive download—the viewer can see the progress of the download (bytes transferred and transfer rate), and
• 3GPP release 6 compliance.
12.5.3 3D Graphics and Mobile Multimedia Middleware MascotCapsule Engine
Hi Corporation Japan has supplied MascotCapsule multimedia middleware with a 3D rendering engine for operators in Japan, Korea, the United States, China, and Europe. The 3D rendering provides unique capabilities for multimedia and games. The MascotCapsule enables 3D applications to run in a mobile phone (and other devices such as PDAs) independent of the operating system or hardware architecture. All three Japanese carriers have adapted the use of this middleware in addition to carriers in other countries (SK Telecom, Korea; China Unicom; Sprint, USA; etc.) and the global phone manufacturers Motorola and Sony Ericsson. The middleware can operate with Java, BREW, Linux, Symbian, or Palm OS. It has a 3D development tool, which interfaces with 3D software packages such as 3DSmax, Lightwave, and Maya (Fig. 12-8).
FIGURE 12-8 3D Graphics and Animation Middleware (MascotCapsule)
There are hundreds of MascotCapsule-based games and applications that have been developed and are available on networks. One of the extensions to the MascotCapsule is the MascotCapsuleFace edition, which permits features of the face such as eyes, eyebrows, and lips (through a JPEG or GIF image) to be animated and used in applications like e-mail or MMS. The “talking head,” which resembles the picture of the person at the other end can be used with text-to-voice converter.
12.6 APPLICATIONS SOFTWARE FUNCTIONALITIES FOR MOBILE MULTIMEDIA
The foregoing sections have shown that the applications software on a multimedia handset needs to provide a rich and intuitive user experience. Typically the applications software will provide the end functionality through the use of the services provided by the OS and middleware. Moreover, from a user’s point of view, it is only the applications that are important, and the user needs to be isolated from the hardware and operating system of the handset as well as the underlying network characteristics.
As multimedia services on most networks are in their infancy, there are major changes occurring in the service delivery architectures, M-commerce, billing, etc. These functions are implemented by new applications software, downloaded over the air, and delivered integrated seamlessly into the operating environment of the mobile phone irrespective of the OS and the underlying hardware.
12.6.1 Macromedia Flash Lite
Macromedia Flash has been widely used in the design of Web sites and other applications with rich animations. It is easy to program and does not require large bandwidths for transmission. Flash animation movies, for example, can run on a fraction of bandwidth needed to code every frame using MPEG 2 compression.
Macromedia Flash Lite has been tailored to meet the resources available on mobile phones in terms of screen size, pixels, colors, and network resources available. Consequently it was only natural for Flash to be used to generate application and animation channels or clips for the mobile content as well. This was indeed the case, and the FOMA network has been using Flash applications extensively.
The Flash content is run on the phones using the client server architecture. The client is provided in the phones and interfaces with the phone software and the operating system. The Flashcast server delivers the content when accessed using the HTTP or HTTPS protocols (Fig. 12-9).
12.6.2 Java for Mobile Devices
Java 2 Microedition (J2ME) is a powerful environment for development of applications for mobile devices such as cell phones or PDAs. Applications written with J2ME are portable and provide quick development cycles for new products such as games, animated clips, etc. J2ME is supported by the Sun Wireless Development Toolkit and Java Netbeans mobility pack for authoring of Java applications for mobile devices.
FIGURE 12-9 Flashcast Delivery of Content
Java used on mobile phones is described by the MIDP for the Connected Limited Device Configuration version 1.1 (CLDC 1.1). The MIDP device profiles provide for the limited environment of mobile phones or PDAs with small memory footprints, and the programs are leaner, needing fewer resources. The CLDC 1.1 configuration is defined under the Java Community Process and defines the types of devices on which such applications can run uniformly. The CLDC target devices are mobile phones with limited memory of 160 to 512 Kbytes. Further mobile phones operate in an environment in which the signals may be lost from time to time due to transmission conditions. Applications based on protocols such as WAP require constant connectivity between the server and the application. On the other hand the Java applications can run on devices that can be disconnected from the network. The CLDC configuration is indeed defined for such devices.
FIGURE 12-10 Java Applications for Mobile Phones
There are two steps to the execution of applications using Java on mobile phones:
• delivery of Java files using the operator’s WAP gateway or Web server and
• execution of files on the mobile phones.
The phones need to have Java support in the software suite for the Java applications to be executed. Java-based application support is today present in a majority of phones (Fig. 12-10).
A number of applications are delivered using Java support in the handsets. For example, Sprint in the United States uses Java extensively for applications in its CDMA network.
The Symbian operating system (e.g., Symbian 40 third edition and others) provides native support for Java MIDP 2.0.
12.7 INTEGRATING MOBILE OFFICE WITH MULTIMEDIA AND TV
Though in its early days, a clear trend is emerging toward the integration of multimedia functionalities in phones that are predominantly smart phones and PDAs with mobile office functions and vice versa. This means that one would expect to find multimedia phones such as the Walkman phone to support applications for Internet, mail, and document viewing and processing. These are made possible through the use of powerful processors and increased memory capabilities that are commonly incorporated in the phones.