10

Future Trends and Developments

Intelligence will drive the next generation of technologies, not big data. Big data systems have some issues, as discussed in Chapter 1, Challenges in Big Data and Traditional AI, and the world is gradually transitioning from the data-centric era to the intelligence-centric generation. Federated learning (FL) will play a core role in wisdom-driven technologies. Thus, the time is now to welcome the world of collective intelligence.

In this chapter, we will talk about the direction of future AI technologies that are driven by the paradigm shift happening with FL. For many AI fields, such as privacy-sensitive areas and fields requiring scalability in machine learning (ML), the benefits and potential of FL are already significant, mainly because of the privacy-preserving and distributed learning aspects that FL naturally supports with its design. You will then learn about the different types of FL as well as the latest development efforts in that area, as seen in the split and swarm learning techniques, which can be considered as evolutional frameworks enhancing FL.

In addition, FL creates a new concept of an Internet of Intelligence, where people and computers exchange their wisdom instead of just data themselves. The Internet of Intelligence for everyone is further accelerated by blockchain technologies as well. This Internet of Intelligence can then form a newly defined concept of collective intelligence that drives another innovation, from data-centric approaches to intelligence-centric or model-centric approaches.

Finally, we will share a collective vision in which FL plays a key role in collaboratively creating intelligence learned by many people and machines around the world.

In this chapter, we will cover the following topics:

  • Looking at future AI trends
  • Ongoing research and developments in FL
  • Journeying on to collective intelligence

Looking at future AI trends

The majority of industry leaders are now aware of the limitations of centralized ML as discussed in the next section.

The limitation of centralized ML

When looking at the future of AI, it is important to first know the fact that many companies today are struggling to extract intelligence and obtain insight from the data they possess. More than half of the data that organizations and companies have collected is usually not used. Traditional approaches to machine learning and data science need data to be organized and consolidated into data lakes and stores in advance of analyzing and training ML models. You need to duplicate and move the data, which will result in delays in realizing and delivering the value of the intelligence extracted from the data, together with certain operational risks and complexities.

In addition, most of the data generated by enterprise companies will be created and processed outside a traditional centralized data center or cloud. It is becoming increasingly unrealistic and inefficient to process data for generating insight in a centralized manner.

Furthermore, based on some market reports out there, most of the largest global organizations and companies will explore FL at least once to create much more accurate, secure, and sustainable models environmentally.

That being said, quite a few industries and markets are gradually becoming aware of the importance of a distributed and FL paradigm, because they are facing the unavoidable issues and limitations of the current centralized AI training with big data, as discussed in Chapter 1, Challenges in Big Data and Traditional AI. FL brings the model to the data where the training process resides instead of bringing the data to the model. Thus, FL is considered to be the future of data science and ML.

In the next section, let’s summarize the points of why FL is beneficial to those companies, especially enterprises that have been facing the aforementioned issues.

Revisiting the benefits of FL

In this section, we will summarize the benefits of FL that have been introduced throughout this book.

Increased model accuracy and generalizability

FL realizes collaborative and distributed learning that can improve the performance of ML models, by training on dispersed datasets locally to continuously incorporate the learning into a global model. This way, more accurate and generalized ML models can be produced.

Further privacy and security

FL provides privacy and security advantages because it won’t require private and raw data by its design and security mechanisms, as we discussed previously in Chapter 2, What Is Federated Learning? and Chapter 9, Case Studies with Key Use Cases of Federated Learning Applications. Thus, FL reduces the potential risk of data misuse, leakage, or exposure to sensitive information. FL is also compliant with many privacy regulations, such as General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and Health Insurance Portability and Accountability Act (HIPAA).

Improved speed and efficiency

FL is also known to realize high computation efficiency, which can accelerate the deployment and testing of ML models as well as decrease communication and computational latency. Due to the decentralized nature of FL, the delay for model delivery and update is minimized, which leads to a prediction by the global model in near real time. Real-time delivery and updates of intelligence are really valuable for time-sensitive ML applications.

FL also helps reduce bandwidth and energy consumption by overcoming system heterogeneity and unbalanced data distribution, which leads to minimizing data storage and transfer costs that can also significantly contribute to reducing the environmental impact.

Toward distributed learning for further privacy and training efficiency

Currently, AI is trained on huge computational servers, usually happening on big machines in big data companies.

As seen in the era of the supercomputer, which can process a huge amount of data and tasks within one machine or one cluster of machines, the evolutional process in technology starts from a central location and gradually transitions to distributed environments.

The same thing is exactly about to happen in AI. Now, the data lake concept is popular to organize and train ML models in one place, but ML already requires distributed learning frameworks.

FL is a great way to distribute a training process over multiple nodes. As shown in many research reports, most data is not fully used to extract insights into ML models.

There are some companies and projects that are trying to use FL as a powerful distributed learning technique, such as the platforms provided by Devron (devron.ai), FedML (fedml.ai), and STADLE (stadle.ai). These platforms are already resolving the issues discussed in The limitation of centralized AI section and have shown a drastic improvement in the ML process in various use cases, as stated in the Revisiting the benefits of FL section.

Based on the AI trends that we have discussed, let’s look into the ongoing research and developments related to FL that cutting-edge companies are conducting now in the next section.

Ongoing research and developments in FL

We now talk about the ongoing research and development projects that are being taken place both in academia and industries around the world. Let’s start with the different types and approaches of FL, and move on to ongoing efforts to further enhance the FL framework.

Exploring various FL types and approaches

In this book, we have visited the most basic algorithms and design concepts of an FL system. In the real world, we need to dig a bit deeper into what types of FL frameworks are available to extract the best performance out of those algorithms. Depending on the data scenario and use cases, we have several approaches in FL, as follows:

  • Horizontal FL and vertical FL
  • Centralized FL and decentralized FL
  • Cross-silo FL and cross-device FL

Now, let’s look at each type of FL in the following sections.

Horizontal FL and vertical FL

Horizontal FL uses datasets with the same feature space or schema across all distributed devices (https://www.arxiv-vanity.com/papers/1902.04885/). This actually means that datasets share the same columns with different rows. Most existing FL projects are based on horizontal FL. Datasets and training processes with horizontal FL are straightforward because the datasets are formed identically, with different data distributions and inputs to be learned. Horizontal FL is also called homogeneous or sample-based FL.

Vertical FL is applied to the cases where different datasets share the same sample ID space but differ in feature space. You can check out this paper (https://arxiv.org/pdf/2202.04309) for further information about vertical FL. Relating these different databases through FL can be challenging, especially if the unique ID for the data is different. The key idea of vertical FL is to improve an ML model by using distributed datasets with a diverse set of attributes. Therefore, vertical FL can handle the partitioned data vertically with different attributes in the same sample space. Vertical FL is also called heterogeneous or feature-based FL.

Centralized FL and decentralized FL

Centralized FL is currently the most common approach and most of the platforms employ this framework. It uses a centralized server to collect and aggregate the different ML models, with distributed training across all local data sources. In this book, we focused on a centralized FL approach, with a scenario where local training agents communicate the learning results to a centralized FL server to create a global model.

Decentralized FL, on the other hand, does not use a centralized server to aggregate ML models. It requires individual ML models trained over local data sources to be communicated among themselves without a master node. In this case, model weights are transferred from each individual dataset to the others for further training. It could potentially be susceptible to model poisoning if an untrusted party could access the intelligence, and this is a common problem derived from peer-to-peer frameworks as well.

Cross-silo FL and cross-device FL

Cross-silo FL is the case where ML models are trained on data distributed across any functional, organizational, and regulatory barriers. In this case, big data is usually stored in a larger size of storage, with training computing capabilities such as cloud virtual machines. In the cross-silo FL case, the number of silos/training environments is relatively small, so not so many agents are needed in the FL process.

Cross-device FL is the case where models need to be trained at scale, often within edge devices, such as mobile phones, Internet of Things (IoT) devices, Raspberry Pi-type environments, and so on. In this case, a huge number of devices are connected for the aggregation of ML models. In the cross-device FL case, the limitation basically lies in the low computing power of those edge devices. The framework also needs to handle a number of disconnected and inactive devices to conduct a consistent and continuous FL process. The training process and its data volume should be limited too.

That concludes the different types of FL that can be applied to a variety of scenarios in ML applications. There are new techniques that try to enhance the FL framework to evolve into the next generation of AI technologies with FL. Let’s look into several advanced approaches in the next section.

Understanding enhanced distributed learning frameworks with FL

There are ongoing efforts to further enhance FL or distributed learning frameworks.

Split learning

Split learning, developed in the MIT Media Lab, is an emerging distributed learning technique that enables partitioning ML models into multiple sections, trains those partitioned ML models at distributed clients, and aggregates them at the end. Split learning does not have to share the data either, so it is considered a privacy-preserving AI as well.

The overall framework is similar to the FL. However, there is a difference in that the neural network is partitioned into multiple sections that will be trained on distributed clients. The trained weights of the section of the neural network are then transferred to the server and clients. The weights of those multiple sections are continuously trained in the next training sessions. Therefore, no raw and private data is shared among the distributed clients, and only the weights of each section are sent to the next client.

Especially, SplitFed (https://arxiv.org/abs/2004.12088) is another advanced technique that combines split learning and FL. SplitFed splits the deep neural network architecture between the FL clients and servers to realize a higher level of privacy than FL. It offers better efficiency than split learning based on the parallel learning paradigm of FL.

Swarm learning

Swarm learning is a decentralized ML solution built on blockchain technology, particularly designed to enable enterprise industries to take advantage of the power of distributed data, which results in protecting data privacy and security.

This can be achieved by individual nodes sharing parameters of ML models derived from the local data.

Parameters shared from the distributed clients are merged into a global model. The difference from the normal FL is that the merge process is not performed by a central server. The distributed nodes and clients choose a temporary leader to perform the merge. That is why swarm learning is truly decentralized, also providing greater fault tolerance and resiliency. The distributed agents have the collective intelligence of a network without sharing local data into one node.

Swarm learning builds on top of blockchain. Blockchain provides the decentralized control, scalability, and fault-tolerance aspects to work beyond the restrictions of a single enterprise. At the same time, blockchain introduces a tamperproof cryptocurrency framework, and the participants can use the framework to monetize their contributions.

BAFFLE

In addition, there is a framework called BAFFLE that stands for Blockchain Based Aggregator Free Federated Learning (https://arxiv.org/abs/1909.07452). BAFFLE is also an aggregator-free, blockchain-driven FL framework that is inherently decentralized. BAFFLE utilizes Smart Contracts (SCs) from the blockchain framework to coordinate round management, as well as model aggregation and updating tasks of FL. Using BAFFLE boosts computational performance. The global model is also decomposed into many sets of chunks, directly handled by the SC.

Now that we have learned about the latest research and developments in the FL field, in the next section, let’s look at a more visionary aspect of the AI, science, and technologies of collective intelligence.

Journeying on to collective intelligence

Big data has been a game changer for the AI movement. While the amount of data generated at the edge and by people will increase exponentially, intelligence derived from that data benefits society. Therefore, the big data era will gradually pass the baton to the collective intelligence era, empowered by FL, in which people will collaboratively create a wisdom-driven world.

Let’s start by defining an intelligence-centric era where the concept of collective intelligence is realized based on FL.

Intelligence-centric era with collective intelligence

Collective Intelligence (CI) is the concept of a large group of single entities acting together in ways that seem intelligent. CI is an emergent phenomenon where groups of people process information to achieve insights that are not understandable by just individual members alone.

Recently, Thomas Malone, the head of the MIT Center for Collective Intelligence, and the person who initially coined the phrase collective intelligence, broadened the definition of CI: “CI is something that can emerge from a group that includes people and computers. CI is a very general property, and superminds can arise in many kinds of systems, although the systems I’ve mostly talked about are those that involve people and computers” (Reference: https://www2.deloitte.com/xe/en/insights/focus/technology-and-the-future-of-work/human-and-machine-collaboration.html).

We are now welcoming the new perspective of CI in technologies empowered by FL.

Data, in the current world of technology, is a great source to extract intelligence. Dispersed datasets around the world can be converted into a collection of intelligence represented by AI technologies. The current trend, as mentioned, is big data, so big data companies are leading not only the technology industries but also the entire economy of the world as well. The future is moving in a CI direction. The vision of CI is even clearer with the emergence of sophisticated ML algorithms, including deep learning, as the intelligence represented by ML models can extract intelligence from people, computers, or any devices that generate meaningful data.

Why does FL promote the idea of CI? The nature of FL is to collect a set of distributed intelligence to be enhanced by an aggregating mechanism as discussed in this book. This itself enables a data-less platform that does not require collecting data from people or devices directly.

With the big data issues discussed throughout the book, we have steered clear of focusing on data-centric platform. However, it is also true that learning big data is very much critical and inevitable to really create systems and applications that are truly valuable and deliver real value in many domains of the world. That is why the big data field is still the most prosperous industry, even if it is facing significant challenges represented by privacy regulations, security, data silos, and so on.

Now is the time to further develop and disseminate the technologies such as FL that can accelerate the era of CI by fundamentally resolving the issues of big data. This way, we can realize a new era of technologies, truly driven by CI that has been backed up by an authentic mathematical basis.

As mentioned, data-centric platforms are the current trend. So many data and auto ML vendors can support and automate the processes of creating ML-based intelligence by organizing data and learning procedures to do so. An intelligence-centric or model-centric platform should be the next wave of technology in which people can share and enhance intelligence that they generate on their own. With FL, we can even realize crowd-sourced learning, where people can collaboratively and continuously enhance the quality and performance of ML models. Thus, FL is a critical and essential part of the intelligence-centric platform to truly achieve a wisdom-driven world.

Internet of Intelligence

The IoT evolved into the Internet of Everything. However, what is the essential information that people want? Is it just big data? Or intelligence derived from data? With 5G technologies, a lot of data can be transferred over the Internet at a much higher speed, partially resolving the latency issues in many AI applications. FL can exchange less information than raw data but still needs to transfer ML models over the Internet.

While lots of research projects are minimizing communications latency in FL, in the future, information related to intelligence will be another entity often exchanged over the web. There will be a model repository such as Model Zoo everywhere, and crowdsourced learning empowered by FL will be more common to create better intelligence over the Internet with people worldwide collaboratively.

This paradigm shift is not just in the AI field itself but also in the wide range of information technologies. As we’ll discuss in the next sections, this Internet of Intelligence movement will be the basis of crowdsourced learning and CI, and will help make intelligence available to as many people as possible in the coming years.

Crowdsourced learning with FL

The collection of intelligence performed by FL naturally makes it a strong fit for moving toward CI. The same thing is applied to a scenario where people can collectively contribute a training process to global ML models.

High-performing ML models in areas such as computer vision and natural language processing have been trained by certain big data companies, often spending a huge amount of money, including hundreds of millions of dollars.

Is there any way to collectively train an ML model that will probably be beneficial for a wide range of people in general? With the advanced framework of FL, that is possible.

FL provides an authentic way to manage the aggregation of multiple trained models from various distributed agents. In this case, the distributed agents themselves may be people worldwide, where each individual user and trainer of the ML model has their own unique datasets that are not available to anybody else because of data privacy, silos, and so on.

This way of utilizing CI is often called crowdsourced learning. However, traditional crowdsourced learning is conducted in a much more limited way, just based on facilitating and recruiting data annotators at a large scale.

With this new paradigm with FL, users on the CI platform can access and download ML models that they are interested in and retrain them if necessary to absorb learning in their own environments. Then, with the framework to share the trained ML models by those users, an advanced aggregation framework of FL could pick up the appropriate models to be federated and make the global model perform better, adopting diverse data that can be only accessible to the users.

This way, intelligence by ML is becoming more available to many individuals in general, not just to specific companies that have a significant amount of data and budgets to train an authentic ML model. In other words, without an FL framework, collaborative learning is difficult and tricky and almost impossible to even automate. This openness of the ML models will move the entire technological world to the next level, and a lot more applications will become feasible, with truly powerful intelligence that is trained by enthusiasts to make the world better.

Summary

In this final chapter of the book, we discussed fascinating future trends and developments in which FL is expected to play a crucial role in the coming decade. In the future, FL is a must-to-have technology from a nice-to-have framework for most enterprises and application providers, because of the inevitable privacy regulations and technology trends requiring scalability with so many users.

As we discussed, future technologies will be empowered by the concept of the Internet of Intelligence, by which people and computers mainly exchange their wisdom altogether to create a more intelligent society and world. Finally, the data-centric technologies will gradually evolve into intelligence-centric technologies because of the current collaborative learning trend with CI, which makes people pay significant attention to FL-related technologies, whose foundations are discussed throughout this book.

This book was written at the dawn of a new age in advancements made possible by AI. There are many uncertainties and many more challenges ahead. We have made great strides in utilizing the big data playbook in the last couple of decades, and we have now outgrown those methods and must adopt new ways of doing things, new technologies, and new ideas to forge ahead. As long as we capture the current moment and invest in new technologies such as FL, we will have a bright future ahead of us.

Further reading

The following are some sources if you wish to dive deeper into some concepts discussed in this chapter:

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.145.17.46