In this chapter, you learned about big data architecture and components for big data pipeline design. You learned about data ingestion and various technology choices available to collect batch and stream data for processing. As the cloud is taking a central place in storing the vast amounts of data being produced today, so you learned about the various services available to ingest data in the AWS cloud ecosystem.

Data storage is one of the central points when it comes to handling big data. You learned about various kinds of data stores, including structured and unstructured data, NoSQL, and data warehousing, with the relevant technology choices associated with each. You learned about data lake architecture and benefits.

Once you collect and store data, you need to perform data transformation to get insight into that data and visualize your business requirements. You learned about data processing architecture along with technology choices to choose open source and cloud-based data processing tools as per your data requirements. These tools help to get data insight and visualization as per the nature of your data and organizational requirements.

Now, there are millions of small devices connected to the internet, referred to collectively as the IoT. You learned about the various components available in the cloud to collect, process, and analyze IoT data to produce meaningful insights. You learned about the ML workflow, which includes data preprocessing, modeling, evaluation, and prediction. Also, you learned about supervised and unsupervised learning with an overview of various popular ML algorithms and available ML frameworks. 

With time, organizations tend to accumulate the technology debt, and many legacy applications are sitting in the data center, creating costs and consuming resources. In the next chapter, you will learn about legacy application transformation and modernization. You will learn about challenges with legacy systems and the techniques used to modernize them.