About the Reviewers
by Jon Lentz, Brian Femiano, Jonathan R. Owens
Hadoop Real-World Solutions Cookbook
- Hadoop Real-World Solutions Cookbook
- Table of Contents
- Hadoop Real-World Solutions Cookbook
- Credits
- About the Authors
- About the Reviewers
- www.packtpub.com
- Preface
- 1. Hadoop Distributed File System – Importing and Exporting Data
- Introduction
- Importing and exporting data into HDFS using Hadoop shell commands
- Moving data efficiently between clusters using Distributed Copy
- Importing data from MySQL into HDFS using Sqoop
- Exporting data from HDFS into MySQL using Sqoop
- Configuring Sqoop for Microsoft SQL Server
- Exporting data from HDFS into MongoDB
- Importing data from MongoDB into HDFS
- Exporting data from HDFS into MongoDB using Pig
- Using HDFS in a Greenplum external table
- Using Flume to load data into HDFS
- 2. HDFS
- Introduction
- Reading and writing data to HDFS
- Compressing data using LZO
- Reading and writing data to SequenceFiles
- Using Apache Avro to serialize data
- Using Apache Thrift to serialize data
- Using Protocol Buffers to serialize data
- Setting the replication factor for HDFS
- Setting the block size for HDFS
- 3. Extracting and Transforming Data
- Introduction
- Transforming Apache logs into TSV format using MapReduce
- Using Apache Pig to filter bot traffic from web server logs
- Using Apache Pig to sort web server log data by timestamp
- Using Apache Pig to sessionize web server log data
- Using Python to extend Apache Pig functionality
- Using MapReduce and secondary sort to calculate page views
- Using Hive and Python to clean and transform geographical event data
- Using Python and Hadoop Streaming to perform a time series analytic
- Using MultipleOutputs in MapReduce to name output files
- Creating custom Hadoop Writable and InputFormat to read geographical event data
- 4. Performing Common Tasks Using Hive, Pig, and MapReduce
- Introduction
- Using Hive to map an external table over weblog data in HDFS
- Using Hive to dynamically create tables from the results of a weblog query
- Using the Hive string UDFs to concatenate fields in weblog data
- Using Hive to intersect weblog IPs and determine the country
- Generating n-grams over news archives using MapReduce
- Using the distributed cache in MapReduce to find lines that contain matching keywords over news archives
- Using Pig to load a table and perform a SELECT operation with GROUP BY
- 5. Advanced Joins
- Introduction
- Joining data in the Mapper using MapReduce
- Joining data using Apache Pig replicated join
- Joining sorted data using Apache Pig merge join
- Joining skewed data using Apache Pig skewed join
- Using a map-side join in Apache Hive to analyze geographical events
- Using optimized full outer joins in Apache Hive to analyze geographical events
- Joining data using an external key-value store (Redis)
- 6. Big Data Analysis
- Introduction
- Counting distinct IPs in weblog data using MapReduce and Combiners
- Using Hive date UDFs to transform and sort event dates from geographic event data
- Using Hive to build a per-month report of fatalities over geographic event data
- Implementing a custom UDF in Hive to help validate source reliability over geographic event data
- Marking the longest period of non-violence using Hive MAP/REDUCE operators and Python
- Calculating the cosine similarity of artists in the Audioscrobbler dataset using Pig
- Trim Outliers from the Audioscrobbler dataset using Pig and datafu
- 7. Advanced Big Data Analysis
- 8. Debugging
- Introduction
- Using Counters in a MapReduce job to track bad records
- Developing and testing MapReduce jobs with MRUnit
- Developing and testing MapReduce jobs running in local mode
- Enabling MapReduce jobs to skip bad records
- Using Counters in a streaming job
- Updating task status messages to display debugging information
- Using illustrate to debug Pig jobs
- 9. System Administration
- 10. Persistence Using Apache Accumulo
- Introduction
- Designing a row key to store geographic events in Accumulo
- Using MapReduce to bulk import geographic event data into Accumulo
- Setting a custom field constraint forinputting geographic event data in Accumulo
- Limiting query results using the regex filtering iterator
- Counting fatalities for different versions of the same key using SumCombiner
- Enforcing cell-level security on scans using Accumulo
- Aggregating sources in Accumulo using MapReduce
- Index
Edward J. Cody is an author, speaker, and industry expert in data warehousing, Oracle Business Intelligence, and Hyperion EPM implementations. He is the author and co-author respectively of two books with Packt Publishing, titled The Business Analyst's Guide to Oracle Hyperion Interactive Reporting 11 and The Oracle Hyperion Interactive Reporting 11 Expert Guide. He has consulted to both commercial and federal government clients throughout his career, and is currently managing large-scale EPM, BI, and data warehouse implementations.
I would like to commend the authors of this book for a job well done, and would like to thank Packt Publishing for the opportunity to assist in the editing of this publication.
Daniel Jue is a Sr. Software Engineer at Sotera Defense Solutions and a member of the Apache Software Foundation. He has worked in peace and conflict zones to showcase the hidden dynamics and anomalies in the underlying "Big Data", with clients such as ACSIM, DARPA, and various federal agencies. Daniel holds a B.S. in Computer Science from the University of Maryland, College Park, where he also specialized in Physics and Astronomy. His current interests include merging distributed artificial intelligence techniques with adaptive heterogeneous cloud computing.
I'd like to thank my beautiful wife Wendy, and my twin sons Christopher and Jonathan, for their love and patience while I research and review. I owe a great deal to Brian Femiano, Bruce Miller, and Jonathan Larson for allowing me to be exposed to many great ideas, points of view, and zealous inspiration.
Bruce Miller is a Senior Software Engineer for Sotera Defense Solutions, currently employed at DARPA, with most of his 10-year career focused on Big Data software development. His non-work interests include functional programming in languages like Haskell and Lisp dialects, and their application to real-world problems.
-
No Comment