2. Reading and Writing Files
by María Carina Roldán, Adrián Sergio Pulvirenti
Pentaho Data Integration 4 Cookbook
- Pentaho Data Integration 4 Cookbook
- Pentaho Data Integration 4 Cookbook
- Credits
- About the Authors
- About the Reviewers
- www.PacktPub.com
- Preface
- 1. Working with Databases
- Introduction
- Connecting to a database
- Getting data from a database
- Getting data from a database by providing parameters
- Getting data from a database by running a query built at runtime
- Inserting or updating rows in a table
- Inserting new rows where a simple primary key has to be generated
- Inserting new rows where the primary key has to be generated based on stored values
- Deleting data from a table
- Creating or altering a database table from PDI (design time)
- Creating or altering a database table from PDI (runtime)
- Inserting, deleting, or updating a table depending on a field
- Changing the database connection at runtime
- Loading a parent-child table
- 2. Reading and Writing Files
- Introduction
- Reading a simple file
- Reading several files at the same time
- Reading unstructured files
- Reading files having one field by row
- Reading files with some fields occupying two or more rows
- Writing a simple file
- Writing an unstructured file
- Providing the name of a file (for reading or writing) dynamically
- Using the name of a file (or part of it) as a field
- Reading an Excel file
- Getting the value of specific cells in an Excel file
- Writing an Excel file with several sheets
- Writing an Excel file with a dynamic number of sheets
- 3. Manipulating XML Structures
- Introduction
- Reading simple XML files
- Specifying fields by using XPath notation
- Validating well-formed XML files
- Validating an XML file against DTD definitions
- Validating an XML file against an XSD schema
- Generating a simple XML document
- Generating complex XML structures
- Generating an HTML page using XML and XSL transformations
- 4. File Management
- 5. Looking for Data
- Introduction
- Looking for values in a database table
- Looking for values in a database (with complex conditions or multiple tables involved)
- Looking for values in a database with extreme flexibility
- Looking for values in a variety of sources
- Looking for values by proximity
- Looking for values consuming a web service
- Looking for values over an intranet or Internet
- 6. Understanding Data Flows
- Introduction
- Splitting a stream into two or more streams based on a condition
- Merging rows of two streams with the same or different structures
- Comparing two streams and generating differences
- Generating all possible pairs formed from two datasets
- Joining two or more streams based on given conditions
- Interspersing new rows between existent rows
- Executing steps even when your stream is empty
- Processing rows differently based on the row number
- 7. Executing and Reusing Jobs and Transformations
- Introduction
- Executing a job or a transformation by setting static arguments and parameters
- Executing a job or a transformation from a job by setting arguments and parameters dynamically
- Executing a job or a transformation whose name is determined at runtime
- Executing part of a job once for every row in a dataset
- Executing part of a job several times until a condition is true
- Creating a process flow
- Moving part of a transformation to a subtransformation
- 8. Integrating Kettle and the Pentaho Suite
- Introduction
- Creating a Pentaho report with data coming from PDI
- Configuring the Pentaho BI Server for running PDI jobs and transformations
- Executing a PDI transformation as part of a Pentaho process
- Executing a PDI job from the Pentaho User Console
- Generating files from the PUC with PDI and the CDA plugin
- Populating a CDF dashboard with data coming from a PDI transformation
- 9. Getting the Most Out of Kettle
- Introduction
- Sending e-mails with attached files
- Generating a custom log file
- Programming custom functionality
- Generating sample data for testing purposes
- Working with Json files
- Getting information about transformations and jobs (file-based)
- Getting information about transformations and jobs (repository-based)
- A. Data Structures
In this chapter, we will cover:
- Reading a simple file
- Reading several files at the same time
- Reading unstructured files
- Reading files having one field by row
- Reading files having some fields occupying two or more rows
- Writing a simple file
- Writing an unstructured file
- Providing the name of a file (for reading or writing) dynamically
- Using the name of a file (or part of it) as a field
- Reading an Excel file
- Getting the value of specific cells in an Excel file
- Writing an Excel file with several sheets
- Writing an Excel file with a dynamic number of sheets
Files are the most primitive, but also the most used format to store and interchange data. PDI has the ability to read data from all kind of files and different formats. It also allows you to write back to files in different formats as well.
Reading and writing simple files is a very straightforward task. There are several steps under the input and output categories of steps that allow you to do it. You pick the step, configure it quickly, and you are done. However, when the files you have to read or create are not simple—and that happens most of the time—the task of reading or writing can become a tedious exercise if you don't know the tricks. In this chapter, you will learn not only the basics for reading and writing files, but also all the how-tos for dealing with them.
Note
This chapter covers plain files (txt, csv, fixed width) and Excel files. For recipes for reading and writing XML files, refer to Chapter 3, Manipulating XML Structures.
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.