Time for action—running and previewing the hello_world transformation
by María Carina Roldán
Pentaho 3.2 Data Integration Beginner's Guide
- Pentaho 3.2 Data Integration Beginner's Guide
- Pentaho 3.2 Data Integration
- Credits
- Foreword
- The Kettle Project
- About the Author
- About the Reviewers
- Preface
- 1. Getting Started with Pentaho Data Integration
- Pentaho Data Integration and Pentaho BI Suite
- Pentaho Data Integration
- Installing PDI
- Time for action—installing PDI
- Launching the PDI graphical designer: Spoon
- Time for action—starting and customizing Spoon
- Time for action—creating a hello world transformation
- Time for action—running and previewing the hello_world transformation
- Installing MySQL
- Time for action—installing MySQL on Windows
- Time for action—installing MySQL on Ubuntu
- Summary
- 2. Getting Started with Transformations
- Reading data from files
- Time for action—reading results of football matches from files
- Time for action—reading all your files at a time using a single Text file input step
- Time for action reading all your files at a time using a single Text file input step and regular expressions
- Sending data to files
- Time for action—sending the results of matches to a plain file
- Getting system information
- Time for action—updating a file with news about examinations
- Time for action—running the examination transformation from a terminal window
- XML files
- Time for action—getting data from an XML file with information about countries
- Summary
- 3. Basic Data Manipulation
- Basic calculations
- Time for action—reviewing examinations by using the Calculator step
- Time for action—reviewing examinations by using the Formula step
- Calculations on groups of rows
- Time for action—calculating World Cup statistics by grouping data
- Filtering
- Time for action—counting frequent words by filtering
- Looking up data
- Time for action—finding out which language people speak
- Summary
- 4. Controlling the Flow of Data
- Splitting streams
- Time for action—browsing new PDI features by copying a dataset
- Time for action—assigning tasks by distributing
- Splitting the stream based on conditions
- Time for action - assigning tasks by filtering priorities with the Filter rows step
- Time for action—assigning tasks by filtering priorities with the Switch/ Case step
- Merging streams
- Time for action—gathering progress and merging all together
- Time for action—giving priority to Bouchard by using Append Stream
- Summary
- 5. Transforming Your Data with JavaScript Code and the JavaScript Step
- Doing simple tasks with the JavaScript step
- Time for action—calculating scores with JavaScript
- Time for action—testing the calculation of averages
- Enriching the code
- Time for action calculating flexible scores by using variables
- Reading and parsing unstructured files
- Time for action—changing a list of house descriptions with JavaScript
- Avoiding coding by using purpose-built steps
- Summary
- 6. Transforming the Row Set
- Converting rows to columns
- Time for action—enhancing a films file by converting rows to columns
- Time for action—calculating total scores by performances by country
- Normalizing data
- Time for action—enhancing the matches file by normalizing the dataset
- Generating a custom time dimension dataset by using Kettle variables
- Time for action—creating the time dimension dataset
- Time for action—getting variables for setting the default starting date
- Summary
- 7. Validating Data and Handling Errors
- Capturing errors
- Time for action—capturing errors while calculating the age of a film
- Time for action—aborting when there are too many errors
- Time for action—treating errors that may appear
- Avoiding unexpected errors by validating data
- Time for action validating genres with a Regex Evaluation step
- Time for action—checking films file with the Data Validator
- Summary
- 8. Working with Databases
- Introducing the Steel Wheels sample database
- Time for action—creating a connection with the Steel Wheels database
- Time for action—exploring the sample database
- Querying a database
- Time for action—getting data about shipped orders
- Time for action—getting orders in a range of dates by using parameters
- Time for action—getting orders in a range of dates by using variables
- Sending data to a database
- Time for action—loading a table with a list of manufacturers
- Time for action—inserting new products or updating existent ones
- Time for action—testing the update of existing products
- Eliminating data from a database
- Time for action—deleting data about discontinued items
- Summary
- 9. Performing Advanced Operations with Databases
- Preparing the environment
- Time for action—populating the Jigsaw database
- Looking up data in a database
- Time for action—using a Database lookup step to create a list of products to buy
- Time for action using a Database join step to create a list of suggested products to buy
- Introducing dimensional modeling
- Loading dimensions with data
- Time for action loading a region dimension with a Combination lookup/update step
- Time for action—testing the transformation that loads the region dimension
- Time for action—keeping a history of product changes with the Dimension lookup/update step
- Time for action—testing the transformation that keeps a history of product changes
- Summary
- 10. Creating Basic Task Flows
- Introducing PDI jobs
- Time for action—creating a simple hello world job
- Receiving arguments and parameters in a job
- Time for action—customizing the hello world file with arguments and parameters
- Running jobs from a terminal window
- Time for action—executing the hello world job from a terminal window
- Using named parameters and command-line arguments in transformations
- Time for action—calling the hello world transformation with fixed arguments and parameters
- Deciding between the use of a command-line argument and a named parameter
- Running job entries under conditions
- Time for action—sending a sales report and warning the administrator if something is wrong
- Summary
- 11. Creating Advanced Transformations and Jobs
- Enhancing your processes with the use of variables
- Time for action—updating a file with news about examinations by setting a variable with the name of the file
- Enhancing the design of your processes
- Time for action—generating files with top scores
- Time for action—calculating the top scores with a subtransformation
- Time for action—splitting the generation of top scores by copying and getting rows
- Time for action—generating the files with top scores by nesting jobs
- Iterating jobs and transformations
- Time for action—generating custom files by executing a transformation for every input row
- Summary
- 12. Developing and Implementing a Simple Datamart
- Exploring the sales datamart
- Loading the dimensions
- Time for action—loading dimensions for the sales datamart
- Extending the sales datamart model
- Loading a fact table with aggregated data
- Time for action—loading the sales fact table by looking up dimensions
- Getting facts and dimensions together
- Time for action—loading the fact table using a range of dates obtained from the command line
- Time for action—loading the sales star
- Getting rid of administrative tasks
- Time for action—automating the loading of the sales datamart
- Summary
- 13. Taking it Further
- A. Working with Repositories
- Creating a repository
- Time for action—creating a PDI repository
- Working with the repository storage system
- Time for action—logging into a repository
- Examining and modifying the contents of a repository with the Repository explorer
- Migrating from a file-based system to a repository-based system and vice-versa
- Summary
- B. Pan and Kitchen: Launching Transformations and Jobs from the Command Line
- C. Quick Reference: Steps and Job Entries
- D. Spoon Shortcuts
- E. Introducing PDI 4 Features
- F. Pop quiz—Answers
Let's do some testing and explore the results:
- Open the
hello_worldtransformation. - Edit the Generate Rows step, and change the limit from
10to1000so that it generates 1,000 rows. - Select the Logging tab window at the bottom of the screen.
- Click on Run.
- In the Log level drop-down list, select RowLevel detail.
- Click on Launch.
- You can see how the logging window shows every task in a very detailed way.
- Edit the Generate Rows step, and change the limit to 10,000 so that it generates 10,000 rows.
- Select the Step Metrics.
- Run the transformation.
- You can see how the numbers change as the rows travel through the steps.
You did some tests with the hello_world transformation and saw the results in the Execution Results window.
The Execution Results window shows you what is happening while you preview or run a transformation.
The Logging tab shows the execution of your transformation, step by step. By default, the level of the logging detail is Basic but you can change it to see different levels of detail—from a minimal logging (level Minimal) to a very detailed one (level RowLevel).
The Step Metrics tab shows, for each step of the transformation, the executed operations and several status and information columns. You may be interested in the following columns:
|
Column |
Description |
|---|---|
|
Read |
Contains the number of rows coming from previous steps |
|
Written |
Contains the number of rows leaving from this step toward the next |
|
Input |
Number of rows read from a file or table |
|
Output |
Number of rows written to a file or table |
|
Errors |
Errors in the execution. If there are errors, the whole row becomes red |
|
Active |
Tells the current status of the execution |
In the example, you can see that the Generate Rows step writes rows, which then are read by the Dummy step. The Dummy step also writes the same rows, but in this case those go nowhere.
For each of the following, decide if the sentence is true or false:
- There are several graphical tools in PDI, but Spoon is the most used.
- You can choose to save Transformations either in files or in a database.
- To run a Transformation, an executable file has to be generated from Spoon.
- The grid size option in the Look and Feel windows allows you to resize the work area.
- To create a transformation, you have to provide external data.
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.