Time for action—aborting when there are too many errors
by María Carina Roldán
Pentaho 3.2 Data Integration Beginner's Guide
- Pentaho 3.2 Data Integration Beginner's Guide
- Pentaho 3.2 Data Integration
- Credits
- Foreword
- The Kettle Project
- About the Author
- About the Reviewers
- Preface
- 1. Getting Started with Pentaho Data Integration
- Pentaho Data Integration and Pentaho BI Suite
- Pentaho Data Integration
- Installing PDI
- Time for action—installing PDI
- Launching the PDI graphical designer: Spoon
- Time for action—starting and customizing Spoon
- Time for action—creating a hello world transformation
- Time for action—running and previewing the hello_world transformation
- Installing MySQL
- Time for action—installing MySQL on Windows
- Time for action—installing MySQL on Ubuntu
- Summary
- 2. Getting Started with Transformations
- Reading data from files
- Time for action—reading results of football matches from files
- Time for action—reading all your files at a time using a single Text file input step
- Time for action reading all your files at a time using a single Text file input step and regular expressions
- Sending data to files
- Time for action—sending the results of matches to a plain file
- Getting system information
- Time for action—updating a file with news about examinations
- Time for action—running the examination transformation from a terminal window
- XML files
- Time for action—getting data from an XML file with information about countries
- Summary
- 3. Basic Data Manipulation
- Basic calculations
- Time for action—reviewing examinations by using the Calculator step
- Time for action—reviewing examinations by using the Formula step
- Calculations on groups of rows
- Time for action—calculating World Cup statistics by grouping data
- Filtering
- Time for action—counting frequent words by filtering
- Looking up data
- Time for action—finding out which language people speak
- Summary
- 4. Controlling the Flow of Data
- Splitting streams
- Time for action—browsing new PDI features by copying a dataset
- Time for action—assigning tasks by distributing
- Splitting the stream based on conditions
- Time for action - assigning tasks by filtering priorities with the Filter rows step
- Time for action—assigning tasks by filtering priorities with the Switch/ Case step
- Merging streams
- Time for action—gathering progress and merging all together
- Time for action—giving priority to Bouchard by using Append Stream
- Summary
- 5. Transforming Your Data with JavaScript Code and the JavaScript Step
- Doing simple tasks with the JavaScript step
- Time for action—calculating scores with JavaScript
- Time for action—testing the calculation of averages
- Enriching the code
- Time for action calculating flexible scores by using variables
- Reading and parsing unstructured files
- Time for action—changing a list of house descriptions with JavaScript
- Avoiding coding by using purpose-built steps
- Summary
- 6. Transforming the Row Set
- Converting rows to columns
- Time for action—enhancing a films file by converting rows to columns
- Time for action—calculating total scores by performances by country
- Normalizing data
- Time for action—enhancing the matches file by normalizing the dataset
- Generating a custom time dimension dataset by using Kettle variables
- Time for action—creating the time dimension dataset
- Time for action—getting variables for setting the default starting date
- Summary
- 7. Validating Data and Handling Errors
- Capturing errors
- Time for action—capturing errors while calculating the age of a film
- Time for action—aborting when there are too many errors
- Time for action—treating errors that may appear
- Avoiding unexpected errors by validating data
- Time for action validating genres with a Regex Evaluation step
- Time for action—checking films file with the Data Validator
- Summary
- 8. Working with Databases
- Introducing the Steel Wheels sample database
- Time for action—creating a connection with the Steel Wheels database
- Time for action—exploring the sample database
- Querying a database
- Time for action—getting data about shipped orders
- Time for action—getting orders in a range of dates by using parameters
- Time for action—getting orders in a range of dates by using variables
- Sending data to a database
- Time for action—loading a table with a list of manufacturers
- Time for action—inserting new products or updating existent ones
- Time for action—testing the update of existing products
- Eliminating data from a database
- Time for action—deleting data about discontinued items
- Summary
- 9. Performing Advanced Operations with Databases
- Preparing the environment
- Time for action—populating the Jigsaw database
- Looking up data in a database
- Time for action—using a Database lookup step to create a list of products to buy
- Time for action using a Database join step to create a list of suggested products to buy
- Introducing dimensional modeling
- Loading dimensions with data
- Time for action loading a region dimension with a Combination lookup/update step
- Time for action—testing the transformation that loads the region dimension
- Time for action—keeping a history of product changes with the Dimension lookup/update step
- Time for action—testing the transformation that keeps a history of product changes
- Summary
- 10. Creating Basic Task Flows
- Introducing PDI jobs
- Time for action—creating a simple hello world job
- Receiving arguments and parameters in a job
- Time for action—customizing the hello world file with arguments and parameters
- Running jobs from a terminal window
- Time for action—executing the hello world job from a terminal window
- Using named parameters and command-line arguments in transformations
- Time for action—calling the hello world transformation with fixed arguments and parameters
- Deciding between the use of a command-line argument and a named parameter
- Running job entries under conditions
- Time for action—sending a sales report and warning the administrator if something is wrong
- Summary
- 11. Creating Advanced Transformations and Jobs
- Enhancing your processes with the use of variables
- Time for action—updating a file with news about examinations by setting a variable with the name of the file
- Enhancing the design of your processes
- Time for action—generating files with top scores
- Time for action—calculating the top scores with a subtransformation
- Time for action—splitting the generation of top scores by copying and getting rows
- Time for action—generating the files with top scores by nesting jobs
- Iterating jobs and transformations
- Time for action—generating custom files by executing a transformation for every input row
- Summary
- 12. Developing and Implementing a Simple Datamart
- Exploring the sales datamart
- Loading the dimensions
- Time for action—loading dimensions for the sales datamart
- Extending the sales datamart model
- Loading a fact table with aggregated data
- Time for action—loading the sales fact table by looking up dimensions
- Getting facts and dimensions together
- Time for action—loading the fact table using a range of dates obtained from the command line
- Time for action—loading the sales star
- Getting rid of administrative tasks
- Time for action—automating the loading of the sales datamart
- Summary
- 13. Taking it Further
- A. Working with Repositories
- Creating a repository
- Time for action—creating a PDI repository
- Working with the repository storage system
- Time for action—logging into a repository
- Examining and modifying the contents of a repository with the Repository explorer
- Migrating from a file-based system to a repository-based system and vice-versa
- Summary
- B. Pan and Kitchen: Launching Transformations and Jobs from the Command Line
- C. Quick Reference: Steps and Job Entries
- D. Spoon Shortcuts
- E. Introducing PDI 4 Features
- F. Pop quiz—Answers
- Open the transformation from the previous tutorial and save it under a different name.
- From the Flow category, drag an Abort step to the canvas.
- Create a hop from the Write to log step to the Abort step.
- Double-click the Abort step. Enter
5as Abort threshold. As Abort message, typeToo many errors calculating age of film!. - Click OK.
- Select the Dummy step and do a preview. As a result, a warning window shows up informing you that there were no rows to display. In the Step Metrics tab, the Abort after 5 errors line becomes red to show you that there was an error:
- The log looks like this:
... - Bad rows.0 - ... - Bad rows.0 - ==================== ... - Abort after 5 errors.0 - Row nr 6 causing abort : [Trois couleurs - Blanc], [null], [Comedy | Drama], [Krzysztof Kieslowski], [Zbigniew Zamachowski, Julie Delpy], [2009/08/18 00:00:00.000], [ ... - Abort after 5 errors.0 - Javascript error: ... - Abort after 5 errors.0 - Could not apply the given format dd/MM/yyyy on the string for 01/01/null : Format.parseObject(String) failed (script#4) ... - Abort after 5 errors.0 - ... - Abort after 5 errors.0 - --> 4:0], [SCR-001] ... - Abort after 5 errors.0 - Too many errors calculating age of film! ... - Abort after 5 errors.0 - Finished processing (I=0, O=0, R=6, W=6, U=0, E=1) ... ... - Spoon - The transformation has finished!! ... - error_handling_with_abort - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : Errors detected!
You forced the abortion of a transformation after five erroneous rows.
Through the use of the Abort step, you force the abortion of a transformation. Its main use is in error handling.
You can use the Abort step to force the abortion as soon as a row arrives to it, or after a certain number of rows as you did in the tutorial. To decide between one and the other option, you use the Abort threshold option. If threshold is 0, the Abort step will abort after the first row arrives. If threshold is N, the Abort step will cause the abortion of the transformation when the row number N+1 arrives at it.
Beyond the error handling situation, you may use the Abort step in any unexpected situation. Examples of that could be when you expect parameters and they are not present or when an incoming file is empty when it shouldn't be. In situations like these, you can force an abnormal ending of the execution just by adding an Abort step after the step that detects the anomaly.
In the Time for action—capturing errors while calculating the age of a film section of this chapter, you sent the bad rows to the log. However, when you capture errors, you can send the bad rows toward any step as long as the step knows how to treat those rows. Let's see an example of that.
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.