Before this chapter digs into the individual requirements of the batch process you develop in Chapter 10, let's spend a little time going over the approach you use to do so. A lot has been said in our industry about various agile processes; so instead of banking on any previous knowledge you may have of the subject, let's start by establishing a base of what agile and the development process will mean for this book.

The agile process has 12 tenets that virtually all of its variants prescribe. They are as follows:

It doesn't matter if you're using Extreme Programming (XP), Scrum, or any other currently hip variant. The point is that these dozen tenets still apply.

Notice that not all of them will necessarily apply in your case. It's pretty hard to work face to face with a book. You'll probably be working by yourself through the examples, so the aspects of team motivation don't exactly apply either. However, there are pieces that do apply. An example is quick delivery of working software. This will drive you through out the book. You'll accomplish it by building small pieces of the application, validating that they work with unit tests, and then adding onto them.

Even with the exceptions, the tenets of agile provide a solid framework for any development project, and this book applies as many of them as possible. Let's get started looking at how they're applied by examining the way you document the requirements for the sample job: user stories.

Now that you've seen the pieces of the development process you're encouraged to use as you learn Spring Batch, let's look at what you'll develop in this book. Figure 3-1 shows what you expect to get in the mail from your stockbroker each quarter as your brokerage account statement.

Figure 3.1. Brokerage statement, formatted and printed on letterhead

If you break down how the statement is created, there are really two pieces to it. The first is nothing more than a pretty piece of paper on which the second piece is printed. It's the second piece, shown in Figure 3-2, that you create in this book.

Figure 3.2. Plain-text brokerage statement

Typically, statements are created as follow. A batch process creates a print file consisting of little more than text. That print file is then sent to a printer that prints the text onto the decorated paper, producing the final statement. The print file is the piece you create using Spring Batch. Your batch process will perform the following functions:

Let's look at what each of these features entails. Your job is provided with a customer-transaction flat file that consists of information about a customer and their transactions for the month. Your job updates existing customer information and adds their transactions to the database. When the transactions have been imported, the job obtains the latest prices for each of the stocks in the database from a web service, in order to calculate each account's current value. The job imports the downloaded prices into the database.

After the initial imports are complete, your job can begin calculating transaction fees. The brokerage makes its money by charging a fee for each transaction it does. These fees are based on how many transactions a customer has in a month. The more transactions a customer has, the less they're charged per transaction. The first step in calculating the transaction fees is to determine what level or tier the user falls into; then you can calculate the price for the customer's transactions. When all the calculations have been completed, you can generate the user's monthly statement.

This list of features is intended to provide a complete view into how Spring Batch is used in a real-world problem. Throughout the book, you learn about the features Spring Batch provides to help you develop batch processes like the one required for this scenario. In Chapter 10, you implement the batch job to meet the requirements outlined in the following user stories:

Get Stocks Closing Price: As the batch process, at the prescheduled execution time, I will query the Yahoo stock web service to obtain the closing prices of all stocks held over the course of the previous month by our customers. I will build a file with this data for future import. Acceptance criteria

Import Stock Prices: As the batch process, when I receive the stock price file, I will import the file into the database for future processing. Acceptance criteria:

Calculate Pricing Tiers: As the batch process, after all input has been imported, I will calculate the pricing tier each customer falls into and store it for future use. Acceptance criteria:

Calculate Fee Per Transaction: As the batch process, after I have completed calculating pricing tiers, I will calculate a brokerage fee per trade that the customer will be charged. Acceptance criteria:

Print Account Summary: As the batch process, after all calculations have been completed, I will print out a summary for each customer. This summary will provide an overview of the customer's account and a breakdown of what makes up the total value of their portfolio. Acceptance criteria:

Print Account Detail: As the batch process, after each account summary I will print the detail makeup of each account. The account detail will provide the customer with a detailed look into the makeup of their account and how their investments are doing. Acceptance criteria:

Print Statement Header: As the batch process, at the top of each page I will print a header. This will provide generic information about the account, the customer, and the brokerage. Acceptance criteria:

As stated before, the goal of this project is to take a real-world example and work through it using the features that Spring Batch provides to create a robust, maintainable solution. In order to accomplish this goal, the example includes elements that may seem a bit complex right now, such as headers, multiple file format imports, and complex output including subheadings. The reason is that Spring Batch provides facilities exactly for these features. Let's dig into how you structure this batch process by outlining the job and describing its steps.

Job Description

In order to implement the statement-generation process, you build a single job with six steps. Figure 3-3 shows the flow of the batch job for this process, and the following sections describe the steps.

Figure 3.3. Stock statement jobflow

Importing Customer Transaction Data

To start the job, you begin by importing the customer and transaction data. Contained in a flat file, this data has a complex format consisting of two record types. The first record type is that of the customer, consisting of the customer's name, address, and account information. The second record type consists of detailed information about each transaction, including the stock ticker, the price paid, the quantity purchased or sold, and the timestamp when the transaction occurred. Using Spring Batch's ability to read multiline records allows you to process this file with minimal coding. You write a data access object (DAO) for the JDBC persistence of the imported data, as shown in Listing 3-1.

Example 3.1. Customer/Transaction Input File

392041928,William,Robinson,9764 Jeopardy Lane,Chicago,IL,60606
HD,31.09,200,08:38:05
WMT,53.38,500,09:25:55
ADI,35.96,-300,10:56:10
REGN,29.53,-500,10:56:22
938472047,Robert,Johnson,1060 Addison St,Chicago,IL,60657
CABN,0.890,10000,14:52:15
NUAN,17.11,15000,15:02:45

Retrieving Stock Closing Prices

After you've imported the customer's information and transactions, you move on to obtaining the stock price information. This is a simple step that retrieves all the closing values for the stocks that have been traded over the previous month. To do this, you create a tasklet (as you did last chapter for the Hello, World batch process) to call the Yahoo web service specified in the requirements, and you download a CSV with the required stock data. This step writes the output to a file similar to that in Listing 3-2, for the next step to process.

Example 3.2. Stock Closing Price Input File

SHLD,71.98

CME,289.65
GOOG,590.83
F,16.28

Importing Stock Prices into Database

This step reads the file and imports the data into the database. This step showcases the strengths of the declarative I/O provided by Spring Batch. Both the input and output of this job require no code on your part. Spring Batch provides the ability to read the CSV file you downloaded in the previous step via stock components of the framework as well as to update the database.

You may wonder why you don't import the data directly. The reason is error handling. You're importing data that was provided to you by a third-party source. Because you can't be sure of the quality of the data, you need to be able to handle any errors that may occur during the import. By writing the data to a file for a future step to process, you can restart the step without having to re-download the stock prices.

Calculating Transaction Fee Tiers

Up to this point, you haven't had to do any real processing of the data you're reading and writing. All you've done is pipe data from a file to a database (with some validation for good measure). When you're finished importing the required data, you begin doing the required calculations. Your brokerage company charges fees based on how much a customer trades. The more the customer trades, the less they're charged per trade. The amounts customers are charged are assigned via tiers; each tier is defined by the number of trades performed over the previous month and has a dollar amount associated with it.

In this step, you introduce an item processor between the reader and writer to determine the tier to which the customer belongs. You declare the reader via XML to load the customer's trade information and the writer to update the customer's account in the same manner.

Calculating Transaction Fees

When you've determined the tier for each customer, you can calculate the fee for each trade. In the previous step, you processed records at the customer level, with each customer being assigned a tier. In this step, you process records at the individual transaction level. As you can imagine, you process many more records in this step than in any previous step; you examine this step in further detail later, when the book talks about scalability options. However, to start, this step looks almost exactly like the previous one, but with different SQL for the reader, different logic in the item processor, and yet another JdbcItemWriter for the writer.

Generating Customer Monthly Statements

The last step seems to be the most complex—but as you know, looks can be deceiving. This step involves the generation of the statements themselves. It demonstrates some of the beauty of applying decoupled solutions to batch problems. By providing a custom-coded formatter, you can do virtually all the work with a single simple class. This step also uses the callbacks for the headers.

All of this sounds great in theory but leaves a lot of questions to be answered. That's good. You'll spend the rest of the book working through how these features are implemented in the processes as well as examining things like exception handling and restart/retry logic. One final item you should be familiar with before you move on, though, is the data model. That will help clear the air regarding how this system is structured. Let's take a look.

Understanding the Data Model

You've seen all the different pieces of the job you create throughout this book. Let's move on to the last piece of the puzzle before you get into actual development. Batch processes are data driven. Because there are no user interfaces, the various datastores end up being the only external interaction the process has. This section looks at the data model used for the sample application.

Figure 3-4 outlines the application-specific tables for this batch process. To be clear, this diagram doesn't encompass all the tables required for this batch job to run. Chapter 2 took a brief look at the tables Spring Batch uses in the job repository. All of those tables will exist in addition to these in your database. Because it isn't uncommon to deploy the batch schema separately, and you reviewed it in the last chapter, I've chosen to leave it out of Figure 3-4.

Figure 3.4. Sample application data model

For the batch application, you have four tables: Customer, Account, Transaction, and Ticker. When you look at the data in the tables, notice that you aren't storing all the required fields to generate the statement. There are fields (such as the totals in the account summary) that you calculate during processing. Other than that, the data model should appear relatively straightforward:

• Customer: This record contains all the customer-specific information, including name and tax identification number.

• Account. For every customer, an account is maintained. For your purposes, each account has a number and a running cash balance from which fees are deducted as needed. The customer's transaction fee tier is also stored at this level.

• Transaction. Each trade has a corresponding record in the Transactions table. The data here is used to determine the current state of the account (how many shares are held, and so on).

• Ticker. For each stock that has been traded by the brokerage company's customers, a record exists in this table containing the ticker and the most recent closing price for the stock.

This chapter discussed the agile development process and how you can apply it to batch development. The chapter continued along those lines by defining requirements via user stories for the sample application you build throughout the course of this book. From this point, the book switches from the "what" and "why" of Spring Batch to the "how."

In the next chapter, you take a deep dive into Spring Batch's concepts of jobs and steps and look at a number of other-specific examples.