Chapter 17. Generic Batch File Processing Framework

 

Simple things should be simple and complex things should be possible.

 
 --Alan Kay

Of all the common elements in the majority of development tools, batch file processing is used quite frequently by game tool programmers. It is not uncommon for a game to contain multiple gigabytes of game content files, so batch file processing is a must when a large volume of data is in need of alteration.

Some examples of batch file processing include generation of normal and texture maps from a collection of source art, deleting all files where the file name matches a particular search string, and recursively copying a folder hierarchy to another location when the directory layout is restructured.

The type of processing done on the files could be pretty much anything, although the code behind recursively iterating through directories and files remains relatively generic. Because effective use of time is essential when developing tools that our coworkers are waiting on, especially when the tool is not overly complex, reusability of common code is crucial. In this chapter, I present a generic batch file processing framework that promotes reusability, strong design, and flexibility.

Goals

The main goal of this framework is to promote reusability of the code that exists in all batch file processing tools, and to ensure that this framework will be sufficiently flexible for all the tools utilizing it.

The framework must be designed to work in either a console or WinForm environment, so the code should remain in a class library and only reference the core assemblies.

Strong design should be promoted through the use of solid OOP techniques. Maintainability is extremely important in any project, so a framework with a solid design results in better tools.

Developers must be able to quickly build tools without using a cumbersome API; the framework should be easy to configure and execute.

A verbose mode where operation progress can be reported to the user should also be available, keeping in mind that other tools should be able to run in silent mode as well.

The framework must have extremely low overhead because large operations demand performance.

Special situations where files are read-only should also be handled safely; the framework should be able to ignore read-only files or force writing if configured to do so. Configuration of the common base code is important, so other options, such as whether or not to recursively travel down directory structures, will also be available.

Lastly, the framework should be able to cancel the current operation. Support that will enable cancellation on a per-transaction basis will be integrated; that is, cancellation will not be supported halfway through the modification of a file, but rather after the current operation finishes. A mechanism will be available to developers so that they can support cancellation during an operation if they wish to worry about data integrity themselves.

Proposed Solution

In order to make a truly generic framework, we have to isolate the code that is different than other batch file processing tools, and build our framework around the code that remains. The work that these utilities perform is the variant data, so a generic framework must be able to support an interface that allows different functions to be attached to it, depending on the work needed. There are two ways that our framework will allow the worker function to be defined: through the use of delegates and through the use of virtual functions accessible through inheritance.

Delegates, the equivalent to function pointers in C++, will allow our system to specify the worker function without requiring inheritance. The delegate approach should only be used in throw-away tools where time is more important than maintainability, because delegates generally promote bad design when compared to the alternate OOP approach.

The other way that we will be able to specify the worker functions will be through inheriting from the base framework class. A virtual function will be called when a file is to be processed and the super class can take care of it appropriately.

For example, if you have a tool that has to recursively open all .txt files in a directory and replace occurrences of a certain phrase with another, you would create a class that inherits from the base framework, and override the process method. In this method, you would open the file, read in the text, perform the substitution, and save the new text back to disk. All the code that handles the recursion, file attributes, pattern matching, and other common I/O operations would be left to the framework, loosely coupled from the tool itself.

The properties, events, and methods of the base framework class will be defined in an interface to ensure strong OOP design. This will allow for a modular approach to even the framework engine itself, if more than one engine is ever used.

A delegate will exist for progress notification, so that users will be able to watch the status of the current operation.

The framework will also provide support for handling read-only files. The ability to skip read-only files will be available, as will the ability to remove the read-only attribute from the file before passing the file off to the worker function.

Implementation

Based on the above goals and proposed solution, the following two components make up the batch file processing framework. See Table 17.1-17.3 for description of Delegate Definitions.

The following interface defines the properties, events, and methods that the framework engine must realize. The code is quite simple, but I will go over the code for the sake of clarity.

Delegate Definitions

Table 17.1. Delegate Definitions

Delegate

Description

FileAccessProcess

This event is fired when the worker function wishes to notify the user about the operation. This delegate is available to the tool regardless of the method chosen to specify the worker function.

FileAccessNotify

This event is fired when the worker function wishes to notify the user about the operation. This delegate is available to the tool regardless of the method chosen to specify the worker function.

 

Table 17.2. Property Definitions

Property

Description

Recursive

This property is used to specify whether or not directories are traversed in a recursive fashion. If this property is false, then only the top-level directory is actually processed.

SkipReadOnly

This property is used to specify whether or not files that are marked with a read-only attribute should be processed by the worker function.

ForceWriteable

This property is used to specify whether or not files that are marked with a read-only attribute should be made writeable and then processed by the worker function.

FilePattern

This property is used to specify the pattern to match when choosing the files to process in a directory. The default pattern is *.*, which processes every file. If the pattern were set to *.txt, then only text files would be processed.

Cancelled

This property is used to specify whether or not the operation has been cancelled. The worker function can check this property each time it is called to see if cancellation is occurring.

 

Table 17.3. Method Definitions

Method

Description

Execute

This method is called by the tool when processing should begin using the set options and worker function. The full path to the directory to begin processing with is sent in as a parameter.

Cancel

This method is fairly self-explanatory; it cancels all remaining operations that have not yet been started, and it sets the Cancelled property so that the worker function knows that it should either stop what it is doing or finish up.

Notify

This method is called by the worker function to fire the OnNotify event. As long as the tool has set this delegate to a function, it will fire when a notification is sent.

The following code composes the file access interface that powers the logic behind each batch processing tool. This interface is implemented and customized for each tool.

using System;
using System.IO;

namespace BatchFileFramework
{
    public delegate void FileAccessProcess(IFileAccessLogic logic, FileInfo fileInfo);
    public delegate void FileAccessNotify(string message);

    public interface IFileAccessLogic
    {
       bool Recursive
        {
            get;
            set;
        }

        bool SkipReadOnly
        {
            get;
            set;
        }

        bool ForceWriteable
        {
            get;
            set;
        }
        string FilePattern
        {
            get;
            set;
        }

        bool Cancelled
        {
            get;
            set;
        }

        void Execute(string fullPath);
        void Cancel();
        void Notify(string message);

        event FileAccessProcess OnProcess;
        event FileAccessNotify OnNotify;
    }
}

The following class implements the IFileAccessLogic interface and houses a lot of the common functionality that is present in almost every batch file processing tool.

using System;
using System.IO;

namespace BatchFileFramework
{
    public class FileAccessLogic : IFileAccessLogic
    {
        private bool verbose = false;
        private bool recursive = false;
        private bool skipReadOnly = false;
        private bool forceWriteable = false;

        private string filePattern = "*.*";

        private bool cancelled = false;
        private bool running = false;

        public event FileAccessProcess OnProcess = null;
        public event FileAccessNotify OnNotify = null;
        public bool Verbose
        {
            get { return verbose; }
            set
            {
                if (!this.running)
                    verbose = value;
            }
        }

        public bool Recursive
        {
            get { return recursive; }
            set
            {
                if (!this.running)
                    recursive = value;
            }
        }

        public bool SkipReadOnly
        {
            get { return skipReadOnly; }
            set
            {
                if (!this.running)
                    skipReadOnly = value;
            }
        }

        public bool ForceWriteable
        {
            get { return forceWriteable; }
            set
            {
                if (!this.running)
                    forceWriteable = value;
            }
        }

        public string FilePattern
        {
            get { return filePattern; }
            set
            {
                if (!this.running)
                    filePattern = value;
            }
        }

        public bool Cancelled
        {
            get { return cancelled; }
            set { cancelled = value; }
        }

        public void Execute(string fullPath)
        {
            cancelled = false;
            running = true;

            if (File.Exists(fullPath))
                Process(this, new FileInfo(fullPath));

            else if (Directory.Exists(fullPath))
                ProcessDirectory(fullPath);

            running = false;
        }

        public void Cancel()
        {
            cancelled = true;
        }

        public void Notify(string message)
        {
            if (!verbose)
            {
                if (this.OnNotify != null)
                    this.OnNotify(message);
            }
        }

        private void ProcessDirectory(string directoryPath)
        {
            ProcessDirectory(new DirectoryInfo(directoryPath));
        }

        private void ProcessDirectory(DirectoryInfo directoryInfo)
        {
            if (cancelled)
                return;

            ProcessFiles(directoryInfo);

            if (recursive)
            {
                foreach (DirectoryInfo subDirectoryInfo in
                     directoryInfo.GetDirectories())
                    ProcessDirectory(subDirectoryInfo);
         }
     }

     private void ProcessFiles(DirectoryInfo directoryInfo)
     {
        foreach (FileInfo fileInfo in directoryInfo.GetFiles(this.filePattern))
          {
              if (cancelled)
                  return;

              FileAttributes attributes = File.GetAttributes(fileInfo.FullName);

             if ((attributes & FileAttributes.ReadOnly) == FileAttributes.ReadOnly)
              {
                   if (skipReadOnly)
                       continue;

                   else if (forceWriteable)
                      File.SetAttributes(fileInfo.FullName, FileAttributes.Normal);

                   else
                       continue;
              }

              Process(this, fileInfo);
              }
        }

        protected virtual void Process(IFileAccessLogic logic, FileInfo fileInfo)
        {
            if (OnProcess != null)
                OnProcess(this, fileInfo);
        }
    }
}

Conclusion

On the Companion Web site are two examples showing a number of features of this framework.

There is a simple listing example that does not perform any file modification, so it is safe to run from the top-level directory of your hard drive for the best performance results. This example shows how to use the delegate approach to specify the worker function.

The other example is a search and replace process that searches for all .txt files in the directory structure and replaces a particular search pattern with another specified word.

Caution

The search and replace example should be used with care so you do not modify the wrong files!

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.219.249.210