The consumption of files is not something that immediately comes to mind when talking about transactions. Camel provides you with a way to do just this, through the use of the File Component.
This recipe will show you how to perform file consumption, and explain how Camel does its best to guarantee an all-or-nothing process, isolated from other threads and optionally processes, that leaves the consumed file in a state consistent with the overall operation.
The Java code for this recipe is located in the org.camelcookbook.transactions.fileconsumption package. The Spring XML files are located under src/main/resources/META-INF/spring and prefixed with fileConsumption.
The File Component in Camel is part of the camel-core library. To consume files from a specific directory, define a file: endpoint URI within a consumer endpoint, that is within a from statement as per the following pattern:
file:///path/to/source
When a file is successfully processed, it is moved into the .camel subdirectory under the source directory, by default. This can be overridden through the move attribute, which takes a relative or absolute path:
file:///path/to/source?move=///path/to/completed
It is also possible to instruct the component to delete completed files rather than moving them:
file:///path/to/source?delete=true
The component scans the directory referenced in the URI, and processes each file as an individual exchange through the route. Exceptions raised while processing the file are handled gracefully, by reattempting the consumption of the entire file.
It is possible to move any failed files to another directory, by providing the failure location in the moveFailed attribute. For example, in the following route, a failed file will be moved to an error directory, where it can be manually inspected; otherwise, on success, its contents will be sent to an output directory:
from("file:" + inputDirectory + "?moveFailed=" + errorDirectory)
.log("Consumed file ${header[CamelFileName]}: ${body}")
.convertBodyTo(String.class)
.choice()
.when(simple("${body} contains 'explode'"))
.throwException(
new IllegalArgumentException("File caused explosion"))
.otherwise()
.to("file:" + outputDirectory)
.endChoice();Note
In this example, the directories are provided as variables when the route is instantiated. See Reusing routing logic through template routes of Chapter 1, Structuring Routes, for an overview of this approach.
File consumption is performed as kind of best-effort transaction. Underneath the covers, the File Component binds a org.apache.camel.spi.Synchronization instance to the exchange as described in the Defining completion actions dynamically recipe in Chapter 7, Error Handling and Compensation. This callback will be invoked by the Camel runtime when that exchange has completed processing.
The File Component provides the logic for moving or deleting, the file if its processing was successful, and either leaving it where it was or moving it to another directory if the file failed. This behavior is configurable through the endpoint URI.
The File Component makes use of the technique covered in the Preventing duplicate invocation of routing logic recipe to ensure that the same files are not reprocessed. Idempotent consumption is disabled by default, but can be enabled by providing a reference to an IdempotentRepository instance in the consumer URI with the idempotentRepository attribute. It is useful in the following situations:
- The file consumer has been set to not move files after processing–when the
noopattribute is set totrue. Here, an in-memory idempotent repository will be used by default, unless overridden by theidempotentRepositoryattribute. - The same files will be uploaded into the source directory repeatedly, for example, the last seven days' worth of transactions.
- There will be more than one Camel runtime scanning the same directory. This is often seen in clusters of servers that are running in an Active/Active configuration, where all of the servers are active and executing the same Camel routes. To implement this use case, you would provide an instance of an
IdempotentRepositoryimplementation whose state is visible to both runtimes. This usually means providing a reference to aJDBCIdempotentRepositorythat refers to a shared database.
There are a large number of other options around consuming files that include the ability to define file name patterns for consumption, ordering, read locking, recursive directory traversals, and others. It is also possible to use the File Component as a producer endpoint in order to write messages into a directory. See the Camel File Component documentation for full details.
- Camel File Component: http://camel.apache.org/file2.html