Reading binary files into memory

Reading an entire binary file into memory can be accomplished via Files.readAllBytes():

byte[] bytes = Files.readAllBytes(binaryFile);

A similar method exists in the InputStream class as well.

While these methods are very convenient for relatively small files, they are not a good choice for large files. Trying to fetch large files into memory is prone to OOM errors and, obviously, will consume a lot of memory. Alternatively, in the case of huge files (e.g., 200 GB), we can focus on memory-mapped files (MappedByteBuffer). MappedByteBuffer allows us to create and modify huge files and treat them as a very big array. They look like they are in memory even if they are not. Everything happens at the native level:

try (FileChannel fileChannel = (FileChannel.open(binaryFile,
EnumSet.of(StandardOpenOption.READ)))) {

MappedByteBuffer mbBuffer = fileChannel.map(
FileChannel.MapMode.READ_ONLY, 0, fileChannel.size());

System.out.println(" Read: " + mbBuffer.limit() + " bytes");
}

For huge files it is advisable to traverse the buffer with a fixed size as follows:

private static final int MAP_SIZE = 5242880; // 5 MB in bytes

try (FileChannel fileChannel = FileChannel.open(
binaryFile, StandardOpenOption.READ)) {

int position = 0;
long length = fileChannel.size();

while (position < length) {
long remaining = length - position;
int bytestomap = (int) Math.min(MAP_SIZE, remaining);

MappedByteBuffer mbBuffer = fileChannel.map(
MapMode.READ_ONLY, position, bytestomap);

... // do something with the current buffer

position += bytestomap;
}
}
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.148.144.228