The beauty of SharedArrays is that the data is maintained as a single copy, and multiple processes can have both read and write access. It is a perfect solution to our problem.
In this solution, we will do the following:
- The master program creates a shared array.
- Using a distributed for loop, the master program commands worker processes to read each individual file into a specific segment of the array.
- Again, using a distributed for loop, the master program commands worker process to perform statistical analysis.
As we have 16 vCPUs, we can utilize all of them.
In practice, we should probably utilize fewer vCPUs so that we can leave some room for the operating system itself. Your mileage may vary depending on what else is running on the same server. The best approach is to test various configurations and determine the optimal settings.