In a risk management use case, we want to estimate the volatility of portfolio returns using a process called Monte Carlo simulation. The concept is pretty simple. First, we develop a risk model based on historical data. Second, we use the model to predict the future in 10,000 ways. Finally, we look at the distribution of security returns in the portfolio and gauge how much the portfolio gains or losses in each of those scenarios.
Portfolios are often measured against benchmarks. For example, a stock portfolio may be benchmarked against the S&P 500 Index. The reason is that portfolio managers are typically rewarded for earning alpha, a term for describing the excess return that is over and above the benchmark's return. In other words, the portfolio manager is rewarded for their skills in picking the right stocks.
In the fixed income market, the problem is a little more challenging. Unlike the stock market, typical fixed income benchmarks are quite large, up to 10,000 bonds. In assessing portfolio risk, we often want to analyze the sources of return. Did the value of a portfolio go up because it was riding the wave in a bull market, or did it go down because everyone is selling? The risk that correlates to market movement is called systematic risk. Another source of return relates to the individual bond. For example, if the issuer of the bond is doing well and making good profit, then the bond has a lower risk and the price goes up. This kind of movement due to the specific individual bond is called idiosyncratic risk. For a global portfolio, some bonds are exposed to currency risk as well. From a computational complexity perspective, to estimate the returns of the benchmark index 10,000 ways, we have to perform 10,000 future scenarios x 10,000 securities x 3 sources of returns = 300 million pricing calculations.
Coming back to our simulation example, we can generate 10,000 possible future scenarios of the portfolio, and the results are basically a set of returns across all such scenarios. The returns data is stored on disk and is now ready for additional analysis. Here comes the problem—an asset manager has to analyze over 1,000 portfolios, and each portfolio may require access to returns data varying between 10,000 to 50,000 bonds depending on the size of the benchmark index. Unfortunately, the production server is limited in memory but has plenty of CPU resources. How can we fully utilize our hardware to perform the analysis as quickly as possible?
Let's quickly summarize our problem:
- Hardware:
- 16 vCPU
- 32 GB RAM
- Security returns data:
- Stored in 100,000 individual files
- Each file contains a 10,000 x 3 matrix (10,000 future states and 3 return sources)
- Total memory footprint is ~22 GB
- Task:
- Calculate statistical measures (standard deviation, skewness, and kurtosis) for all security returns across the 10,000 future states.
- Do that as quickly as possible!
The naive way to just load all of the files sequentially. Needless to say, loading 100,000 files one by one is not going to be very fast no matter how small the files are. We are going to use the Julia distributed computing facility to get it done.