Physicians and hospitals alike receive payments from various external organizations, such as pharmaceutical companies who engage sales representatives to not only educate practitioners on their products, but also provide gifts or payments in kind or otherwise. In theory, gifts or payments made to physicians are not intended to influence their prescribing behavior, and pharmaceutical companies adopt careful measures to maintain checks and balances on payments being made to healthcare providers.
In 2010, President Obama's signature Affordable Care Act (ACA), also known in popular parlance as Obamacare, went into effect. Alongside the ACA, a separate legislation known as the Sunshine Act made reporting items of monetary value (directly or indirectly) mandatory for pharmaceutical companies and other organizations. While such rules existed in the past, rarely were such rules available in the public domain. By making detailed payment records made to all physicians available publicly, the Sunshine Act introduced an unprecedented level of transparency in monetary dealings involving healthcare providers.
The data is freely available on the website of CMS Open Payments at https://openpaymentsdata.cms.gov.
The site provides an interface to query the data, but does not have any means to perform large-scale data aggregation. For example, if a user wanted to find the total payments made in the state of CT, there is no simple and easy way to run the query through the default web-based tool. An API that provides the functionality is available, but requires a degree of familiarity and technical knowledge to use effectively. There are third-party products that provide such facilities, but in most cases they are expensive, and end users cannot modify the software to their particular needs.
In this tutorial, we will develop a fast, highly efficient web-based application to analyze tens of millions of records that capture payments made to physicians in 2016. We will be using a combination of a NoSQL database, R, and RStudio to create the final product - the web-based portal through which end users can query the database in real time.
The technologies we will use to develop the application are as follows:
- Kdb+ NoSQL database: http://www.kx.com
- R
- RStudio
For the tutorial, I will be using the VM image we downloaded for our Hadoop exercise. The tools can also be installed on Windows, Mac, and other Linux machines. The choice of the VM is mainly to provide a consistent and local OS independent platform.