kdb+ is a time-series, in-memory, columnar database that has been used in the financial industry for almost 20 years. It is one of the fastest database platforms available for performing large-scale data mining, but one that is not as well-known as other NoSQL tools due to the fact that it has been used almost exclusively by hedge funds and investment banks for most of its existence. In particular, due to its speed and low overhead in processing vast amounts of data, it is used by algorithmic trading desks that engage in high-frequency trading.

With kdb+, it is fairly simple to analyze tens of millions and even hundreds of millions of records on a laptop. The main constraints would be at a hardware level - such as the amount of memory, disk space, and CPU that is available to process the data. In this tutorial, we will install the free 32-bit edition of kdb+ available for non-commercial use.

kdb+ is not open source, but academic institutes can use the 64-bit license at no charge by writing to [email protected].

There are certain key characteristics of kdb+ that make it very well suited to large-scale data analysis:

• Low-level implementation: The database is written in C, thus reducing common causes of performance issues with most contemporary NoSQL databases that rely heavily on Java, which implements multiple layers of abstraction to provide processing capabilities
• Architectural simplicity: The entire binary for the kdb+ database is about 500-600 KB. This is a fraction of the size of an MP3 song and can be easily downloaded even on a dial-up connection
• MapReduce: The database implements an internal MapReduce process that allows queries to execute across multiple cores simultaneously
• No installation: The database requires no system-level privileges and users can start using kdb+ with their user account on most systems
• Enterprise-ready: The database has been used for nearly 20 years and is a very mature product used in global enterprise environments for analysis of high-frequency trading data among other applications
• Wide availability of interfaces: The database has a wide range of interfaces for languages such as C, C++,C#, Java, R, Python, MATLAB, and others to allow easy integration with existing software

The steps to install kdb+ are given as follows. Please note that if you are using the Packt Data Science VM, no additional installation is necessary. The instructions have been provided primarily for users who would like to install the software afresh.

1. Visit www.kx.com and click on the Download drop-down option from the Connect with us menu item. You may also directly go to the download page located at https://kx.com/download/:

The download page is as shown in the following screenshot:

2. Click on Download on the next page.
3. You'll be taken to https://kx.com/download/ where you can select the respective download of your choice after agreeing to the terms. If you are using the VM, download the Linux-86 version.
4. Select Save File to save the downloaded ZIP file in your Downloads folder:

For Mac or Linux systems, this will be the ~/ folder. In Windows, copy the ZIP file under C: and unzip to extract the q folder. The following instructions are mainly for Linux-based systems:

$ cd Downloads/ # cd to the folder where you have downloaded the zip file 

$ unzip linuxx86.zip  
Archive:  linuxx86.zip 
  inflating: q/README.txt             
  inflating: q/l32/q                  
  inflating: q/q.q                    
  inflating: q/q.k                    
  inflating: q/s.k                    
  inflating: q/trade.q                
  inflating: q/sp.q                   
 
$ mv ~/Downloads/q ~/ 
$ cd ~/q 
$ cd l32 
$ ./q 
KDB+ 3.5 2017.06.15 Copyright (C) 1993-2017 Kx Systems 
l32/ 1()core 3830MB cloudera quickstart.cloudera 10.0.2.15 NONEXPIRE   
 
Welcome to kdb+ 32bit edition 
For support please see http://groups.google.com/d/forum/personal-kdbplus 
Tutorials can be found at http://code.kx.com/wiki/Tutorials 
To exit, type \ 
To remove this startup msg, edit q.q 
q)\

/NOTE THAT YOU MAY NEED TO INSTALL THE FOLLOWING IF YOU GET AN ERROR MESSAGE STATING THAT THE FILE q CANNOT BE FOUND. IN THAT CASE, INSTALL THE REQUISITE SOFTWARE AS SHOWN BELOW 
 
$ sudo dpkg --add-architecture i386 
$ sudo apt-get update 
$ sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386