Imported Libraries

Keep in mind that any user-submitted Python code is always prefixed by WebSim™ with the following:

• import scipy as sp # python library that supplements numeric modules
• from numpy import * # library that contains N-dim array object, linear algebra, etc.
• import scipy.stats as ss # import statistical functions
• delay = 1 # alpha delay
• dr = WQSim_DataRegistry.Instance() # dr is initialized as data registry object
• # use dr.GetData() to access market data
• valid = dr.GetData_m_b("USA:TOP3000") # GetData_m_b() accesses Universe data. The result valid is a matrix of Boolean values that specifies which instruments are valid in the universe on a given day. The GetData_m_b() function is used to access data of type Boolean matrix(_m_b stands for matrix, boolean)

Accessing Data

WebSim™ application is enabled to access market data on the back end by using the Python code, GetDataon, the data registry. For example:

• shares_outstanding=dr.GetData("sharesout") # to access outstanding shares data
• ltDebt=dr.GetData("debt_lt") # for long-term debt data
• closePrice=dr.GetData("close") # for close price data

Generate() Function, Indices, and Alpha Expression

As explained before, the market data can be thought of as a matrix of values provided for each stock for every date the data is made available. The dates are mapped to date indices di and the instruments are mapped to instrument indices ii as shown in Table 31.3.

Market data, e.g. close price, would be arranged in the form of a matrix as shown in Table 31.4.

Now to access Apple’s close price on date Jan 7, 2010, we need to use close(3,2).

The sole purpose of the Generate( ) function (should be implemented in your code) is to populate the resulting alpha vector with stock weights for every stock. The Generate function will evaluate the alpha expression for every date (it acts like a loop that iterates through all date indices), hence its arguments are di (which is the date index corresponding to the current date) and alpha (the resulting vector that needs to be filled). For example:

• closePrice = dr.GetData("close")
• def Generate(di,alpha):
• # Alpha expression goes here

Data can be accessed using dataname[di-delay,ii].

dataName[date index, instrument index] would give you the value of dataname, for that particular date, and that particular instrument. To assign expression to an alpha, use alpha[instrument index] = expression.

For example, alpha[ii] = dataname[di,ii].

An example alpha code to use close price data is given below:

• closePrice = dr.GetData("close")
• def Generate(di,alpha):
• alpha[:] = 1./closePrice[di-delay,:]
• # The above statement is equivalent to the expression '1/close'
• # “:” here refers to all instrument indices 0 <= ii< alpha.shape[0]

Since we are using the list-slicing functionality (“:”) of Python, it executes the expression for each instrument (corresponding column cell in the matrix).

Another example that uses returns data to define alpha[:] is given below:

• returnsMat = dr.GetData("returns")
• def Generate(di,alpha):
• alpha[:] = -(returnsMat[di-delay,:]) # reversion(returns always tends to mean)

To access data for an instrument over a certain window period, say n days, use dataname[di-delay-n: di-delay, ii]. An example for this is given below:

• closePrice = dr.GetData("close")
• def Generate(di,alpha):
• alpha[:] = mean(close[di-delay-10:di-delay,:], axis = 0) # taking the mean of the close price for the last 10 days
• alpha[:] = where(valid[di,:],alpha[:],nan) # valid check

The user should note that the window period chosen should be less that the number of lookback days (namely set to 256 days, by default). This value can be retrieved using Python function: Build.GetBackdays().

The last line is added as a validity check for values in the resultant alpha array. This will ensure that the values for instruments that don’t belong to the TOP3000 universe will be filtered out. Notice that this uses the valid variable, which was initialized as mentioned in the Python code header. This is explained further in the next section.

Using Multiple Data Simultaneously

The following example shows us how to access and use multiple data at the same time. Use vectorization wherever possible. Avoid using loops as they are slow. This example uses NumPy’s built-in math function (numpy.subtract is called automatically). Here, the alpha vector is assigned expression close – high.

• # Different data need different variables
• closePrice=dr.GetData("close")
• highPrice=dr.GetData("high")
• def Generate(di,alpha):
• alpha[:] = closePrice[di-delay,:] - highPrice[di-delay,:]

Using Custom-Defined Functions

The following py code shows us how to define custom functions and use them. It also uses the NumPy function numpy.where(). This should be used instead of loops to perform vector comparison.

• closePrice=dr.GetData("close")
• lowPrice=dr.GetData("low")
• def Generate(di,alpha):
• # np_max (defined below) can be called in this code
• alpha[:] = np_max(closePrice[di-delay,:], lowPrice[di-delay-1, :])
• # This is equivalent to the expression 'Max(close, Delay(low, 1))'
• def np_max(data1, data2):
• return where(data1 > data2, data1, data2) # numpy.where() performs vector comparison

Incorrect Way to Assign Alpha Values

• closePrice=dr.GetData("close")
• def Generate(di,alpha):
• alpha[:] = closePrice[di-delay,:] # equivalent to the expression 'close'
• alpha = ones(alpha.shape[0])
• # Tries to assign all ones to the alpha vector using numpy.ones
• # Notice the lack of [:] after “alpha.” This assigns a new object to alpha rather than modify the existing object alpha is pointing to. The effect of this statement is that WebSim™ loses track of the alpha vector
• alpha[:]=1./closePrice[di-delay,:] # equivalent to the expression “1/close”
• # This statement has no effect because of the statement before it. The final alpha value is “close” instead of “1/close”

Use of Valid Matrix

The valid matrix has a list of valid instruments (for example, 3,000 instruments for TOP3000) and is automatically available:

• closePrice=dr.GetData("close")
• def Generate(di,alpha):
• alpha[:] = 1./closePrice[di-delay,:]
• alpha[:] = onlyValid(alpha, di) # User defined function. Filters out values for invalid instruments
• def onlyValid(x, di):
• myValid = valid[di-delay, :] # myValid vector has yesterday’s valid values only
• return where(myValid, x, nan)
• # We use NumPy’s where() function to filter out invalid instruments Always assign numpy.nan() to filter out instruments. “0” is a valid alpha value

The above valid-check function can be inserted at the end of all your alpha codes as:

• alpha[:] = where(valid[di-delay,:],alpha[:],nan)

Using Statistics Functions Available in SciPy

A list of SciPy’s statistical functions can be found at SciPy.org. We will be using the scipy .rankdata() function here. This assigns ranks to alpha weight, dealing with ties appropriately.

• high=dr.GetData("high")
• def Generate(di,alpha):
• alpha[:] = ss.rankdata(high[di-delay, :]) # using SciPy’s rank function. The equivalent expression is Rank(high)
• alpha[:] = where(valid[di-delay,:],alpha[:],nan)

The above alpha expression ranks (on a scale from 0 to 1) adjusted High prices.

The following alpha example shows the usage of a SciPy function called scipy.kurtosis() on close price for 11 days.

• closePrice = dr.GetData("close")
• def Generate(di,alpha):
• alpha[:] = ss.kurtosis(close[di-delay-11:di-delay,:], axis = 0) # using SciPy’s kurtosis function. The equivalent expression is kurtosis(close,11)
• alpha[:] = where(valid[di-delay,:],alpha[:],nan)

The following alpha example shows the usage of a SciPy function called scipy.skewness() on returns for 10 days.

• returns = dr.GetData("returns")
• def Generate(di, alpha):
• alpha[:] = ss.skew (returns[di-delay-10:di-delay,:], axis = 0) # using SciPy’s skewness function. The expression is equivalent to skewness(returns,10)
• alpha[:] = where(valid[di-delay,:],alpha[:],nan)

Using Statistics Functions Available in NumPy

A list of NumPy’s statistical functions can be found here: NumPy.org.

The alpha example below shows usage of NumPy’s mean and standard deviation function numpy.std():

• closePrice = dr.GetData("close")
• def Generate(di,alpha):
• alpha[:] = mean(close[di-delay-5:di-delay,:], axis = 0)/std(close[di-delay-5:di-delay,:], axis = 0)
• # equivalent to expression [Sum(close,5)/5]/ StdDev(close,5)
• alpha[:] = where(valid[di,:],alpha[:],nan)

The alpha example below shows usage of NumPy’s maximum function numpy.amax():

• closePrice = dr.GetData("close")
• def Generate(di, alpha):
• alpha[:] = amax(close[di-delay-20:di-delay,:], axis = 0) # finds maximum close over past 20 days
• alpha[:] = where(valid[di,:],alpha[:],nan)

Accessing and Using Industry Data

• closePrice = dr.GetData("close")
• industry = dr.GetData("industry") # accessing industry data
• def Generate(di,ti,alpha):
• ind = unique(where(industry[di-delay, :] > 0, industry[di-delay, :], 0))
• indclose = zeros(ind.shape[0]) # initialize a NumPy array full of zeros
• for i in xrange(ind.shape[0]):
• indclose[i] = mean(where((industry[di-delay, :] == ind[i]) * (valid[di-delay, :]), closePrice[di-delay, :], 0))
• alpha[:] = where( (industry[di-delay, :] == ind[i]) * (valid[di-delay,:]), indclose[i], alpha[:])

Industry data here is a NumPy array of indices assigned for every available industry. Industries such as Forestry, Metal Mining, Electrical Work, Meat Packing Plants, Textile Mills, Book Printing, etc., have indices assigned to them. The alpha above shows how this industry data can be accessed and used.

Table 31.5 shows a sample of instrument indices, industry indices (values of industry[di-delay,ii]), close data.

For the above alpha example, the industry close array values for each “I” will be as follows:

• Since indclose[i] = Average of closePrice[di-delay,ii] if current instrument belongs to current industry,
  • indclose[0] = (5.0+2.0)/2 = 3.5 (shown in table)
  • indclose[1] = (-2.0-3.0-4.0)/3 = -3
  • indclose[2] = (1.0+5.0)/2 = 3
• Then alpha[ii] is assigned the value of indclose[industry[di-delay,ii]]

Note that you can also access and use sector and subindustry data using dr.GetData() in your alphas.

Simple Linear Regression Model

In the code given below, five days’ values of close price and vwap as training sample are used to calculate regression weights w1 and w2.

• Model formula: close[di] = w1 * close[di-1] + w2* vwap[di-1]
• closePrice=dr.GetData("close")
• vwapPrice=dr.GetData("vwap")
• def Generate(di,alpha):
• CloseX=zeros((5,closePrice.shape[1])) # initialize NumPy array
• CloseY=zeros((5,closePrice.shape[1]))
• VwapX=zeros((5,vwapPrice.shape[1]))
• for dnum in xrange(5):
• CloseX[dnum,:]=closePrice[di-delay-dnum-1,:]
• VwapX[dnum,:]=vwapPrice[di-delay-dnum-1,:]
• CloseY[dnum,:]= closePrice[di-delay-dnum,:]
• for ii in xrange(alpha.shape[0]):
• SampleX=hstack((CloseX[:,[ii]],VwapX[:,[ii]])) # NumPy function hstack() stacks arrays in sequence horizontally(column wise)
• ParaX=LinearRegres(SampleX,CloseY[:,[ii]])
• alpha[ii]=float(ParaX[0])*closePrice[di-delay,ii]+float(ParaX[1])*vwapPrice[di-delay,ii] # equivalent to expression: w1 * close[di-1] + w2* vwap[di-1]
• alpha[:] = where(valid[di,:],alpha[:],nan)
• def LinearRegres(xArr,yArr):
• xMat=matrix(xArr)
• yMat=matrix(yArr)
• xTx= xMat.T*xMat
• res=linalg.det(xTx) # NumPy’s linear algebra function that computes the determinant of the array
• if res==0.0 or isnan(res): # if res is 0 or nan
• return matrix([[nan],[nan]])
• else:
• return xTx.I*(xMat.T*yMat) # matrix.I takes the matrix’s inverse and matrix.T takes its transpose.