The goal of the event-based backtester is to achieve better accuracy in the trading arena. We will consider the internals of the trading system by using the trading system we built in the last chapter and we will use the market simulator to simulate the external constraints of the market.
In this section, we will create an EventBasedBackTester class. This class will have a queue between all the components of the trading systems. Like when we wrote our first Python trading system, the role of these queues is to pass events between two components. For instance, the gateway will send the market data to the book through a queue. Each ticker (price update) will be considered an event. The event we implemented in the book will be triggered each time there is a change in the top of the order book. If there is a change in the top of the book, the book will pass a book event, indicating that there is a change in the book. This queue will be implemented using the deque from the collection library. All the trading object components will be linked to one another by these queues.
The input for our system will be the Yahoo finance data collected by the panda DataReader class. Because this data doesn't contain any orders, we will change the data with the process_data_from_yahoo function. This function will use a price and will convert this price to an order.
The order will be queued in the lp_2_gateway queue. Because we need to fake the fact that this order will disappear after each iteration, we will also delete the order. The process_events function will ensure that all the events generated by a tick have been processed by calling the call_if_not_empty function. This function has two arguments:
- A queue: This queue is checked if empty. If this queue is not empty, it will call the second argument.
- A function: This is the reference to the function that will be called when the queue is not empty.
We will now describe the steps we will take to build the event-based backtester.
- In the following code, we will import the objects we created during Chapter 7, Building a Trading System in Python. We will use the trading system we built as a backtester:
from chapter7.LiquidityProvider import LiquidityProvider
from chapter7.TradingStrategyDualMA import TradingStrategyDualMA
from chapter7.MarketSimulator import MarketSimulator
from chapter7.OrderManager import OrderManager
from chapter7.OrderBook import OrderBook
from collections import deque
import pandas as pd
import numpy as np
from pandas_datareader import data
import matplotlib.pyplot as plt
import h5py
- To read all the elements from a deque, we will implement the call_if_not_empty function. This function will help to call a function as long as a deque is not empty:
def call_if_not_empty(deq, fun):
while (len(deq) > 0):
fun()
- In the code, we will implement the EventBasedBackTester class. The constructor of this class will build all the deque needed to have all the components communicate. We will also instantiate all the objects in the constructor of EventBasedBackTester:
class EventBasedBackTester:
def __init__(self):
self.lp_2_gateway = deque()
self.ob_2_ts = deque()
self.ts_2_om = deque()
self.ms_2_om = deque()
self.om_2_ts = deque()
self.gw_2_om = deque()
self.om_2_gw = deque()
self.lp = LiquidityProvider(self.lp_2_gateway)
self.ob = OrderBook(self.lp_2_gateway, self.ob_2_ts)
self.ts = TradingStrategyDualMA(self.ob_2_ts, self.ts_2_om,
self.om_2_ts)
self.ms = MarketSimulator(self.om_2_gw, self.gw_2_om)
self.om = OrderManager(self.ts_2_om, self.om_2_ts,
self.om_2_gw, self.gw_2_om)
- The process_data_from_yahoo function will convert the data created by the panda DataReader class to orders that the trading system can use in real time. In this code, we will create a new order that we will then delete just after:
def process_data_from_yahoo(self,price):
order_bid = {
'id': 1,
'price': price,
'quantity': 1000,
'side': 'bid',
'action': 'new'
}
order_ask = {
'id': 1,
'price': price,
'quantity': 1000,
'side': 'ask',
'action': 'new'
}
self.lp_2_gateway.append(order_ask)
self.lp_2_gateway.append(order_bid)
self.process_events()
order_ask['action']='delete'
order_bid['action'] = 'delete'
self.lp_2_gateway.append(order_ask)
self.lp_2_gateway.append(order_bid)
- The process_events function will call all the components as long as we have new orders coming. Every component will be called as long as we didn't flush all the events in the deque:
def process_events(self):
while len(self.lp_2_gateway)>0:
call_if_not_empty(self.lp_2_gateway,
self.ob.handle_order_from_gateway)
call_if_not_empty(self.ob_2_ts,
self.ts.handle_input_from_bb)
call_if_not_empty(self.ts_2_om,
self.om.handle_input_from_ts)
call_if_not_empty(self.om_2_gw,
self.ms.handle_order_from_gw)
call_if_not_empty(self.gw_2_om,
self.om.handle_input_from_market)
call_if_not_empty(self.om_2_ts,
self.ts.handle_response_from_om)
- The following code will instantiate the event-based backtester by creating the eb instance. Because we are going to load the same GOOG financial data, we will use the load_financial_data function. Then, we will create a for-loop backtester where will feed, one by one, the price updates to the event-based backtester:
eb=EventBasedBackTester()
def load_financial_data(start_date, end_date,output_file):
try:
df = pd.read_pickle(output_file)
print('File data found...reading GOOG data')
except FileNotFoundError:
print('File not found...downloading the GOOG data')
df = data.DataReader('GOOG', 'yahoo', start_date, end_date)
df.to_pickle(output_file)
return df
goog_data=load_financial_data(start_date='2001-01-01',
end_date = '2018-01-01',
output_file='goog_data.pkl')
for line in zip(goog_data.index,goog_data['Adj Close']):
date=line[0]
price=line[1]
price_information={'date' : date,
'price' : float(price)}
eb.process_data_from_yahoo(price_information['price'])
eb.process_events()
- At the end of this code, we will display the curve representing the cash amount within the trading period:
plt.plot(eb.ts.list_total,label="Paper Trading using Event-Based BackTester")
plt.plot(eb.ts.list_paper_total,label="Trading using Event-Based BackTester")
plt.legend()
plt.show()
The new code that we introduce in this section is the code for the trading strategy. Our first trading strategy that we implemented in our trading system was an arbitrage strategy. This time, we will continue the example of the dual-moving average trading strategy.
This code shows that the logic of the trading strategy uses the same code as the for-loop backtester. The create_metrics_out_of_prices and buy_sell_or_hold_something functions are untouched. The main difference is regarding the execution part of the class. The execution takes care of the market response. We will be using a set of variables related to the paper trading mode to show the difference between actual and paper trading. Paper trading implies that every time the strategy sends an order, this order is filled at the price asked by the trading strategy. On the other side of the coin, the handle_market_response function will consider the response from the market to update the positions, holdings, and profit and loss.
- We will code the TradingStrategyDualMA class inspired by the TradingStrategy class that we coded in Chapter 7, Building a Trading System in Python. This class will take care of keeping track of two series of values, the values for paper trading and the values for backtesting:
class TradingStrategyDualMA:
def __init__(self, ob_2_ts, ts_2_om, om_2_ts):
self.orders = []
self.order_id = 0
self.position = 0
self.pnl = 0
self.cash = 10000
self.paper_position = 0
self.paper_pnl = 0
self.paper_cash = 10000
self.current_bid = 0
self.current_offer = 0
self.ob_2_ts = ob_2_ts
self.ts_2_om = ts_2_om
self.om_2_ts = om_2_ts
self.long_signal=False
self.total=0
self.holdings=0
self.small_window=deque()
self.large_window=deque()
self.list_position=[]
self.list_cash=[]
self.list_holdings = []
self.list_total=[]
self.list_paper_position = []
self.list_paper_cash = []
self.list_paper_holdings = []
self.list_paper_total = []
- For each tick received, we will create a metric to make decisions. In this example, we use the dual-moving average trading strategy. Therefore, we will use two moving averages that we will build tick by tick. The create_metric_out_of_prices function calculates the short and long moving averages:
def create_metrics_out_of_prices(self,price_update):
self.small_window.append(price_update)
self.large_window.append(price_update)
if len(self.small_window)>50:
self.small_window.popleft()
if len(self.large_window)>100:
self.large_window.popleft()
if len(self.small_window) == 50:
if average(self.small_window) >
average(self.large_window):
self.long_signal=True
else:
self.long_signal = False
return True
return False
- The buy_sell_or_hold_something function will check whether we have a long signal or a short signal. Based on the signal, we will place an order and we will keep track of the paper trading position, cash, and profit and loss. This function will also record the value of the backtested values of position, cash, and profit and loss. We will keep track of these values to create a chart of our trading execution.
def buy_sell_or_hold_something(self, book_event):
if self.long_signal and self.paper_position<=0:
self.create_order(book_event,book_event['bid_quantity'],'buy')
self.paper_position += book_event['bid_quantity']
self.paper_cash -= book_event['bid_quantity'] * book_event['bid_price']
elif self.paper_position>0 and not self.long_signal:
self.create_order(book_event,book_event['bid_quantity'],'sell')
self.paper_position -= book_event['bid_quantity']
self.paper_cash -= -book_event['bid_quantity'] * book_event['bid_price']
self.paper_holdings = self.paper_position * book_event['bid_price']
self.paper_total = (self.paper_holdings + self.paper_cash)
self.list_paper_position.append(self.paper_position)
self.list_paper_cash.append(self.paper_cash)
self.list_paper_holdings.append(self.paper_holdings)
self.list_paper_total.append(self.paper_holdings+self.paper_cash)
self.list_position.append(self.position)
self.holdings=self.position*book_event['bid_price']
self.list_holdings.append(self.holdings)
self.list_cash.append(self.cash)
self.list_total.append(self.holdings+self.cash)
- As shown, the signal function will call the two prior functions:
def signal(self, book_event):
if book_event['bid_quantity'] != -1 and
book_event['offer_quantity'] != -1:
self.create_metrics_out_of_prices(book_event['bid_price'])
self.buy_sell_or_hold_something(book_event)
- The following function differs from the original function execution that we implemented in Chapter 7, Building a Trading System in Python. This one will keep track of the profit and loss, position, and the cash:
def execution(self):
orders_to_be_removed=[]
for index, order in enumerate(self.orders):
if order['action'] == 'to_be_sent':
# Send order
order['status'] = 'new'
order['action'] = 'no_action'
if self.ts_2_om is None:
print('Simulation mode')
else:
self.ts_2_om.append(order.copy())
if order['status'] == 'rejected' or
order['status']=='cancelled':
orders_to_be_removed.append(index)
if order['status'] == 'filled':
orders_to_be_removed.append(index)
pos = order['quantity'] if order['side'] == 'buy' else
-order['quantity']
self.position+=pos
self.holdings = self.position * order['price']
self.pnl-=pos * order['price']
self.cash -= pos * order['price']
for order_index in sorted(orders_to_be_removed,reverse=True):
del (self.orders[order_index])
- As shown, the following function will handle the market response:
def handle_market_response(self, order_execution):
print(order_execution)
order,_=self.lookup_orders(order_execution['id'])
if order is None:
print('error not found')
return
order['status']=order_execution['status']
self.execution()
- The following function will return the profit and loss of the strategy:
def get_pnl(self):
return self.pnl + self.position * (self.current_bid + self.current_offer)/2
When we run this example, we will obtain the following chart. We can observe that the curve is the same as the prior one. This means that the trading system that we created and the paper trading have the same reality:
We will now modify the market assumptions by changing the fill ratio used by the market simulator. We are getting a fill ratio of 10%, and we can see that the profit and loss is profoundly impacted. Since most of our orders are not filled, we will not make money where the trading strategy was supposed to make money:
The chart reminds us of the importance of having a fast system. If we place an order, in most cases, the order is rejected. This will negatively impact the profit and loss of the trading strategy.