AlgoTraderAlgoTrader Documentation

Chapter 17. Market Data

17.1. Creation of Bars based on Ticks
17.2. Creation of Bars based on Bars
17.2.1. Esper Bar Aggregation
17.2.2. Java Bar Aggregation
17.3. Numeric Precision
17.4. Price normalization
17.5. Market Data Gap Checking
17.6. Generic Events

AlgoTrader provides Market Data Interfaces with the following market data providers:

AlgoTrader allows having multiple market data interfaces active at the same time so market data received from different market data providers can be compared in real-time.

To enable either of those Market Data Interfaces the following two steps have to be executed:

  1. The correct Spring Profile has to be activated according to Section 25.1, “Starter Classes”

  2. For Bloomberg market-data-bb and for InteractiveBrokers market-data-ib has to be added to the VM argument server-engine.init when running the AlgoTrader server.

Market Data Events itself are available in different types:

As the following diagram shows, market data providers deliver Price Events (TradeVO, BidVO & AskVO) or individual TickVO Fields.

In back testing AlgoTrader supports both Ticks or Bars. In both live trading and back testing Price events and Tick events can be aggregated into Bar events

Market Data Event Types

Figure 17.1. Market Data Event Types


Inside each strategy the MarketDataCacheService keeps a copy of the last MarketDataEvent for each Security. For further details see Section 7.2.17, “Market Data Cache Service”.

Through the live data recording feature (see Section 18.2, “Live Data Recording”) tick data can automatically be aggregated into bar-data.

In addition, it is possible to aggregate ticks into bars through Esper code in both Simulation and Live Trading by using an Esper time_batch window:

select
  first(last) as open,
  max(last) as high,
  min(last) as low,
  last(last) as close
from
  TickVO.win:time_batch(1 min)
group by
  securityId;

and expr_batch for constant volume bars:

select
  max(last) as high,
  min(last) as low,
  first(last) as open,
  last(last) as close
from
  TickVO.win:expr_batch(sum(volume) > 1000)
group by
  securityId;

Warning

Some market data providers (e.g. Interactive Brokers) will only provide market data snapshots at regular time intervals. This can cause deviations in the bar high and low price.

Due to clock differences between the local machine and the market data provider's servers there might be slight deviations in the bar open and close price. It is therefore important to enable system clock synchronization on the server where AlgoTrader is installed.

You might need to have more than one aggregation level of bars in your strategy. For this you can initially create the smallest bar size you require according to Section 18.2, “Live Data Recording” and then aggregate these bars to get larger bars.

There are several ways to do this:

  • Using Esper

  • Using Java

In General different Securities are traded with different numeric precision (e.g. S&P Futures prices are 2 digits, whereas FX prices are usually 5-6 digits and crypto currencies are up to 8 digits). To accommodate different numeric precisions, AlgoTrader provides the following fields inside the class Security:

There are also other parameters that describes order minimums and maximums for quantity and price. They are described in section: Section 22.2.2, “Crypto-Order Constraints”

Price normalization comes into play when multiple market data and/or trading adapters are in use that use different price multipliers / contract sizes for the same instrument. For example one adapter might quote prices in dollars with a contract size of 10 where as another one might quote them in cents with a contract size of 1000. The default contract size will be stored in the security_family table.

Since a continuous data feed of market data is essential for most trading strategies, AlgoTrader contains a feature that automatically warns if no market data has been received for a prolonged period of time. For this purpose the class Security has a property maxGap, that defines the maximum number of minutes allowed without any market data updates. This is enforced by the CHECK_SECURITY_TICK_GAPS statement which can be turned on by changing the following property inside conf-core.properties. Alternatively the properties can be changed via Section 2.3, “VM Arguments”

# enables security tick gap check
statement.securityTickGap = true

Crypto currency exchanges are typically using web sockets to deliver market data. Web socket connections are typically not very stable and it can happen that a web socket connection disconnects or suddenly stops receiving data. For this purpose AlgoTrader has a feature that automatically reconnects the corresponding adapter if no market data has been received for a prolonged period of time. This is enforced by the CHECK_ADAPTER_TICK_GAP statement which can be turned on by changing the following property inside conf-core.properties. Alternatively the properties can be changed via Section 2.3, “VM Arguments”

# enables adapter tick gap check
statement.adapterTickGap = true

In addition to MarketDataEvents (i.e. TickVOs and BarVOs) there are general purpose Generic Events that can contain any type of information (e.g. virtual market data, signals, exposure values, etc.). A Generic Event class has to subclass GenericEventVO. Subscription to Generic Events is based on the class of the Generic Event. There are two typical use cases for Generic Events:

  • External data: For example feeding Corporate Actions events from external data provider.This use case requires an external data adapter to be implemented. AlgoTrader has an integrated dividend data feed to demonstrate the feature (through the Intrinio data provider). A strategy can subscribe to these events and use them as market signals or inputs. Our Appendix K, Example Strategy "Dividend Capture" shows how this is done.

    Example: To subscribe to a Generic Event of type DividendEventVO (which is a subclass of GenericEventVO), the following needs to be done within the strategy:

    getGenericEventsService().subscribe("MSFT", MarketDataEventType.DIVIDENDS, 
    
                                                               getStrategy(), "INTR", Collections.singletonMap("selector", "MSFT"));
    public void onGenericEvent(final GenericEventVO event) {
    
                            if (event instanceof DividendEventVO) { //handle Dividend event }}

    In distributed mode AlgoTrader must be strarted with the iNTRDividendGenericEvents Spring profile which would enable the dividend data feed while the strategy must be started with genericEventsService profile, which enables dividend data propagation between server and strategy.

  • Strategy to strategy communication: two (or more) strategies can exchange information between each other. A custom object extending GenericEventVO must be introduced and both strategies must have references to it.

    A subscriber strategy can subscribe to particular events in the following way:

    getSubscriptionService().subscribeGenericEvents(Collections.singleton(CustomEventVO.class))

    A publisher strategy can emit custom events like this in distributed mode:

    eventDispatcher.broadcast(customEvent, EventRecipient.REMOTE_ONLY);

    And in embedded mode:

    eventDispatcher.broadcast(customEvent, EventRecipient.ALL_LOCAL_LISTENERS)

    Alternatively instead of broadcasting, it is possible to send the events directly to the desired strategy as follows:

    eventDispatcher.sendEvent(strategy.getName(), customEvent);

    If the event is sent directly to the strategy, the target strategy doesn't need to be explicitly subscribed to this event.

All generic events can be persisted in InfluxDB by setting the following property in conf.properties:

#Defines whether or not to record generic events dataSource.persistGenericEvents = true

Alternatively it can be overriden as a VM parameter.

Downloaded (or recorded) generic events can be used in simulations (both InfluxDB and CSV files are supported).

It is also possible to feed Generic Events via CSV Files or InfluxDB in Simulation Mode. The following properties control the data sources:

#sample CSV dataSource configuration
dataSource.0.subsetName = currentTick
dataSource.0.dataSourceType = CSV
dataSource.0.eventType = TICK
dataSource.0.dataFilesDir = files
dataSource.0.feedAllFiles = false

#sample Influx dataSource configuration
dataSource.1.dataSourceType = DB
dataSource.1.eventType = DIVIDENDS
            

The configuration above will feed tick data from CSV and dividend data from InfluxDB. See more sample configurations in conf.properties

The file name of the CSV File has to be according to this schema:

<className>.<rank>.csv

  • className is the fully qualified class name (e.g. ch.algotrader.event.Signal)

  • rank is the sort order for situations where there are multiple GenericEvent types for the same time stamp