Deniz's Notes: Mash-up technologies for a simple market data consumer

In this text i use following open source java libraries to simulate a simple market data consumer application:

Spring : Dependency injection
JeroMQ(ZeroMQ) : Pub/Sub between Provider and Aggregator
Chronicle Queue: Messaging between Aggregator and Writer
Smile Jackson: Object Serialization
Apache Commons Pool2 : Object Pooling
Guava: AtomicDouble
SLF4J and logback : Logging framework
Java 7 and Maven

Market Data Consumer

This application simulates a market data consumer. The ecosystem consists of two providers of market data, one aggregator service that consumes data from the providers and one file writer that writes the results to a CSV file. The communication between the components should occur using a messaging framework(JeroMQ and Chronice Queue).

Provider

The two providers will publish data that conforms to the following interface:

interface IMarketValue

{

String getProviderId ();

String getName();

double getValue();

}

Lets assume that providers are named “Orion“ and “Polaris”. The lists of instrument names for Orion are:

AAA

BBB

CCC

DDD

EEE

FFF

The instrument names for Polaris are:

AAA

BBB

CCC

GGG

HHH

III

JJJ

Providers generate random numbers (from 5.5 to 10.5) for the values as quick as possible without no throttling.

Aggregator

Aggregator service listens to data from both providers. For instruments that are only available in one provider, the aggregator should pass the data down as is to the file writer. For instruments that are available from both providers, the aggregator should take the average of the last incoming data from both providers. E.g.

After receiving:

Orion – AAA – 6.3

Orion – FFF – 7.0

Polaris – AAA – 7.0

Polaris – BBB – 8.0

the aggregator should pass down:

AAA – 6.65

FFF – 7.0

BBB – 8.0

However, if Orion then publishes:

Orion – BBB – 7.0

the next aggregator output should contain:

BBB – 7.5

The aggregator contains a dependency while processing messages which introduces a 1 second delay to the processing loop which must be simulated. Newest available data should be always processed for a given Name in the aggregator at any time. I.e. if the aggregator is stuck in a processing loop and 5 updates come through for AAA, the next processing run should use the latest available data for AAA.

File Writer

The file writer outputs in the following format and the entries should be in alphabetical order. Each new set of values should create a new file and there should be only one entry for each instrument.

Instrument,Value

AAA,6.65

BBB,5.5

FFF,7.0

Every time the file writer receives a new set of values it should create a new version of the file. Should the file writer not be able to write to the output file, then an alternate filename should be used, but once the file is once more accessible it should return to the original filename. It should also not create many files and only a new file when a file is not accessible.