"Good intentions never work, you need good mechanisms to make anything happen." says Jeff Bezos.

Exchange service workflow and integration to OCAP service

Author Lei Zhou (Software Engineer at ArcBlock)

This year, I was fortunate enough to be able to join ArcBlock as software engineer. This is one of the most exciting experience to work on OCAP service. OCAP service is rolling out some new features aimed at making OCAP service more efficient and more powerful. In July, OCAP and OCAP Playground launched the realtime coin price feature with exchange service, which was used to collect and provide the realtime price of several cryptocurrencies for other services. Today, we will go through the whole workflow of Exchange service and integration to OCAP service.

What is the architeture?

Exchange service is an Elixir application to collect, parse and save cryptocurrency data. Elixir has been a "first class" language at ArcBlock since day one. The service is divided into four parts which are Exchanges API modules, Scheduler, Publisher and Filesystem.

Exchanges API modules

Currently, we collect the cryptocurrency data from 6 exchanges, and those exchanges have different kind of interfaces and data structures. In order to handle this situation, there are 6 exchanges API modules to gather and format the cryptocurrency data from exchanges, and each module focuses on one exchange. After the data is processed by those modules, the format of parsed data is unified.

Scheduler

For now, we are retrieving BTC and ETH realtime coin price from exchanges. An influx of those data is as the input of Exchange service. There is a scheduler used to trigger the publisher periodically to collect the new data from exchanges. If you want to perform periodic tasks, there are some libraries that make this easier for you (but they themselves use GenServers under the hood!), like Quantum. In Exchange service we use Quantum, a cron-like job scheduler of Elixir.

Publisher

We use Phoenix PubSub to broadcast messages, and use GenServer to handle the events. GenServers are essential part of OTP, which simplifies repeating task and letting programmer concentrate on the logic of the applciation. GenServers are used a lot in the Elixir and Erlang world to separate the interface-logic (like a webserver, a terminal-reader/writer or a GUI) from the business logic.

The idea behind pub-sub part is rather simple - start separate process(exchanges API modules), wait for the parsed crypto data returned by these modules, then group the data from 6 exchanges API modules together, and boardcast the data to filesystem and Amazon Kinesis.

Amazon Kinesis make it easy to collect, process and analyze real-time, streaming data, so we can get timely insights and serve with Exchange service, instead of having to wait until all data is collected before the processing can begin.

Filesystem and LKG

Publisher boardcasts data to filesystem, which sync to Amazon S3. Amazon S3 are file storage to help us with backup, archive, recovery.

When there is an issue such as exchange API request error, Last known Good is a good problem recovery option to store the lastest data for each exchange.

Why monitor & analyse service?

We use Datadog to build real-time interactive dashboard and send mertics and events for manipulation and graphing.

At the service level, the purpose of monitoring and evaluation is to track implementation and outputs systematically, and measure the effectiveness of service. It helps to determine exactly when a new version deploy and when changes may be needed. Monitoring and evaluation forms the basis for modification to let us make informed decisions.

For metrics, I use Statix to gather stats and some custom metrics, which are sent to StatsD, and from there to datadog, where I can visualize them.

How to integrate to OCAP service？

Since our OCAP service uses GraphQL. With GraphQL, there is no need for versioning as we can easily add new fields and types to our GraphQL API without impacting existing queries to serving a versionless API. I added a scheduled job, which periodcally retrieved the latest cryptocurrency data from Amazon Kinesis, and a resolver which can provide the real-time price for related fields.

Learning more about ArcBlock Exchange service

I hope you enjoyed seeing how Exchange service working, and how integerate to OCAP service. More cryptocurrencies supported on our Exchange service are on the way.

If you want to work with a high performing engineering organization, you should join ArcBlock!