Server-Sent Events and fiat currencies, CFDs and futures ticker on an example

Have you ever did something completely different that you initially wanted to do? For me this is frequently the case. But sometimes time is not wasted and I am happy to achieve something or learn something new.

Yesterday I logged in to my laptop for half an hour to check steemit posts and ended up with evening session that took place until 2 AM local time writing SSE client with real time feed for all available fiat currencies, CFDs and derivatives from FPForex.ru. This sounds weird for me even today as yesterday I did not know that SSE exist.

I had been working before on witness ICO price feed for Golos blockchain and noticed that solution provided by @someguy123 in Golosfeed-JS uses cheerio js (https://github.com/cheeriojs/cheerio) to extract data from web page tables that contain different instruments' prices.

I was curious why @someguy123 had to extract those data from tables so I logged in to the website and MY EYES HURT. The design looks sooo 90's and indeed in the footer we see "1995 - 2016" the website has been there for around 20 years ;) I soon noticed it includes real time data at http://www.forexpf.ru/quote_show.php. This looks fast. Almost like Poloniex ;) but there is no rest API.

I suspected that this must be some ancient AJAX code I started the debugger and whoops: inside the php page sits an iframe (another web page) generated server side using "JSP" (Java server pages). This thing contains some args after url. Digging deeeeper I found out that there are javascript modules that capture an access token from iframe url address and subscribe to SSE server to fetch data and update the tables.

Then I thought..."What the heck are Server-Sent Events (SSEs)", it is not AJAX, not websockets and found an article here https://www.html5rocks.com/en/tutorials/eventsource/basics/ with a quite good introduction to the subject.

And I came up with a crazy idea to catch this token and subscribe to a (sub)set of available instruments to get live ticker for fiat currencies, CFDs and futures directly from the server. Of course similar data are provided by brokers like IB, OANDA or other ones but 1) this was rather to solve a challenge 2) the website has a few Russian based instruments and currency pairs not easily available elsewhere.

And guess what - I succeeded. The outcome of this reverse-engineering exercise is an experimental tool https://github.com/cryptomental/social-chains-experiments/blob/master/fpticker.py that:

1. Opens a keep-alive session and grabs SID token
2. Subscribes to any instrument, subset, or all of them
3. Streams ticker data in raw SSE format or dict/JSON

The available instruments are:

* INDEXES_CFD = ["DJIA", "SP500", "NASD_COMP", "NASD100", "FTSE100",
                   "DAX", "AEX" "CAC40", "SMI", "RTST", "USD_INDEX"]
* STOCKS = ["GAZ_ADR", "GMK_ADR", "LUK_ADR", "ROS_ADR", "RTK_ADR",
              "SNG_ADR", "TAT_ADR", "VTB_ADR"]
* METALS = ["GOLD", "SILVER", "PLATINUM", "PALLADIUM", "ALUM",
              "COPPER", "NICKEL", "BRENT", "LIGHT"]
* RUB = ["USDRUB", "EURRUB"]
* MINI = ["NASDAQ", "SnP500"],
* CURRENCY_CFD = ["AUD/JPY", "AUD/USD", "CAD/JPY", "CHF/JPY",
                    "EUR/AUD", "EUR/CAD", "EUR/CHF", "EUR/GBP",
                    "EUR/JPY", "EUR/USD", "GBP/CHF", "GBP/JPY",
                    "GBP/USD", "USD/CAD", "USD/CHF", "USD/JPY"]
* CURRENCY_FUTURES = ["AUD_FUT", "CAD_FUT", "CHF_FUT", "EUR_FUT",
                        "GBP_FUT", "JPY_FUT"]
* INDEX_FUTURES = ["DJIA_FUT", "SP500_FUT", "NASD100_FUT", "NIK_FUT"]

Raw stream example for currencies:

1;S=USDRUB;T=14:33:46;B=+64.398;NCH=0.627;NCHP=0.98;A=+64.408
1;S=EURRUB;T=14:33:57;B=-68.458;NCH=0.685;NCHP=1.01;A=-68.478
1;S=EURRUB;T=14:33:57;B=-68.457;NCH=0.684;NCHP=1.01;A=-68.477
1;S=EURRUB;T=14:33:58;B=+68.458;NCH=0.685;NCHP=1.01;A=+68.478
1;S=EURRUB;T=14:33:59;B=-68.458;NCH=0.685;NCHP=1.01;A=-68.478

Metals, stocks:

1;S=ROS_ADR;T=14:34:17;B=+5.330;NCH=0.030;NCHP=0.57;A=5.335
1;S=RTK_ADR;T=14:09:55;B=7.70;NCH=-0.11;NCHP=-1.41;A=+7.74
1;S=SNG_ADR;T=14:34:30;B=4.549;NCH=-0.041;NCHP=-0.89;A=+4.553
1;S=TAT_ADR;T=14:35:22;B=36.46;NCH=0.28;NCHP=0.77;A=-36.49
1;S=VTB_ADR;T=14:35:29;B=2.083;NCH=0.000;NCHP=0.00;A=-2.085
1;S=GOLD;T=14:35:29;B=+1211.71;NCH=-0.52;NCHP=-0.04;A=+1212.01
1;S=SILVER;T=14:27:26;B=+16.64;NCH=0.00;NCHP=0.00;A=+16.67
1;S=PLATINUM;T=13:54:22;B=+936.00;NCH=0.00;NCHP=0.00;A=+946.00
1;S=PALLADIUM;T=14:35:23;B=-743.30;NCH=1.70;NCHP=0.23;A=-746.30
1;S=ALUM;T=14:25:00;B=-1760.00;NCH=0.00;NCHP=0.00;A=-1763.00
1;S=COPPER;T=14:33:08;B=-5592.00;NCH=-20.00;NCHP=-0.36;A=-5597.00
1;S=NICKEL;T=14:18:30;B=+11302.00;NCH=4.00;NCHP=0.04;A=+11402.00
1;S=BRENT;T=14:35:16;B=-49.30;NCH=0.27;NCHP=0.55;A=-49.33
1;S=LIGHT;T=14:34:49;B=+48.20;NCH=0.30;NCHP=0.63;A=+48.23

Dict/JSON:

{'A': '+1212.44',
 'B': '+1212.14',
 'NCH': '-0.09',
 'NCHP': '-0.01',
 'S': 'GOLD',
 'T': '14:42:52'}
{'A': '+16.67',
 'B': '+16.64',
 'NCH': '0.00',
 'NCHP': '0.00',
 'S': 'SILVER',
 'T': '14:40:06'}
{'A': '+946.00',
 'B': '+936.00',
 'NCH': '0.00',
 'NCHP': '0.00',
 'S': 'PLATINUM',
 'T': '13:54:22'}

These are just static snapshots, normally data is streamed real time. Looks pretty cool.
Treat this implementation as an experiment, or a toy, but you can access real time data with it.

---

This post might be too technical, but well, I am writing about some original discoveries here and actually I am writing this not to forget afterwards. I think I found another application for steemit being a diary ;)

---

A few references:

• Server-Sent events specification

https://html.spec.whatwg.org/multipage/comms.html#server-sent-events

• What the heck are Server-Sent Events (SSEs)

https://www.html5rocks.com/en/tutorials/eventsource/basics/