"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization.
I guess trading and networking do have many unseen similarities.
"Below the 950-millisecond level, where computerized trading occurs so quickly that human traders can't even react, no fewer than 18,520 crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well. Start looking at 1sec intervals and you will see spikes that hit 100% of capacity - even on networks running at 25% average utilization
I've had great success using appliances from network monitoring vendor Corvil - http://www.corvil.com/ and TS-A's TipOff - www.ts-a.com/TipOff/tipoff.html
Both can decode most market data feeds formats and drill down to provide a slew of details when trying to debug jitter and latency on networks where sub-millisecond thresholds are essential for analysis.
Disclaimer: I'm no way affiliated with any of these companies or products.
Tieing the two together, this post shows how a lot of 'conventional'
network thinking needs to be turned on its head when it comes to networks
for trading floors:
> Nanosecond Trading Could Make Markets Go Haywire
> Nanosecond Trading Could Make Markets Go Haywire | WIRED
>
> "Below the 950-millisecond level, where computerized trading occurs so
> quickly that human traders can't even react, no fewer than 18,520
> crashes and spikes occurred."
>
> Anyone who has managed a network knows that when you look at your
> MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well.
> Start looking at 1sec intervals and you will see spikes that hit 100% of
> capacity - even on networks running at 25% average utilization.
>
> I guess trading and networking do have many unseen similarities.
Tieing the two together, this post shows how a lot of 'conventional'
network thinking needs to be turned on its head when it comes to networks
for trading floors:
This all is very familiar to anyone who's looked at ethernet (or other
networks) for real-time control purposes, such as flight control of
aircraft or rockets or for autos or other ground vehicles.
Though the finance people are pushing it a lot more than the rocket
and aircraft control people I know... I guess markets crash faster
than rockets!
"Below the 950-millisecond level, where computerized trading occurs so
quickly that human traders can't even react, no fewer than 18,520
crashes and spikes occurred."
Anyone who has managed a network knows that when you look at your
MRTG/Cacti graphs at 5min, 10min ,15min intervals - all looks well.
Start looking at 1sec intervals and you will see spikes that hit 100% of
capacity - even on networks running at 25% average utilization.
given a serialized interface the network is 100% utilized everytime a
packet is sent, if you're measuring at the microsecond level it's far
more germain what the queue depth is rather than whether a packet is
currently on the wire or not. A 5 minute 1 minute or 1 second average is
a pretty good measure of how much of the time it wasn't being utilitized
during the same interval.
The fm4000 based Aristas have hardware hooks to stream the latency data
in the form of queue thresholds at you via LANZ, that can take your
visibility down to 800 or so usec snapshot of the queue. The broadcom
devices doesn't have a comparable functionality.
I guess trading and networking do have many unseen similarities.
I'd be careful about analogizing them to much. high speeding trading
strategies to a very large extent depend on the high speed parties being
about to make more granular decisions faster that other participants can
complete transactions, forwarding engines are mostly looking to get rid
of packets as expeditiously as possible.
Anecdotally, I had an interview years ago for a small-ish futures trading company based in London. The interviewer had to pause the interview part way through whilst he investigated a 10ms latency spike that the traders were noticing on a short point-to-point fiber link to the London Stock Exchange. He commented that the traders were far better at 'feeling' when an connection was showing even a trace of lag compared to normal than anything he'd set up by way of monitoring (not sure how good his monitoring was, though.)
This was my experience in a callcenter as well; network type problem reports
always came in from the floor managers before Nagios came forth with an
opinion.
When I used to run an ISP network, our NOC always talked about "that
porn guy" who would call the *exact* *momment* the NNTP server had any
type of stutter...
I guess there's always a canary for the coal mine -- eh?
Anecdotally, I had an interview years ago for a small-ish futures
trading company based in London. The interviewer had to pause the
interview part way through whilst he investigated a 10ms latency spike
that the traders were noticing on a short point-to-point fiber link to
the London Stock Exchange. He commented that the traders were far
better at 'feeling' when an connection was showing even a trace of lag
compared to normal than anything he'd set up by way of monitoring (not
sure how good his monitoring was, though.)
This was my experience in a callcenter as well; network type problem reports
always came in from the floor managers before Nagios came forth with an
opinion.
This has nothing to do with a gut feeling or instinct. Trading companies today monitor P&L near realtime and traders will begin to experience low fill rates or worse be rejected by trading counter parties when prices are too far off or out of the money. The longer a system takes to responds to market quotes the lower fills rates they begin to notice and higher execution costs. Trades today in the equity markets must be within the national best bid, best offer price range or companies can be fined by the SEC which is why latency an jitter can be problematic in financial networks.
Am I the only one who thinks that if network jitter can make you fall outside
the acceptable price window, maybe, just maybe, the market is just too damned
volatile for its own good?
I've had an interesting discussion with some financial heads about
a simple idea.
What if the exchange, on every inbound trade, inserted a random
delay, say between 0 and 60 seconds, before processing it?
Almost all of this computer based, let's be closer to the exchange stuff
becomes junk, immediately. Anyone "long" (where long is probably more
than 10 minutes, with a 60 second jitter) in a security wouldn't notice.
I mean, if the general public has to get 15 minute delayed quotes so
they don't manipulate the market, shouldn't the big guys?
See what happened on Wednesday with Apple's stock. With no good cause it looks like various parties started to try and short it. You can see the initial result from 12pm->1pm, the 'quick buck' 1pm-1:30pm rise, then the start of some more shorting at which point you can see the pattern emerge where the automatic trading algorithms started doing their thing. Definitely too volatile.
Some longer term players, will use delayed data as they are trading longer
term, and dont care too much so if the orders were delayed a bit more,
these players most likely wouldn't care/notice.
But also you have to consider, there are a large degree of shorter term
players, who are in/out of the market and play both sides, these do have
real-time data feeds, and do care about latency. Some shops go as far as to
only use a certain length patch cables from their trading PC to the switch
port they are connected to. Also consider when news releases are announced,
the markets often do move quite fast, and a LOT of money can be made/lost
in seconds, so delaying the orders, could and would affect the outcome of
the trades.
Also consider that a vast majority of the trades are automated by
computers, and some want their servers setup as close to the exchange as
possible, in fact the CME group released that they will offer/lease data
center space:
"One such project is a 428,000-square-foot data center in the western
suburbs of Chicago opened by the CME Group, which owns the Chicago
Mercantile Exchange<http://topics.nytimes.com/top/reference/timestopics/organizations/c/chicago_mercantile_exchange/index.html?inline=nyt-org>.
It houses the exchange’s Globex electronic futures and options trading
platform and space for traders to install computers next to the exchange’s
machines, a practice known as co-location — at a cost of about $25,000 a
month per rack of computers."
