One of the biggest and fastest traders in financial markets is abandoning its core business after its revenue engine stalled,