Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
German stock trading platform Xetra down, all securities affected (thestarphoenix.com)
151 points by ciceryadam on July 1, 2020 | hide | past | favorite | 100 comments


Trading has continued hours ago, probably even before this article was even published.

In a first statement, they blamed algo-traders which created (and within milliseconds deleted) a ton of pointless orders, like buying Wirecard for 0,05EUR (which is ~1/100th of the current price). Yes, they explicitly called out orders on Wirecard. So basically... Denial of Service by bot-traders.


the order entry gateways normally have rate limits

and if a member does bad things the exchange can just cut them off

several good incentives for members to exercise control over their client's order flows


Have a source for that statement? All I found were speculations by traders.


It was on German n-tv. Although they now state the opposite here: https://www.n-tv.de/wirtschaft/der_boersen_tag/Arger-ueber-F...


Is it possibly a bug where they meant to buy 1/100 the number of orders at 100x the price?


This is such an easy problem to solve. All orders must stand for at least 1 minute before they can be cancelled. It stops all this crazy nonsense to try to move the price and arbitrage the pennies.


Even without the obvious impacts to market makers and the subsequent price increases to investors, this doesn’t help.

Most things trade with price/time priority. This means that first tie breaker is price but second tie breaker on what fills is who had their order into the market first. This means that market makers layer orders in as early as possible, some strategies I’ve seen layering them weeks or months in advance.

So this proposal would add cost to the market without changing anything about the cancel race.

The real solution to this sort of problem (which I don’t believe actually caused this issue) is the same mechanism you put in any request/reply system, rate limits. Most exchanges have them.


Rate limits, or discrete (in time) instead of continuous trading. An auction every minute or so, for three hours a day. That would be much better.

(When I started in equities in Hong Kong in 2007, the stock market was open for 4 hours a day... 10 to 12:30, two hour lunch break to drink and socialise^W^W^W have important business discussions with clients, then 14:30 to 16:00. Today it's open 5.5 hours a day. EDIT to add: "Reactions from both brokers and the restaurant industry were mixed." (1)

XETRA is open 8.5 hours a day without lunch break, while Frankfurt (FSX) is open 12 hours a day, from 8 to 8. WHY???)

(1) https://en.wikipedia.org/wiki/Hong_Kong_Stock_Exchange#Tradi...


We have these in Europe & the UK now with MiFID II, on several venues

They are called Periodic, or Frequent Batch, auctions.

This paper & earlier works it's based on were very influential in the design and regulatory framework for these: https://academic.oup.com/qje/article/130/4/1547/1916146

Here's a good intro to them: https://www.fca.org.uk/publications/research/periodic-auctio...

Volumes are small relative to the market as a whole, but execution quality is very good.


> WHY???

Shouldn't the question be "why not"?

More APIs than not on this planet are considered business critical and doesn't just close. Stock markets are strange as they must be highly available for a few hours and then can be shut down for the day. Imagine if AWS closed every night.


A lot of Amazon retail does close every night. You can put in a request but it won't be fulfilled and shipped until morning. Getting rid of that is a nice idea, but also "instant fulfillment" is not great either without "instant returns." The idea of "instant trading" seems like it can instantly put the system into a bad state with no recourse to fix it.


AWS doesn't close, which was the example.

The real reason stock markets close is because financial institutions are very interested in keeping it that way. It is lucrative to play pretend stock markets after hours.

(This is a very real question all over Europe right now, and it is interesting to see which institutions talk the loudest about how beneficial a shortening of opening hours would be to just about everyone.)


>WHY???

The exchange wants to make more money, and people want to be able to trade at whatever time they feel like.


It turns out they don't; they really want to meet their benchmarks, and trade where the liquidity is.

One data point - 40% of the French stock exchange's turnover is now in the closing auction. Other markets have a similar trend. Longer trading hours mean thinner liquidity throughout the day, and the direction of travel in Europe is strongly towards shorter trading hours.


The answer is also “foreigners”. Regular working hours for a German exchange can produce some pretty brutal trading hours for say, London. When I worked in finance they started serving breakfast at IIRC 5am, since some trading desks started their day then and ended around noon. Expanding those hours can entice foreigners to trade during non-insane local hours.


> Regular working hours for a German exchange can produce some pretty brutal trading hours for say, London.

Berlin and London time are only off by one hour.

You probably meant the US instead of Berlin/London or there is something going on I'm not aware of?


Yes; I picked the time zones out of thin air, and I chose poorly.


Tradegate is even 08-22...


That's not a solution, it would mean most market makers instantly left your exchange, or just quote really widely (providing a worse spread, so people wishing to trade have to pay a worse price), because holding your orders in the market for a minute is incredibly dangerous given how much the value of the underlying product could move at any time, especially when it's correlated with something traded on another exchange.


Usually what works for such abuses is either to charge extra (tiers depending on volume) to discourage abuse or at lest to cover losses, or throttling the biggest offenders when getting unreasonable peaks in order to keep the system always running.

Both are used in many other sectors. If you have "industrial level" needs you usually have to pay extra either for the service or for additional infrastructure needed to provide you that service, or you simply get everything throttled to sustainable level (think water, electricity, etc). Most traders would not see any difference and the ones that would will have to think twice before DoSing.


>They could put tiered charging and above a certain volume of transactions apply an additional fee (one time or per transaction). This is a model used in many other sectors.

It's actually quite common for exchanges to offer discounts/rebates to firms that trade more, because these firms are essentially providing a service to the exchange (market making, and generating turnover). Much like how other industries offer discounts for buying in bulk.

>The other option is to rate limit the biggest offenders when the system's performance limit is reached.

I don't know why people always suggest solutions like this when it comes to exchanges. If it was an e-commerce service and somebody suggested "let's rate limit customers to reduce load", it's be shot down as a lazy solution (and a great way to lose customers). If a platform isn't good enough to support the load its customers place on it, then it needs to be improved. If AWS goes down during the world's biggest shopping day, we don't blame the customers, we blame Amazon.


> It's actually quite common for exchanges to offer discounts/rebates to firms that trade more

I reworded a bit to give a better idea of what I meant. In general there are bulk discounts but there's always some form of "QoS". Either you get throttled above a ceiling, or you pay through the nose to have that ceiling higher for yourself and have everyone else throttled.

All tiered systems work like this. The fixed price goes up and gives you access to better service, higher limits, lower price per usage, etc. As long as you don't have virtually unlimited capacity you can't treat your system as if you do.

Mobile operators have to do the same. They charge a base contract price that sets the tier, then charge per usage within that tier, give different ceilings, QoS priorities, and throttling rates. And no matter how big you are as a customer at some point you either pay a surcharge or get throttled so the system stays up and other customers are also served. (source: I worked both for a large telco, and for a customer that payed ~3million E per month to get the consumption costs for min/MB down to almost 0).

> I don't know why people always suggest solutions like this when it comes to exchanges.

By any chance do the suggestions always come after the exchange was down due to high transaction volumes? Sure, they could implement unlimited capacity. But since this is a tad unrealistic and actually crashing the system is worse than limiting to keep it just below crashing, perhaps the suggestion makes sense.

There aren't many sectors that I can think of which give you "unlimited usage no matter what". There's "rate limiting" even in hospitals, where lives are at stake.


>By any chance does the suggestion come after the exchange was down due to the high volume? Sure, they could have unlimited capacity. But since this is a tad unrealistic and actually crashing the system is worse than limiting to keep it just below crashing, perhaps the suggestion makes sense.

I don't see a problem with rate limiting in general, if it's applied fairly. Good design should already entail that (it shouldn't be possible for customers to "crash" an exchange any more than it's possible for them to crash google.com). And many exchanges in fact have something like you described (customers can pay to buy more transactions-per-second capacity). But I don't think referring to customers as abusers or applying punitive fines is the right approach; if the exchange API let the customers crash it, that's the exchange's problem, and the customers shouldn't be blamed for taking advantage of whatever rate limit the exchange gave them (even if it gave them too much for its system to handle).


FWIW, eurex and xetra do have rate limits and will throttle and disconnect abusers.

There are also limits on average order to trade ratios (this is required by MIFID2 IIRC).


You’re right - the requirement on order/trade ratios is more a policy requirement, but venues are required to have limits to control excessive message rates.


Lots of exchanges have rate limits and messaging fees. And even if they don't, if you cause problems they'll shut you off.


To be fair, executions ("customer purchases") are relatively rare events. If we want to keep with the analogy e-commerce, you can think of orders/cancels as merchants repricing their wares and changing their on-offer quotas hundreds of times per second. Each. As a marketplace owner you probably would want to at least rate-limit that.

In a physical store you wouldn't get to the shelves from the throng of clerks running around with sticker guns.

> If AWS goes down during the world's biggest shopping day, we don't blame the customers, we blame Amazon.

Yes. We do. But the customers blame us. It may not be our fault, but it's still our problem.


Do you know how Amazon stays up during Black Friday? Rate limiting.


It could be a solution if markets agreed to the same rules. To me, this form of trading has questionable economic effects in my opinion.


>To me, this form of trading has questionable economic effects in my opinion.

Have you ever actually researched the economic effects, or is that just a gut feeling?

If you look at it historically, as machines have come to dominate market making, spreads (the difference between buy and sell price) have continually fallen, meaning institutional investors and retailers can buy what they want at a better price. Machines are able to offer these better deals because they know they can get out of their position faster if the market suddenly moves; removing their ability to react faster would remove their ability to offer better prices.


The part about spreads being smaller because machines are making markets is true, but there are important caveats. The one that comes to mind first is that the tight market is only there for small quantities. If you want to trade in size, then you are out of luck.

Machine based market making tends to work well when markets are operating "normally". When some regime-changing news comes out, it's not uncommon for the over-fit algorithms to perform badly so the managers just turn them off. I.e. liquidity disappears just when it's needed most.

https://www.wsj.com/articles/thinning-liquidity-in-key-futur...


well, yes, if you have a market moving trade, you'll have to pay a premium for it to be executed. MM are not there to give money away.

Similarly, if your house is on fire, it is hard to complain that buyers go away until they can evaluate how much the hashes are worth.

Market makers mostly provide a service for retail investors.


To me that sounds like customers are getting a slight discount on everyday transactions, but getting creamed the market suddenly moves faster


I actually did read about it. Wikipedia alone can enlighten you about respective controversies. I think critical voices sound more reasonable but it is an open question at least. It is still my opinion as stated.

Show me the produce, some say it is liquidity, and I withdraw my criticism.


I don't see any controversies mentioned on https://en.wikipedia.org/wiki/Market_maker. https://en.wikipedia.org/wiki/High-frequency_trading mentions some controversies about high-frequency trading in general, but none specifically about market making.


That is why I call that gambling and not investing.


Why do you call it gambling? The market maker doesn't want to take a position on the stock, that's why he tries to get out quickly if it starts moving. He just wants to provide liquidity (be willing to buy and sell to anyone at any time, so that buyers don't need to wait for a matching seller to come along), and collect a small fee for that in the form of a spread.


But who are the customers?

Why do investors need orders filled instantly?

It seems the only customers of this service are day-trading troublemakers.


A minute? Have you ever traded before? Prices move so fast that this would destroy market makers and so on.

Sometimes I cancel 4-5 orders before getting a fill on an option trade, manually.


Is it even a legitimate problem? Exchanges are well-equipped to deal with this kind of thing and direct access to exchanges is moderated by broker-dealers. As a normal user (algo trader or otherwise), you cannot directly plug into the exchange and spam orders. And anyways, exchanges (even IEX) _want_ the arbitrage and liquidity that market makers offer.


Or make taxes on gains start at 100% and exponentially decay as you hold a certain instrument longer. If the final tax is 30%, that could be reached e.g. after a week.


Should that rule also apply to retailers of other products, so they pay 100% tax if they turn over wholesale products too quickly? Why or why not?


I fully agree with you. I have no idea why you are down-voted so heavily.


> I fully agree with you. I have no idea why you are down-voted so heavily.

Maybe because they claimed it's easy to solve when the solution they proposed in fact has a whole lot of undesirable consequences that are obvious to anyone more familiar with how markets work.


Sending out orders and withdrawing them immediately after is market abuse, there is a regulation in Europe against it (MIFID II): the real question about the "whole lot of undesirable consequences" is, to whom? and who would benefit instead?

The truth is that the real effects of algo trading are still poorly understood as a system, and they have been able to crash markets a couple of time in the past, for no reason. Do we really need this form of speculation "because liquidity"?


>sending out orders and withdrawing them immediately after is market abuse

It's abuse if it's done for the purpose of manipulating the market (spoofing) or deliberately to slow it (quote stuffing), but not if it just happens because somebody put in a bunch of orders and then realised their mistake.

>"a whole lot of undesirable consequences" is, to whom

To all the institutional and retail traders on the exchange, who have to pay worse prices for their trades because the market makers can't quote so tightly.


> but not if it just happens because somebody put in a bunch of orders and then realised their mistake.

Surely that would be an occasion thing though, not several times a second for a prolonged time?

I can see how throttling of some kind could work, as long as it's not too aggressive.


Market makers routinely have to adjust their quotes as the price moves. If they set a price and then the market moves significantly, the price will soon be incorrect. If they leave the order on the book then they might get undesirable fills (losing money) or it might just sit there and never get executed because their price is worse than what the market is offering. In either case, it's useful for them to be able to adjust their quotes by cancelling previous quotes that are not up to date with the latest information.

Imagine if every time you go to the gas station and see the big price for a gallon you write it down. Should the gas station still have to offer you the lowest price over the past {day,week,month} if the price has gone up? "Cancelling" orders is this same idea of changing the offer, only at a much smaller time scale because markets trade extremely fast.


It’s not abuse if you’re trying to get your order filled. I make and immediately cancel orders frequently, simply to get a fill. Derivative prices move fast


Really interested in reading a technical post-mortem for that.


I'd like to see one too - they are rarely published (BaFIN will get it I'm sure). Usually the markets are reopened, some trades cancelled and everyone continues.


didn't get one for the same thing in April (4 hour outage), no reason to expect we'll get one this time either


They used to run on OpenVMS. I don't know if that's still the case.


Pretty sure it’s Linux now. One thing I like about t7 is they offer private exchange simulation environmentS in the cloud you can spin up and test against.


Germany's Financial Authority is having a bad week.


Deutsche Boerse AG runs the XETRA system and is a private, publicly traded company. While they are of course overseen by the financial authorities, the German stock exchanges are private institutions.


Interestingly, "public" has a different meaning in English than in German in the context of a company.

In English, a public company is a company whose shares can be freely traded by the public.

In German, a public company is a company that is owned to the government. Companies that are listed on the stock market are called "kotiert".

This could hint at a cultural difference, with the Germans more strongly identifying with the government ("if it belongs to the government, it belongs to everyone"), whereas the English speaking countries might tend to identify more strongly with the economy ("if everyone owns or at least could own some shares of it, it belongs to everyone").


I'm not a financial expert, but I think "börsennotiert" (listed on the stock exchange) is much more common than "kotiert", at least in Germany.


I'm German, and I literally just saw the word "kotiert" for the first time. Definitely use "börsennotiert" instead.


I believe "kotiert" is a charming Swiss-German word (one of many, like "Velo" instead of "Fahrrad" for bike).



kotiert seems surprisingly like "cotée" for "société cotée en Bourse" in French so I wouldn't be surprised if it were the Swiss that used that term.


That "kotiert" may be literally translated into "excrementized" or "fecesized" certainly doesn't help in making it more popular.


Yep, never heard of "kotiert" before.



But "public education" refers to state run/funded schools :)

Also in the UK the "Public Sector" is government organisations.

The "private sector" is for-profit/comercial organisations, many of which are publicly traded comapanies, which is normally shortened to simply "public companies". So you could be a public company in the private sector that provides services to a public school, which is part of the private education sector.

Language is fun.


In Ireland, both are used. A public [owned/sector] company is a government owned company while a public [limited] company (plc) is a company who's shares can be freely traded. Just "Public company" on its own can be context dependent, as publicly owned companies are often referred to as just public companies when referring to things like public-private partnerships or privatisation.

At least public sector and public vs private in the context of privatisation are used in the UK also.


The phrase "publicly owned" in English implies government-owned (which seems like the German.) I'm not sure that the phrase "public company" is a common one. "Publicly-traded"? Yes. A company "going public"? Yes. And people would know what you meant if you said "public company" in context, but they might think you were referring to a "public utility," which may or may not be government owned.

Also public and private schools (in the US.)


A quick Google shows common usage of "Public Company" in US publications, in the manner described by GP.


I think that is more of a US thing, to my knowlegde public company means the same in english and german.


I get most confused by "public schools" in the UK which are the privately owned fee paying ones.


They are open to the public (in contrast to a private teacher). They are public in the same way a pub is a public house.


I think he is talking about Wirecard AG scandal and that company filling for insolvency. It's a huge thing that affected millions of people world wide specially their payment card solution in UK that got blocked and many companies that rely on them got their user cards blocked


Yup. I was just reading about the wirecard fiasco and now xetra is down.


This also affects trading platforms that match xetra price spreads during normal trading hours, such as LUS which is used by HSBC for Trade Republic (the German Robin Hood).


Charge some some small fee per transaction to pay for something worthwhile in society and that will limit the profitability of these billions of micro transactions as well.

All these big finance companies should just be public utilities. They provide little to no benefit to society. They create nothing. They siphon so much money. The cost of the instability/fragility they create is sent downstream to the taxpayer. What society purposely sign up for this?


Most exchanges have per trade fees, although many do offer rebates for market makers or liquidity providers.


Really interested to learn those two terms - "market maker", "liquidity provider". I think I kinda understand the second one (traders who are bringing money from outside the market). Does anyone know where I can read more about this kind of thing?


A market maker is an entity which isn't interested in the underlying assets at all. Their goal is to always end with a 0 asset, all cash balance sheet (over some small time interval). Their goal is to buy X at $100 and sell it at $100.01 a million times a day.

`Liquidity providing` is the side of the trade which had a `resting order` on the order book which traded against a new order (the liquidity taker). Various exchanges have various different fee schedules for your designation as a market maker, how much volume you do, whether you are taking or supplying liquidity, and what kind of order you are placing.


I'm pretty sure Wikipedia has you covered... But TL;DR: traders that always have a buy and sell transaction active, making sure the market allows to buy/sell instantly at any time.


In other articles about that I read that this is related to a software component called "System T7". There was a very similar incident in April related to that component.


The T7 trading system is used by the following exchanges for derivatives trading: Eurex, European Energy Exchange (EEX), Powernext.

Source: https://www.eurexchange.com/exchange-en/technology/t7

Are those down as well?



Yes, they were at least partly :- https://www.eurexchange.com/exchange-en/trading/production-n... (but not Powernext - they are still on a different platform )


According to the comment IDs, you beat me to the punch by 4 comments. Looks like i need to upgrade my HN commenting stack to use an FPGA ...


Or do as the "Flash Boys": use a microwave link with direct path, and post from the network card, avoiding the kernel stack...

I've enjoy your humorous reference.


EEX was down today as well.


T7 isn't just a component, it's the name for the whole trading platform for Deutsche Borse's markets (and it's generally very good)


It is pretty much a step above all the other exchanges in Europe.

Asking around here, nobody is reporting any issues though, so it is possible only some stocks are affected.


I wouldn't say that, and I've interfaced with most of them

it's better than Euronext Optiq, but that's about it


Xetra, Eurex and their T7 users Wiener Borse, Budapest Stock Exchange, and some others were halted most of the morning - luckily the stocks can be traded elsewhere, but the derivatives not so much.


yes, I just saw the discussions in the support chat.


I'm not sure about the particular component but Xetra/Eurex indeed went down in April. Same pattern as today: it went down around the opening hours and got fixed around lunch time. They have some structural problems that have likely been left unsolved.


What's the game plan in this scenario, will all orders be invalidated or what happens to them? I'd imagine it would really mess up market makers and others.


It does. Most orders were cancelled and maybe trades too.


It was just a technical issue, right?


My guess is that it's Kubernetes-related issue.


That seems oddly specific without detail. Why do you think it's related to Kubernetes?


I think it was a jibe at Kubernetes and "cloud" in general.


I think it’s joke. It made me LOL.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: