Bringing Bundle and Atomic arbitrage on prediction market? [bip Part 2]
Next part of my possibly build in public series about cooking a prediction market that actually bring value to the ecosystem!
Hi everyone, first of all, thanks for your DMs and questions, and to everybody who joined us on discord ! Second, i am pretty sure this project can have a future, it look like somes people are interested and others are doubting, and, fair enough.
Special mention to the polymarket dude who tried to hire me (and other PM) (i will accept if this project fail 🙋♀️) we had a cool conversation:
So today, i would like to introduce a better mechanism than classic FIFO matching engine: Frequent Batch Auction. I don’t know if my current proposed disign is common, someone told me about dual flow batch auction where you can learn more about here after i showed him my idea, it’s pretty close, but my architecture contains a CLOB.
Basic flow of a exchange
First of all, here is how the “global” architecture look like, it’s a simplified schema, some things might be missing.
Let me explain each component and why it is important for an exchange. I will also show how a user request ends up in the orderbook. I should mention that I have never worked for an exchange, so I might be wrong somewhere. Feel free to correct me.
The goal of an exchange is to process trades as fast as possible while keeping the system fault resilient and deterministic. If you cannot handle load, or if one part of the exchange fails, you certainly don’t want to lose people’s balances. We cannot use a database in the hot path because constantly locking the database and dealing with network overhead is far too slow. This is why the matching engine keeps everything in memory. The downside is that if anything goes wrong, you could lose all the data, which we cannot afford.
First, you have the input interface of the exchange. This can be a REST API, a Websocket server, a FIX server, or any system that allows a trader to submit orders. For scaling, this part must be deployed on many servers behind a load balancer. Received orders are sent into a queue, which is consumed by the sequencer, or they can be sent directly to the sequencer.
The sequencer’s role is to timestamp and add a sequence number to each request. This ensures that we process requests in the correct order, provides determinism, and helps with auditing. This cannot be handled by a distributed system like REST servers because CPU clocks could be desynchronized (wrong timestamp), or could cause race conditions in the sequence ID. I will call this sequence ID “seq”. The seq is an incremental ID assigned to each event. It is useful for several purposes:
To know if a message was dropped somewhere.
To sort messages without relying on timestamps, which can be inaccurate.
To replay the message queue from a specific index in the correct order.
After adding the timestamp and seq to a message, we add it to an event queue that is persisted in storage. This serves as a backup that we can replay to resynchronize the state of our risk and matching engine.
The risk engine’s role is to validate orders before they reach the matching engine. It checks that orders are correctly formatted and that the trader has sufficient balance for the requested action. Essentially, it validates everything to prevent errors and rejects any suspicious requests. In our prediction market, the risk engine performs basic checks. We don’t have complex perpetual markets or simulations.
The matching engine then receives the orders. It processes them, matches them against other orders, places them into the CLOB, commits balance changes, and generates events as output. These events can include order filled, order cancelled, BBO, orderbook, balance snapshots, and positions snapshots, etc... After processing a certain number of sequences, the matching engine saves a snapshot of users’ balances and positions, including the last processed sequence ID. This allows us to replay the sequenced queue to restore user states after a crash, without replaying everything from day zero.
The events generated by the matching engine are sent back to an event queue, which does not need to be persisted. The goal is simply to fan out these events to traders as quickly as possible and in a fair way. You don’t want some traders to see prices before others. The current disign is simple and not distribued, you can learn more about distributed ME etc.. here
Bringing Bundle and Atomic arbitrage on prediction market !
To begin with my explanations, i have to explain the root idea behind the title of this section. I was thinking about how to reward makers while having a more fair competitions between bots. I did end up into thinking about this:
Instead of giving to everybody a speed-bump of 500ms+ (up to 3s on polymarket), things can be done in a better way from my point of view, What if the market is processing by batch of 500ms? This would give the time to informed makers to protect themselve from adverse selection, Give the time to people with less sophisticated infra but good models to compete with faster player with poor models, basically erase/reduce the speed race.
Instead of a FIFO (time priority) queue, we would have a semi FIFO, where passive traders can trade “for free”, and allow arbitrageurs and market makers to jump across the queue for a small tip (see this as a gas fees to get validated on the blockchain before others in a block for example). Then, give back the paid tip to the counterparty (makers) to compensate them from being picked off. This give a extra rebate to makers that will linearly grow with the competition over this opportunity, without necessarily introducing fees into the exchange, keeping it free for everybody.
The main issue is how to make this tip/bribe thing fair? At the beginning, i did think about sending the fees in pro-rata way, to the counterparty, for the filled amount. If someone pay 10$ bribe to fill 100$ and got only 10$ filled, the counterparty would recieve 1$, distributed among all the matched order at pro-rata regarding the filled size. (trader A match 90% of the order he recieve 90% of the tip etc..) This might seems pretty smart and fair for the unfilled orders, but for exemple, if we redistribute the tip based on the % of the order filled, anybody can pick off someone for 10$ using IOC and put a order size of 100000$, with a tip of 10$ and only pay near 0. SO the tip should always be paid fully whatever the amount is filled.
The problem is that it can be quickly expensive if you want to submit multiple orders, cancels, amend etc.. and you probably wanna reduce this cost while still having priority over free users…
The solution!
So here is a better idea, i can re-disign fully how things works by implementing a Bundle auction system. Each “bundle” is a batch of operations that can be executed in an ordered and atomic way, allowing deterministic atomic arbitrage across multiple prediction markets.
Each 500ms, a auction will begin for EACH listed markets allowing traders to submit, edit, cancel bundles, and bribe during a “bidding war” to have your bundle executed within the firsts. Bundle could have different executions rules for each tx inside, to determine if the whole bundle should revert or if we allow some tx to fail.
The auction duration have to be determined regarding the amount of orders to process, the speed of traders etc.. it can be reduced by a lot, or augmented, to keep a smooth venue.
Exemple:
i want to buy NOW BTC-UP-5m and cancel x,x,x,x orders, but if the order cancel fail, it’s ok.
I want to buy “elon musk turn gay before 2027?” and buy “elon musk annouce he is gay before 2027”
To prevent spam/attacks, a revert will cost 10% of the tip, or be variable on a rolling window based on the revert ratio per bundle or notional traded. This concept also prevent from sandwitch attack on the “dummy auction system” as you can only submit 1 bundle per auction. decrease spam of failed orders and also improve the fairness within the latency race.
The paid tip will be the bundle after you in the queue, this incentive into setting a correct pricing. This introduce some new mechanic and arbitrage, as Each bundle will have their own values, traders could try to predict the cost of being the counterparty or take a opportunity, try to make profit from this, making a even better liquidity on the platform.
The people who are not willing to pay a tip, will just be at the end of the queue, and that’s ok! The tip paid for only canceling orders or failed bundle, will fund the platform, or be redistributed into incentives, such as daily airdrop. Matched orders will still redistribute the tip to the counterparty. Maybe the undistributed tip can be sent into a liquidity pool to MM the platform, or be left on the orderbook like breadcrumbs (e.g you delete a buy order at 0.50% with 10$ tip, the 10$ will be “given” to the exchange LP, and placed into the book at the same price, allowing other kind of arbs or maybe make the protocol profit even?
For those who wonder what i did think about to build this, This is, for example, what MEV bots do on the blockchain to compete for arbitrage opportunities while being protected in a private mempool. They send bundles to the builder with a tip, and the bundle with the highest tip is the one added to the next block, while the other transactions are reverted for free (you don’t need to pay gas fees).
I also had few people yapping about us taking somes fees, well, we will only charge few bp 2-5bp, wish is less than half of a tick size (10bp). polymarket spread is 100bp (1%) and take up to 1.5% fees and redistribute only 20% to makers. Paying a fees on a low spread market, will still be cheaper than crossing the book on polymarket.
Also, fees would be split with the counterparty (maker rebate), and affiliate program. the platform have to get some incentive to bring and invite people, like a referral program where you an earn from your refere up to 3 tier of sub affiliate. the platform would keep mostly peanut on those fees, this will also incentive more market makers to quote as they will earn rebates. and also allow us to apply a toxicity based fees (that will go 100% to the maker), as i described on this article
If this project sound cool, your free to join the dedicated discord server: https://discord.gg/D8CjJvmPc2
Incredible work here, the bundle auction concept is genius. The way it turns the tip mechanism into a self-correcting pricing signal while protecting makers from adverse selection is kinda brillant. I ran into similar issues trying to arb prediction markets last yer and the latency wars made it impossible to compete without burning capital on infra. One thing that's interesting is how the revert cost (10% of tip) will impact bundle spam in high-volatility periods tho, it might create weird incentives for someone with deep pockets to DDoS the auction phase.
Twin, twin your posts make me moist twin.