Back to Announcements

Post Mortem: Transaction Fetcher Incident and HFT Order Handling

January 26, 2026
Post Mortem: Transaction Fetcher Incident and HFT Order Handling

Summary

During Epoch 38, XRPLiquid's backend transaction collection system experienced failures that resulted in incomplete snapshot data. Several users reported that their bots were not accumulating points correctly.

After investigation, we determined that the root cause was not malicious spam, but rather the entry of new HFT-style (high-frequency trading) market makers submitting significantly more orders than our system was originally designed to handle.

We have since recalculated all rewards for Epoch 38 and distributed them correctly. The backend system has been upgraded to handle this level of order load, and we've been monitoring stability for the past two weeks.

What Happened

The Trigger

A new user began submitting orders at a rate far exceeding typical usage patterns—approximately 30x more orders per block than a normal user. This dramatically increased the volume of transactions our system needed to query and process.

Why This Caused Problems

XRPLiquid tracks on-chain transactions with an overlapping collection window to ensure no user activity is missed. Our system queries the XRPL for each registered user's transactions, processes new ones, and skips duplicates from previous queries.

When the HFT user's transaction volume increased dramatically:

  1. Query size exploded: Instead of fetching a few transactions per user, the system was now fetching tens of thousands of transactions per query for this single user.

  2. Public node limits hit: We were using public XRPL nodes for data collection. The massive query load caused requests to fail or timeout.

  3. Fallback cascade: Our fallback mechanism was designed to restart the entire snapshot process if any user's data collection failed. This created an endless loop where the HFT user's collection would fail, triggering a full restart, which would fail again.

  4. Incomplete snapshots: The result was that only transactions from users processed before the failure point were captured in Epoch 38 snapshots.

Resolution

Immediate Actions

  • Recalculated Epoch 38 rewards: We manually collected and verified all transaction data for Epoch 38 and recalculated rewards for all affected users.
  • Distributed corrected rewards: All Epoch 38 reward checks have been issued based on accurate data.

System Upgrades

We implemented several architectural improvements to prevent this from happening again:

1. Per-User Independent Retries with Exponential Backoff

Previously, a failure for any single user would cause the entire snapshot to fail and restart. The system now handles each user's data collection independently. If one user's collection fails, it retries with exponential backoff without affecting other users.

2. Improved Request Distribution Across Nodes

The load generated by high-frequency order submissions is now distributed more evenly across our node infrastructure, preventing any single node from being overwhelmed.

3. Upgraded Node Infrastructure

We've moved away from relying solely on public nodes and upgraded our infrastructure to handle higher query loads reliably.

Handling HFT Market Makers

The platform has been successfully adapted to support users with high order submission rates. HFT-style market making is a legitimate trading strategy, and we welcome these participants to the XRPLiquid ecosystem.

That said, we will be monitoring for abusive behaviors such as spam (submitting excessive orders with no intent to trade) and spoofing (placing orders to manipulate prices with the intent to cancel before execution). These activities undermine market integrity and fair reward distribution.

Monitoring Going Forward

To ensure a fair system for all participants, we are actively monitoring several metrics:

  • Order fill rate: The ratio of executed orders to submitted orders—unusually low fill rates may indicate spam or spoofing
  • Transaction query load: Per-user transaction volumes during snapshot collection
  • System health: Node response times and success rates

Our goal is to maintain the infrastructure necessary to support diverse trading strategies while protecting the integrity of the rewards program.

Current Status

The upgraded system has been running stably for two weeks with no incidents. We're confident that the infrastructure can now handle HFT-level activity while continuing to serve all users reliably.

If you notice any issues with your points accumulation or have questions about your Epoch 38 rewards, please reach out to us.

Questions?

If you have questions about this incident, the system upgrades, or your rewards:

Thank you for your patience and continued support.

- The XRPLiquid Team