Latest events
26-05-2024 Service outage

One of our core servers had all disks full which resulted into frozen VMs serving some core APIs and cryptocurrency nodes. After a cleanup and resuming VMs, some cryptocurrency nodes became unsynced with their networks due to system time desync while VMs were suspended. We have reset system time on all VMs as well as hypervisor's system time which resolved the problem.

All pending orders (500+) that were stuck during that time were processed.

14-05-2024 Service downtime (~3 hours)

Our backend engine node server became unresponsive due to hardware failure related to the network adapter's kernel bug (related and absolutely same behaviour reported here)

10 minutes after server unresponsiveness we were eventually able to login via SSH by some attempts to see that the problem is the failing network adapter caused by IOMMU-related kernel bug triggered inside our Tor router VM.

We have assumed safe to soft-reboot the server since there were no signs of any physical server intervention. We have also checked the bootloader integrity and changed disk encryption keys after a reboot.

No disk or RAM modules were ever pulled from the server. No USB ports were triggered or unknown devices attached to the server.

Conclusion: Not a coldboot attack attempt - safe to operate. Migration to other hardware will still be performed later.

25-03-2024 - 05-04-2024 XMR RPC API degraded performance / stuck XMR payouts Since Mar 25 we are experiencing intermittent API errors coming from `monero-wallet-rpc` resulting into randomly failed payouts. We will wait for the Monero core team to investigate the issue and release a patch but there is no ETA. If your order's XMR payout is stuck, the error will be cleared within an hour and another payout attempt will be done. We have implemented a temporary workaround to automatically monitor failed `transfer` API calls that had a response { 'code' => -38, 'message' => 'no connection to daemon' } and repeat them after some minutes. (Node and RPC version 0.18.3.3)
UPDATE:
We have identified the cause after some debugging and found out that there were reports of this issue already (1, 2, 3, 4, 5) but the core team refuse to address it, suggesting enabling hugepages instead of assigning and fixing the obvious bug. We are not going to enable hugepages because 1) we can't due to internal reasons 2) the issue is a broken Monero node code and not anything else, but what devs are currently proposing is sealing a leaking boat with a duct tape. The bug is approximately 2-years old.
UPDATE:
Fixed by recompiling without libunwind (unset LIBUNWIND_FOUND). This solution was never proposed or mentioned by the core team in relevant Github issues.
08-03-2024 XMR node degraded performance The RPC client from the v0.18.1.0 release we are running had async problems blocking many RPC calls probably due to the current attack on the Monero network. Compiling and launching the RPC client from v0.18.3.1 while keeping the node at v0.18.1.0 seemed to solve the issue. The v0.18.3.1 node crashes some minutes after launch so we will keep the node at the older version till some major update, since we are currently not using pre-built binaries from the core Monero team due to some of their members having serious infosec issues, such as rogue OS (Windows) usage for development of Monero.
30-12-2023 ETH node downtime Ethereum node maintenance took 22 hours due to a bug named "holiman" that caused blockchain corruption. We had no choice but to resync the whole block data. We have also spun up a second backup node to prevent long downtimes in future.
Node status information
BTC ONLINE since Fri May 10 10:41:21 2024
Software: bitcoincore
Updated: Wed May 29 12:48:53 2024
Height: 845641
BTC ONLINE since Fri May 10 10:38:35 2024
Software: bitcoincore
Updated: Wed May 29 12:48:53 2024
Height: 845641
BTCLN ONLINE since Sun May 26 02:41:16 2024
Software: lightningcore
Updated: Wed May 29 12:48:53 2024
Height: 845641
DASH ONLINE since Fri May 10 10:36:38 2024
Software: dashcore
Updated: Wed May 29 12:48:53 2024
Height: 2079287
ETH ONLINE since Sun May 26 10:48:53 2024
Software: geth
Updated: Wed May 29 12:48:53 2024
(patched with this commit by gituser543 for signing transactions with external private keys)
Additional information (click to expand)
Occasionally, ETH (and ERC-20) transactions you send to us may take a bit longer to be detected (10-15 minutes). The Ethereum node software stack that is released as "stable" on the official site currently contains a lot of critical bugs. Random crashes, blockchain desync and even blockchain files corruption happen to our Geth instance at least 2 times per week, which makes this software extremely unstable for the production use. Geth core development team are extremely incompetent and refuse to admit or investigate such bugs while their universal solution is to resync the whole blockchain whenever their product brakes itself (example issue of an unfixed bug). Note that Ethereum is the only project with official software stack being broken among the cryptocurrencies we receive, thus, we count on your understanding and that someday the whole Ethereum development finally gets some competent developers or at least a node not written in insecure for production environment languages such as Go/Java/NodeJS. Things got especially worse after the PoS merge, since they now often throw an excuse accusing beacon chain software of being broken, which makes it simple for them to disclaim the responsibility. We use Lighthouse for the beacon chain and it's the only client that is stable and developed by the competent programmers. It's written in Rust, which isn't our favourite language as well due to dependency chain security risks, but that's still much better than Go or Java. We made some health monitoring scripts to determine when Geth gets into a broken state. If your transaction wasn't detected in time, it means our node was restarted some minutes ago and it's resynchronizing and your transaction will be detected once synchronization finishes.
Height: N/A
LTC ONLINE since Sun May 26 10:58:32 2024
Software: litecoincore
Updated: Wed May 29 12:48:53 2024
Height: 2693658
XMR ONLINE since Thu May 16 13:10:33 2024
Software: monerocore
Updated: Wed May 29 12:48:53 2024
Height: 3159680