• Fantom Logo
  • Technology Tools Enterprise Ecosystem Developers
  • Fantom DeFi
      • Sub menu

        Intro to Fantom

        Fantom is a fast, scalable and secure platform for digital assets.

      • Sub menu

        Stake on Fantom

        Earn rewards by securing the network.

      • Sub menu

        FTM token

        FTM is the primary token of the Fantom ecosystem.

      • Sub menu

        The Consensus

        Discover Lachesis, Fantom’s ultra-fast consensus protocol.

      • Sub menu

        Opera Chain

        The Fantom mainnet, with staking and EVM support.

    • WALLETS

      • Sub menu

        Fantom Wallets

        Store, send, receive, and stake FTM on desktop, mobile, and hardware wallets.

      EXPLORER

      • Sub menu

        FTMScan

        A new Fantom explorer built by the Etherscan team.

      • Sub menu

        Opera Explorer

        Explore the network transactions, blocks, and nodes on Opera.

      • Sub menu

        OKLink

        OKLink is an alternative explorer for the Fantom network

    • GUIDES

      • How to use Fantom Wallet

      • How to set up Ledger Nano with Fantom

    • Sub menu

      Enterprise

      Discover Fantom’s solutions for enterprise clients.

    • Sub menu

      Solutions

      Take a deep dive in real world use cases.

      • Sub menu

        About us

        Meet the people behind Fantom.

      • Sub menu

        Partners and integrations

        Learn how we're creating a cross-chain ecosystem on our innovative platform.

      • Sub menu

        Blog

        Learn about Fantom through our general and tech articles.

      • Sub menu

        Podcasts and interviews

        Check out all the Fantom related interviews and AMAs.

      • Sub menu

        FAQ

        Do you have questions? We have answers!

      • Sub menu

        Community

        Join our wonderful community and share ideas.

      • Sub menu

        Careers

        Come work with us!

    • Sub menu

      Documentation

      Start building on Fantom

      GET STARTED

      Deploy a smart contract Create a fixed-cap asset Create a local test network GraphQL API docs

      GUIDES

      Set up Metamask Run a validator node
    • Sub menu

      Technical papers

      Read our technical papers and latest research essays.

      Covalent API
      The Graph's Subgraphs
      Verified contracts
      Frontend JS Utilities
      DeFi Smart Contracts
      Developers’ chat
Fantom Logo
menu
Fantom Logo close
  • Technology
        • Sub menu

          Intro to Fantom

        • Sub menu

          Stake on Fantom

        • Sub menu

          FTM token

        • Sub menu

          The Consensus

        • Sub menu

          Opera Chain

  • Tools
      • WALLETS

        • Sub menu

          Fantom Wallets

        EXPLORER

        • Sub menu

          FTMScan

        • Sub menu

          Opera Explorer

        • Sub menu

          OKLink

        GUIDES

        • How to use Fantom Wallet

        • How to set up Ledger Nano with Fantom

  • Enterprise
        • Sub menu

          Enterprise

        • Sub menu

          Solutions

  • Ecosystem
      • Sub menu

        About us

      • Sub menu

        Partners and integrations

      • Sub menu

        Blog

      • Sub menu

        Podcasts and interviews

      • Sub menu

        FAQ

      • Sub menu

        Community

      • Sub menu

        Careers

  • Developers
      • Sub menu

        Documentation

      • GET STARTED

      • Deploy a smart contract

      • Create a fixed-cap asset

      • Create a local test network

      • GraphQL API docs

      • GUIDES

      • Set up Metamask

      • Run a validator node

      • Covalent API

      • Verified contracts

      • DeFi Smart Contracts

      • The Graph's Subgraphs

      • Frontend JS Utilities

      • Developers’ chat

      • Sub menu

        Technical papers

blog-image
February 27, 2021

Fantom network incident analysis

Development

On Thursday, February 25 2021 at 3.04 PM UTC, Fantom Opera mainnet halted new block confirmations, which resulted in a temporary outage. 

The core developer team and Fantom validator community immediately responded and successfully resumed the network within 7 hours.

This feat required consensus across 39 validators in many different time zones; while the community quickly diagnosed the issue, most outage time was due to waiting for sufficient stake weight to come back online.

To be clear, no funds were at risk as a result of the network halt.

Cause

One of the biggest validators slowed down the block emission, which caused a second big validator to slow down as well. The other validators kept producing blocks, but the two lagging ones were not able to catch up. These two validators are big enough to represent more than 1/3W of stake, and they caused a domino effect that halted new block confirmations. 

Recovery

Our core developer team coordinated with the validators to apply a temporary patch to slow down the event emission rate to allow lagging nodes to catch up. Once all the nodes were synced again, they started to confirm blocks and finalize epochs.

Validator nodes size

The incident has highlighted that the network needs to make some changes regarding the validator nodes’ requirements and their staking power distribution.

We identified two issues:

  1. The recent increase in the value of FTM made creating new nodes prohibitive. While this is not directly related to the incident, it accentuated point 2.
  2. Most of the staking power is concentrated among, which have more delegations than other nodes.

We’ll make a proposal for stakers and delegators to vote on the on-chain governance. We’re still defining the parameters but will likely involve lowering the minimum amount of FTM required to run a node and putting a cap on the node size. 

Resolution

  1. Governance proposal to change requirements to run a validator node and incentivize the creation of new high-quality nodes. We’ll make the proposal in the short term.
  2. Deploy the go-opera upgrade. In this upgrade, we completely rewrote the p2p code, optimizing CPU time and memory usage. More changes are coming to the emitter and the general network performance. In earlier tests, the new code achieved 10k tps on medium/low spec nodes without transaction execution. The new code needs to go through extensive compatibility and stress tests and will be available later on.

Conclusion

We want to publicly thank all Fantom validators that coordinated and worked together to solve the issue and bring the network live as soon as possible. A big thank you to the community as well for being incredibly supportive and understanding during those hours. While we’re not happy with the incident, the silver lining is that this growing pain happened sooner rather than later, allowing us to become aware of the issues and prepare for the proposed fixes.

Read more

Card image cap
New Fantom Wallet is Here!
READ STORY
Card image cap
Fantom General Update | March 13, 2023
READ STORY
Card image cap
Mayor Bimps – Fantom Community Spotlight
READ STORY
Card image cap
New Fantom Wallet is Here!
READ STORY
Card image cap
Fantom General Update | March 13, 2023
READ STORY
Card image cap
Mayor Bimps – Fantom Community Spotlight
READ STORY
  • Technology
    • Intro to Fantom
    • Stake on Fantom
    • FTM token
    • The Consensus
    • Opera Chain
  • Tools
    • Fantom Wallets
    • Opera Explorer
  • Enterprise
    • Enterprise
    • Solutions
  • Ecosystem
    • About us
    • Partners and integrations
    • Blog
    • Podcasts and interviews
    • FAQ
    • Community
    • Careers
    • Media kit
  • Developers
    • Documentation
    • Technical papers
    • Covalent API
    • Verified contracts
    • DeFi Smart Contracts
    • GraphQL API
    • Frontend JS utilities
    • Developers’ chat
Social
  • Twitter
  • Discord
  • Telegram
  • Telegram ANN
  • Reddit
Fantom Logo

© 2023 Fantom Foundation

  • Cookie Policy
  • Privacy Policy
  • Terms of Service
  • Cookie Policy

The information on this website is for informational purposes only. It contains high-level summaries and is not intended to include all material information regarding the Fantom ecosystem and risk factors associated with holding FTM.