What are Substreams on The Graph protocol?

What Are Substreams on The Graph Protocol?

Substreams are an advanced framework within The Graph protocol, designed to dramatically accelerate and enhance the process of indexing and processing blockchain data. Developed by StreamingFast (a core developer team in The Graph ecosystem), Substreams represent a shift from the traditional, sequential data extraction model to a parallelized, streaming approach—enabling potentially over 100x faster indexing speeds compared to legacy methods+1.

Core Features and Benefits

  • High-Speed, Parallel Indexing:
    Substreams use a horizontally scalable engine to process historical blockchain data in parallel, drastically reducing the time it takes to sync and index large datasets. For example, tasks that previously took weeks may now complete in hours or even minutes.
  • Composability and Modularity:
    Substreams are built with modular Rust-based components (Substreams modules), allowing developers to create, compose, and reuse data streams or analytical pipelines. These modules can be shared and leveraged by others, fostering collaboration in the web3 ecosystem.
  • Multi-Chain Support:
    Initially targeted at Solana, Substreams now support Ethereum, Polygon, and other chains, opening up fast and efficient querying for many blockchain networks.
  • Real-Time and Historical Data:
    Substreams can power both real-time notifications and deep, time-series analytics, enabling new classes of dapp features such as block explorers, trading engines, cross-chain bridges, large-scale analytics, and more.
  • Integration with Subgraphs:
    Substreams can serve as data sources for Subgraphs, combining the efficiency of Substreams with the query capabilities of The Graph’s existing subgraph infrastructure. This allows developers to load enriched data from Substreams directly into familiar subgraph entities for further processing and querying.

How Substreams Work

Traditional subgraphs in The Graph extract, transform, and load blockchain data in a step-by-step, sequential manner. In contrast, Substreams break down the data pipeline into smaller, parallelized operations:
  1. Extract: Retrieve data using Firehose (streaming data extraction).
  2. Transform: Process data with parallelized computation modules (Substreams).
  3. Load: Feed transformed data into databases or other endpoints.
  4. Query: Access the processed data using GraphQL or other supported APIs.

Substreams-Powered Subgraphs

By leveraging Substreams as data sources, Subgraphs can achieve unprecedented performance and flexibility. Developers can decide where to place their business logic—either in Substreams modules for high parallelization or in subgraphs for more control over sequential processing. Substreams-powered subgraphs have also enabled indexing rewards on The Graph Network, further incentivizing the development and use of this high-performance data infrastructure.

Developer Resources

  • StreamingFast Substreams documentation offers tutorials and technical guides for getting started.
  • The Graph has provided example repositories and workshop videos to help new developers build Substreams-powered subgraphs quickly.
In summary:
Substreams are The Graph protocol's innovative answer to bottlenecks in blockchain data processing: they enable massively faster, modular, and more scalable on-chain data extraction and transformation—both as standalone data pipes and as powerful accelerators for subgraphs+6.
You're viewing a shared conversation. Your questions will start a new chat.