What Are Substreams on The Graph Protocol?
Substreams are an advanced framework within The Graph protocol, designed to dramatically accelerate and enhance the process of indexing and processing blockchain data. Developed by StreamingFast (a core developer team in The Graph ecosystem), Substreams represent a shift from the traditional, sequential data extraction model to a
parallelized, streaming approach—enabling potentially over 100x faster indexing speeds compared to legacy methods
1234+1.
Core Features and Benefits
-
High-Speed, Parallel Indexing:
Substreams use a horizontally scalable engine to process historical blockchain data in parallel, drastically reducing the time it takes to sync and index large datasets. For example, tasks that previously took weeks may now complete in hours or even minutes
14.
-
Composability and Modularity:
Substreams are built with modular
Rust-based components (Substreams modules), allowing developers to create, compose, and reuse data streams or analytical pipelines. These modules can be shared and leveraged by others, fostering collaboration in the web3 ecosystem
1678.
-
Multi-Chain Support:
Initially targeted at Solana, Substreams now support Ethereum, Polygon, and other chains, opening up fast and efficient querying for many blockchain networks
19.
-
Real-Time and Historical Data:
Substreams can power both real-time notifications and deep, time-series analytics, enabling new classes of dapp features such as block explorers, trading engines, cross-chain bridges, large-scale analytics, and more
16.
-
Integration with Subgraphs:
Substreams can serve as data sources for Subgraphs, combining the efficiency of Substreams with the query capabilities of The Graph’s existing subgraph infrastructure
7105. This allows developers to load enriched data from Substreams directly into familiar subgraph entities for further processing and querying.
How Substreams Work
Traditional subgraphs in The Graph extract, transform, and load blockchain data in a step-by-step, sequential manner. In contrast, Substreams break down the data pipeline into smaller, parallelized operations:
- Extract: Retrieve data using Firehose (streaming data extraction).
- Transform: Process data with parallelized computation modules (Substreams).
- Load: Feed transformed data into databases or other endpoints.
- Query: Access the processed data using GraphQL or other supported APIs23.
Substreams-Powered Subgraphs
By leveraging Substreams as data sources, Subgraphs can achieve unprecedented performance and flexibility. Developers can decide where to place their business logic—either in Substreams modules for high parallelization or in subgraphs for more control over sequential processing
10. Substreams-powered subgraphs have also enabled indexing rewards on The Graph Network, further incentivizing the development and use of this high-performance data infrastructure
68.
Developer Resources
- StreamingFast Substreams documentation offers tutorials and technical guides for getting started68.
- The Graph has provided example repositories and workshop videos to help new developers build Substreams-powered subgraphs quickly8.
In summary:
Substreams are The Graph protocol's innovative answer to bottlenecks in blockchain data processing: they enable massively faster, modular, and more scalable on-chain data extraction and transformation—both as standalone data pipes and as powerful accelerators for subgraphs
1627+6.