Resumable Ledger Streams
Read checkpoint, transaction, and event history, then follow live data with resumable streams
streamCheckpoints, streamTransactions, and streamEvents read ledger history in order and then
follow new data. They are available on SuiGrpcClient, SuiGraphQLClient, and on client.core of
either, with the same options and results.
Stream
import { SuiGrpcClient } from '@mysten/sui/grpc';
const client = new SuiGrpcClient({
network: 'mainnet',
baseUrl: 'https://fullnode.mainnet.sui.io:443',
});
const controller = new AbortController();
let count = 0;
for await (const frame of client.streamCheckpoints({ signal: controller.signal })) {
console.log(frame.checkpoint.sequenceNumber, frame.resumeToken);
if (++count === 10) break;
}Requests start when iteration begins. Without a start, a stream follows from the current tip;
start: { checkpoint: 1000 } replays from checkpoint 1000 and then follows live data. Streams
reconnect after transient failures and continue from the last frame, and
onStatus reports the connection state.
break closes the stream. Aborting the signal also closes it, but rejects the pending iteration
with the signal's reason.
try {
for await (const frame of client.streamCheckpoints({ signal: controller.signal })) {
console.log(frame.checkpoint.sequenceNumber);
}
} catch (error) {
if (!controller.signal.aborted) throw error;
}Filter
filter takes the same TransactionFilter and EventFilter as listTransactions and
listEvents, and include takes the same TransactionInclude as listTransactions. Every
predicate you set must match. Shared types are in SuiClientTypes from @mysten/sui/client.
for await (const frame of client.streamTransactions({
filter: { sender: '0x123' },
include: { effects: true, events: true },
start: { checkpoint: 1000 },
})) {
const tx = frame.transaction.Transaction ?? frame.transaction.FailedTransaction;
console.log(tx.digest, tx.effects.status);
}include: { transaction: true } also returns system transactions such as genesis; see
transaction data for how to narrow them.
Index whole checkpoints
include: { transactions } on streamCheckpoints delivers each checkpoint with all of its
transactions, in execution order, as the same Transaction or FailedTransaction results as
streamTransactions. A checkpoint is yielded only once all of its transactions have been read, so
an indexer can commit a checkpoint and its resume token together:
for await (const frame of client.streamCheckpoints({
start: savedToken ? { resumeToken: savedToken } : { checkpoint: 1000 },
include: { transactions: { effects: true, events: true, balanceChanges: true } },
signal: controller.signal,
})) {
await database.transaction(async (db) => {
for (const result of frame.checkpoint.transactions) {
const tx = result.Transaction ?? result.FailedTransaction;
await db.processTransaction(frame.checkpoint.sequenceNumber, tx);
}
await db.saveResumeToken(frame.resumeToken);
});
}Save and resume progress
Each item frame has a resumeToken. Store it with the work done for that item, then pass it back as
the start:
import type { SuiClientTypes } from '@mysten/sui/client';
async function processEvents(savedToken?: SuiClientTypes.StreamResumeToken) {
for await (const frame of client.streamEvents({
filter: { eventType: '0x123::example::Created' },
start: savedToken ? { resumeToken: savedToken } : { checkpoint: 1000 },
signal: controller.signal,
})) {
await database.transaction(async (db) => {
await db.processEvent(frame.event);
await db.saveResumeToken(frame.resumeToken);
});
}
}A resume token start continues after the frame that carried the token, so save a token only once its frame is processed. Commit each item's work and its token together, or make processing idempotent, so a replay after a crash is harmless.
Tokens are opaque strings, so they fit in a text column. Resume on the same transport and chain,
with the same method, filter and order; otherwise the stream throws SuiStreamError with code
IncompatibleToken. include and frames can change between runs.
Read a finite range
Pass an end to stop once a range is covered. Checkpoint starts are inclusive and ends are
exclusive, so this reads checkpoints 1000 through 1999, waiting for the server to index them if
needed:
for await (const frame of client.streamEvents({
start: { checkpoint: 1000 },
end: { checkpoint: 2000 },
signal: controller.signal,
})) {
console.log(frame.event);
}To read the history indexed now and stop, set follow: false. Ascending end and follow: false
both need a start. A token records where to resume, not where to stop, so pass the same end or
follow: false again when resuming a finite read.
client.streamTransactions({
start: { checkpoint: 1000 },
follow: false,
});order: 'descending' reads newest first, from the tip or from start toward genesis, and never
follows. start is still inclusive and end exclusive, so start: { checkpoint: 2000 } with
end: { checkpoint: 1000 } reads 2000 down to 1001.
A descending read without an end completes with reason: 'genesis', or with
reason: 'retentionLimit' at the oldest checkpoint a server that prunes history still keeps:
client.streamTransactions({
filter: { sender: '0x123' },
order: 'descending',
});Any other read of history the server has pruned fails with SuiStreamError code
HistoryUnavailable.
Control frames
By default a stream yields only items. frames adds control frames:
for await (const frame of client.streamEvents({
start: { checkpoint: 1000 },
end: { checkpoint: 2000 },
frames: { completion: true, progress: true },
})) {
switch (frame.$kind) {
case 'Event':
console.log(frame.event);
break;
case 'Progress':
await saveResumeToken(frame.resumeToken);
break;
case 'Complete':
console.log(frame.reason);
break;
}
}| Frame | Option | When |
|---|---|---|
Complete | completion | Once, after a finite range finishes, even an empty one. Not sent on abort, failure or exit. |
Progress | progress | At resumable positions no item reported, so sparse filters can save their place. |
Save a progress token only after every earlier item is processed.
Choose a delivery mode
delivery | Latency | Use when |
|---|---|---|
'subscribe' (default) | Lowest | You want new data as soon as the server has it. |
'poll' | Up to pollInterval ms | Your provider does not serve subscriptions. |
client.streamEvents({
start: { checkpoint: 1000 },
delivery: 'poll',
pollInterval: 1000,
});Finite reads always read history. On GraphQL, 'subscribe' uses
subscriptions, so the provider must serve them.
Connection status
onStatus reports what the stream is doing, for a status indicator, health check or alert. Frames
carry the data and the resume tokens; statuses are only for monitoring.
let state: 'connecting' | 'connected' | 'reconnecting' = 'connecting';
for await (const frame of client.streamEvents({
filter: { eventType: '0x123::example::Created' },
signal: controller.signal,
onStatus(status) {
if (status.$kind === 'Connected') state = 'connected';
if (status.$kind === 'Retrying') {
state = 'reconnecting';
if (status.reason !== 'Unavailable') console.warn(status.reason, status.error);
}
},
})) {
console.log(frame.event);
}Each attempt reports Connecting, then Connected once the server responds. A retryable failure
reports Retrying, and the next attempt starts again from Connecting:
Connecting → Connected → Retrying → Connecting → Connected| Status | Fields | When |
|---|---|---|
Connecting | attempt | An attempt starts. |
Connected | The first page, live frame or opened GraphQL subscription. | |
Retrying | attempt, delay, reason, error | The attempt failed with error; the next starts in delay ms. |
attempt counts consecutive failures without progress, the same count retry.maxAttempts bounds,
so Connecting reports 0 on the first attempt. Retries explains reason.
Connected is reported at most once per attempt.
onStatus is called synchronously and not awaited. Throwing from it ends the stream.
Retries
client.streamEvents({
start: { checkpoint: 1000 },
retry: { initialDelay: 250, maxDelay: 30_000, maxAttempts: 10 },
onStatus(status) {
if (status.$kind === 'Retrying') console.log(status.reason, status.attempt, status.delay);
},
});Failures that may clear retry with exponential backoff, and the stream resumes from its last frame.
maxAttempts (10 by default) bounds consecutive failures without progress; once it runs out, the
stream throws the last error. Progress counts per phase: a failed history or gap page resets the
count if paging moved the resume point, and a failed subscription or live feed resets it only if the
feed itself moved it. maxAttempts: 0 disables retries, including reconnecting a live feed that
ended. Startup requests (the chain identity and the indexed tip) have their own maxAttempts
budget, and the count starts over once the stream begins reading.
reason says why: Unavailable (the server could not be reached, the connection dropped or a live
feed ended) is routine on long-lived streams; RateLimited and ServerError can point at a
problem.
Retries don't bound waiting: a stream waiting for the index or for a sparse filter to match can stay quiet indefinitely. Use a signal to bound the total time:
client.streamEvents({
start: { checkpoint: 1000 },
end: { checkpoint: 2000 },
signal: AbortSignal.timeout(60_000),
});Errors
Once retries are exhausted or a failure is not retryable, transport errors are thrown unchanged:
RpcError on gRPC and SuiGraphQLRequestError on GraphQL, including GraphQL response errors.
Filters that use MVR names throw MvrRequestError when an MVR request fails. Other stream failures
throw SuiStreamError from @mysten/sui/client with a code:
| Code | Cause |
|---|---|
InvalidOptions | An option, bound or filter was rejected. |
InvalidToken | Not a resume token, or an unsupported or corrupted one. |
IncompatibleToken | A token from another stream, or a changed order on resume. |
HistoryUnavailable | The server has pruned history the stream needs. |
Protocol | The server returned an invalid response. |
Disconnected | The live feed kept ending without progress until retries ran out. |
import { SuiStreamError } from '@mysten/sui/client';
try {
for await (const frame of client.streamEvents({ start: { resumeToken: saved } })) {
await process(frame);
}
} catch (error) {
if (error instanceof SuiStreamError && error.code === 'IncompatibleToken') {
// The saved token belongs to another chain, transport, or filter.
}
throw error;
}gRPC options
SuiGrpcClient streams take a grpcFilter: a native TransactionFilter or EventFilter from the
generated types, for predicates the shared filter can't express. On streamTransactions and
streamEvents it replaces filter. On streamCheckpoints it streams only checkpoints that contain
a matching transaction.
For arbitrary GraphQL selection sets, use native subscriptions.