tip: 35
title: Local Snapshot File Format
description: File format to export and import ledger state
author: Luca Moser (@luca-moser) , Max Hase (@muXxer) 
discussions-to: https://github.com/iotaledger/tips/pull/25
status: Active
type: Standards
layer: Interface
created: 2022-05-06
replaces: 9

Summary

This TIP defines a file format for local snapshots which is compatible with Stardust. The version of the snapshot file format described in this TIP is Version 2.

Motivation

Nodes create local snapshots to produce ledger representations at a point in time of a given milestone to be able to:

  • Start up from a recent milestone instead of having to synchronize from the genesis transaction.
  • Delete old transaction data below a given milestone.

For Stardust, this file format has to be assimilated to support protocol parameters and to contain the milestone at the point of the snapshot index in order extract startup metadata from it.

Detailed design

Since a UTXO based ledger is much larger in size, this TIP proposes two formats for snapshot files:

  • A full format which represents a complete ledger state.
  • A delta format which only contains diffs (created and consumed outputs) of milestones from a given milestone index onwards.

This separation allows nodes to swiftly create new delta snapshot files, which then can be distributed with a companion full snapshot file to reconstruct a recent state.

Formats

All types are serialized in little-endian

Full Ledger State

A full ledger snapshot file contains the UTXOs (outputs section) of a node's confirmed milestone (Ledger Milestone Index). The diffs contain the diffs to rollback the outputs state to regain the ledger state of the snapshot Target Milestone Index.

While the node producing such a full ledger state snapshot could theoretically pre-compute the actual snapshot milestone state, this is deferred to the consumer of the data to speed up local snapshot creation.

Delta Ledger State

A delta ledger state local snapshot only contains the diffs of milestones starting from a given Full Snapshot Target Milestone Index. A node consuming such data must know the state of the ledger at Full Snapshot Target Milestone Index.

Schema

Output

Defines an output.

Name Type Description
Output ID Array<byte>[34] The ID of the output which is a concatenation of the transaction ID + output index.
Block ID Array<byte>[32] The ID of the Block in which the transaction was contained which generated this output.
Milestone Index Booked uint32 The milestone index at which this output was generated.
Milestone Timestamp Booked uint32 The UNIX timestamp in seconds of the milestone which produced this output.
Output Length uint32 Denotes the length of the output.
Output oneOf
BasicOutput
AliasOutput
FoundryOutput
NFTOutput
Consumed Output

Defines a consumed output.

Name Type Description
Output Array<byte>[Output length] The serialized Output (see above).
Target Transaction ID Array<byte>[32] The ID of the transaction that spent this output.
Milestone Diff

Defines the diff a milestone produced by listing the created/consumed outputs and the milestone payload itself.

Name Type Description
Milestone Diff Length uint32 Denotes the length of the milestone diff.
Milestone Payload Length uint32 Denotes the length of the milestone payload.
Milestone Payload Array<byte>[Milestone Payload length] The milestone payload in its serialized binary form.
Treasury Input
only included if milestone contains a receipt
Name Type Description
Treasury Input Milestone ID Array<byte>[32] The ID of the milestone this input references.
Treasury Input Amount uint64 The amount of this treasury input.
Created Outputs Count uint32 The amount of outputs generated with this milestone diff.
Created Outputs anyOf
Output
Consumed Outputs Count uint32 The amount of outputs consumed with this milestone diff.
Consumed Outputs anyOf
Consumed Output
Protocol Parameters Milestone Option

This Milestone Option is used to signal to nodes the commencing of new protocol parameters, including new protocol version or PoW difficulty.

Name Type Description
Milestone Option Type byte Set to value 1 to denote a Protocol Parameters Milestone Option.
Target Milestone Index uint32 Then name of the network from which this snapshot was generated from.
Protocol Version byte The to be applied protocol version.
Protocol Parameters (uint16)ByteArray The protocol parameters in binary, serialized form.
Protocol Parameters

Defines protocol parameters.

Name Type Description
Protocol Version byte The version of the protocol.
Network Name (uint8)string Then name of the network from which this snapshot was generated from.
Bech32HRP (uint8)string The human-readable part of the addresses within the network.
MinPoWScore uint32 The minimum PoW score.
BelowMaxDepth uint8 The below max depth parameter.
RentStructure
Name Type Description
VByteCost uint32 The token price per virtual byte
VBFactorData uint8 The factor to use for data fields
VBFactorKey uint8 The factor to use for indexed fields
TokenSupply uint64 The token supply.
Full snapshot file format

Defines what a full snapshot file contains.

Name Type Description
Version byte Denotes the version of this file format. (Version 2)
Type byte Denotes the type of this file format. Value 0 denotes a full snapshot.
Genesis Milestone Index uint32 The index of the genesis milestone of the network.
Target Milestone Index uint32 The index of the milestone of which the SEPs within the snapshot are from.
Target Milestone Timestamp uint32 The UNIX timestamp in seconds of the milestone of which the SEPs within the snapshot are from.
Target Milestone ID Array<byte>[32] The ID of the milestone of which the SEPs within the snapshot are from.
Ledger Milestone Index uint32 The index of the milestone of which the UTXOs within the snapshot are from.
Treasury Output Milestone ID Array<byte>[32] The milestone ID of the milestone which generated the treasury output.
Treasury Output Amount uint64 The amount of funds residing on the treasury output.
Protocol Parameters Milestone Option Length uint16 Denotes the length of the Protocol Parameters Milestone Option.
Protocol Parameters Milestone Option
Protocol Parameters Milestone Option that is active at the milestone of which the UTXOs within the snapshot are from.
Outputs Count uint64 The amount of UTXOs contained within this snapshot.
Milestone Diffs Count uint32 The amount of milestone diffs contained within this snapshot.
SEPs Count uint16 The amount of SEPs contained within this snapshot.
Outputs
Output
Milestone Diffs
Milestone Diff
SEPs
SEP Array[32]
Delta snapshot file format

Defines what a delta snapshot contains.

Name Type Description
Version byte Denotes the version of this file format. (Version 2)
Type byte Denotes the type of this file format. Value 1 denotes a delta snapshot.
Target Milestone Index uint32 The index of the milestone of which the SEPs within the snapshot are from.
Target Milestone Timestamp uint32 The UNIX timestamp in seconds of the milestone of which the SEPs within the snapshot are from.
Full Snapshot Target Milestone ID Array<byte>[32] The ID of the target milestone of the full snapshot this delta snapshot builts up from.
SEP File Offset uint64 The file offset of the SEPs field. This is used to easily update an existing delta snapshot without parsing its content.
Milestone Diffs Count uint32 The amount of milestone diffs contained within this snapshot.
SEPs Count uint16 The amount of SEPs contained within this snapshot.
Milestone Diffs
Milestone Diff
SEPs
SEP Array[32]

Updating an existing Delta snapshot file

When creating a delta snapshot, an existing delta snapshot file can be reused.

In order to do that, the following steps need to be done:

  1. Open the existing delta snapshot file.
  2. Read the existing delta snapshot file header.
  3. Verify that Version and Full Snapshot Target Milestone ID match between the existing and new delta snapshot.
  4. Seek to the position of Target Milestone Index and replace it with the new value.
  5. Seek to the position of Target Milestone Timestamp and replace it with the new value.
  6. Seek to the position that is written in the existing SEP File Offset and truncate the file at this position.
  7. Add the additional Milestone Diffs at this position.
  8. Add the new SEPs.
  9. Seek to the position of SEP File Offset and replace it with the new value.
  10. Seek to the position of Milestone Diffs Count and replace it with the new value.
  11. Seek to the position of SEPs Count and replace it with the new value.
  12. Close the file.

Drawbacks

Nodes need to support this new format.

Rationale and alternatives

  • In conjunction with a companion full snapshot, a tool or node can "truncate" the data from a delta snapshot back to a single full snapshot. In that case, the Ledger Milestone Index and Target Milestone Index would be the same. In the example above, given the full and delta snapshots, one could produce a new full snapshot for milestone 1350.
  • Since snapshots may include millions of UTXOs, code generating such files needs to stream data directly onto disk instead of keeping the entire representation in memory. In order to facilitate this, the count denotations for SEPs, UTXOs and diffs are at the beginning of the file. This allows code generating snapshot files to only have to seek back once after the actual count of elements is known.

Copyright

Copyright and related rights waived via CC0.