BSV blockchainBSV blockchain

Teranode Prometheus Metrics Reference

Metric Types

TypeDescription
CounterA cumulative metric that represents an increasing counter whose value can only increase OR be reset to zero.
Used for counting events or operations (e.g., number of requests, errors).
CounterVecA Counter that includes additional labels/dimensions.
Allows for breaking down the counter by various labels (e.g., counting errors by type, requests by status code).
GaugeA metric that represents a single numerical value that can arbitrarily go up and down.
Used for measured values like items in a channel or queue.
GaugeVecA Gauge that includes additional labels/dimensions.
Allows for breaking down the gauge by various labels (e.g., index readiness by name).
HistogramSamples observations (such as duration or size) and counts them in configurable buckets.
Also provides a sum of all observed values and count of observations.
Used for measuring distributions of values (e.g., request durations, response sizes).
HistogramVecA Histogram that includes additional labels/dimensions.
Allows for breaking down the histogram by various labels (e.g., duration by state, operation by scan type).

Block Model Metrics

Metric NameTypeDescription
teranode_block_from_bytesHistogramHistogram of Block.FromBytes
teranode_block_validHistogramHistogram of Block.Valid
teranode_block_check_merkle_rootHistogramHistogram of Block.CheckMerkleRoot
teranode_block_get_subtreesHistogramHistogram of Block.GetSubtrees
teranode_block_get_and_validate_subtreesHistogramHistogram of Block.GetAndValidateSubtrees

Alert Service Metrics

Metric NameTypeDescription
teranode_alert_healthCounterNumber of calls to the Health endpoint

Asset Service HTTP Metrics

All metrics are CounterVec type with labels: function (handler function name), operation (operation result).

Metric NameTypeDescription
teranode_asset_http_get_transactionCounterVecNumber of Get transactions ops
teranode_asset_http_get_transactionsCounterVecNumber of Get transactions ops
teranode_asset_http_get_subtreeCounterVecNumber of Get subtree ops
teranode_asset_http_get_block_headerCounterVecNumber of Get block header ops
teranode_asset_http_get_best_block_headerCounterVecNumber of Get best block header ops
teranode_asset_http_get_blockCounterVecNumber of Get block ops
teranode_asset_http_get_block_legacyCounterVecNumber of Get legacy block ops
teranode_asset_http_get_subtree_dataCounterVecNumber of Get subtree data ops
teranode_asset_http_get_last_n_blocksCounterVecNumber of Get last N blocks ops
teranode_asset_http_get_utxoCounterVecNumber of Get UTXO ops
teranode_asset_http_get_merkle_proofCounterVecNumber of Get merkle proof ops

Block Assembly Service Metrics

Metric NameTypeDescription
teranode_blockassembly_healthCounterNumber of calls to the health endpoint of the blockassembly service
teranode_blockassembly_add_txHistogramHistogram of AddTx in the blockassembly service
teranode_blockassembly_remove_txHistogramHistogram of RemoveTx in the blockassembly service
teranode_blockassembly_get_mining_candidate_durationHistogramHistogram of GetMiningCandidate in the blockassembly service
teranode_blockassembly_submit_mining_solution_chGaugeNumber of items in the SubmitMiningSolution channel in the blockassembly service
teranode_blockassembly_submit_mining_solutionHistogramHistogram of SubmitMiningSolution in the blockassembly service
teranode_blockassembly_block_assembler_get_mining_candidateCounterNumber of calls to GetMiningCandidate in the block assembler
teranode_blockassembly_block_assembler_candidate_time_clock_skewCounterMining-candidate polls refused because the parent’s median-time-past floor is above the two-hour future bound, so no valid block timestamp exists; a non-zero rate means the local clock is far behind the network and block production has stopped
teranode_blockassembly_subtree_storedHistogramHistogram of subtree stored duration in block assembler
teranode_blockassembly_transactionsGaugeNumber of transactions currently in the block assembler subtree processor
teranode_blockassembly_queued_transactionsGaugeNumber of transactions currently queued in the block assembler subtree processor
teranode_blockassembly_subtreesGaugeNumber of subtrees currently in the block assembler subtree processor
teranode_blockassembly_dequeue_staleness_secondsGaugeSeconds since the subtree processor’s consumer goroutine last passed through its dequeue branch; growing alongside a non-zero queued_transactions means intake is queuing unboundedly because the consumer is stuck elsewhere
teranode_blockassembly_liveness_heartbeat_age_secondsGaugeSeconds since the main select loop last beat its liveness heartbeat (0 before the loop starts and after it stops); the value blockassembly_livenessStallTimeout is compared against
teranode_blockassembly_tx_meta_getHistogramHistogram of reading tx meta data from txmeta store in block assembler
teranode_blockassembly_reorgCounterNumber of reorgs in block assembler
teranode_blockassembly_reorg_durationHistogramHistogram of reorg in block assembler
teranode_blockassembly_get_reorg_blocks_durationHistogramHistogram of GetReorgBlocks in block assembler
teranode_blockassembly_update_best_blockHistogramHistogram of updating best block in block assembler
teranode_blockassembly_best_block_heightGaugeBest block height in block assembly
teranode_blockassembly_current_block_heightGaugeCurrent block height in block assembly
teranode_blockassembly_generate_blocksHistogramHistogram of generating blocks in block assembler
teranode_blockassembly_current_stateGaugeCurrent state of the block assembly process
teranode_blockassembly_add_tx_counterCounterNumber of transactions added in the blockassembly service
teranode_blockassembly_state_transitions_totalCounterVecTotal number of state transitions (labels: from, to)
teranode_blockassembly_state_duration_secondsHistogramVecTime spent in each state (label: state)
teranode_blockassembly_utxo_index_readyGaugeVecStatus of UTXO index readiness (label: index_name)
teranode_blockassembly_utxo_index_wait_secondsHistogramVecTime taken waiting for UTXO index to become ready (labels: index_name, status)
teranode_blockassembly_get_unmined_tx_iterator_secondsHistogramVecTime taken to get unmined transaction iterator (labels: full_scan, status)
teranode_blockassembly_iterator_processing_secondsHistogramVecTime taken to process all transactions from the unmined iterator (label: full_scan)
teranode_blockassembly_iterator_transactions_totalCounterVecTotal number of transactions processed from the unmined iterator (label: full_scan)
teranode_blockassembly_iterator_transactions_statsCounterVecStatistics about transactions processed from the iterator (labels: full_scan, category)
teranode_blockassembly_mark_transactions_on_longest_chain_secondsHistogramTime taken to mark transactions as mined on longest chain
teranode_blockassembly_mark_transactions_on_longest_chain_totalCounterTotal number of transactions marked as mined on longest chain
teranode_blockassembly_sort_transactions_secondsHistogramVecTime taken to sort unmined transactions by createdAt (label: transaction_count_bucket)
teranode_blockassembly_validate_parent_chain_secondsHistogramTime taken to validate parent chain for unmined transactions
teranode_blockassembly_validate_parent_chain_filtered_totalCounterTotal number of transactions filtered out during parent chain validation
teranode_blockassembly_add_directly_secondsHistogramTime taken for individual AddDirectly calls to subtree processor
teranode_blockassembly_add_directly_totalCounterTotal number of transactions added directly to subtree processor
teranode_blockassembly_add_directly_batch_secondsHistogramTime taken to add all unmined transactions to subtree processor
teranode_blockassembly_queue_shed_totalCounterNumber of ingest batches shed because the queue stayed full past blockassembly_queueFullWaitTimeout. Only possible when blockassembly_maxQueueItems is positive — alert on this
teranode_blockassembly_held_limit_shed_totalCounterNumber of batches shed because block assembly held blockassembly_maxTransactionsInMemory transactions. A subset of queue_shed_total. Sustained growth means no block is being produced or the limit is too tight
teranode_blockassembly_queue_wait_secondsHistogramTime an ingest handler waited for queue room before the batch was accepted or shed
teranode_blockassembly_queue_head_age_secondsGaugeHow long the oldest queued batch has been waiting (0 when the queue is empty). The signal the validator’s Kafka backpressure controller reads
teranode_blockassembly_tip_lag_blocksGaugeNumber of blocks block assembly is behind the blockchain tip, recomputed on each new-block announcement and reset to 0 on a successful advance. Two caveats for alert authors: it reads 0 for an equal-height reorg stall (tip hash differs but height matches), and it holds its last value if announcements stop arriving — pair it with teranode_blockassembly_processing_stuck_total for stall alerting
teranode_blockassembly_stale_tip_candidates_totalCounterMining candidates answered with an empty block on the blockchain tip because block assembly stayed on an older tip for more than 10 seconds. A sustained non-zero rate means block assembly is stuck behind the chain. A shorter gap is the normal delay after a new block and is not counted
teranode_blockassembly_processing_stuck_totalCounterVecCount of block-assembly failures that left the assembler behind the tip or its work unfinished, and of work it gave up. reason is one of reorg_blocks_fetch (fetching the blocks for the reorg/catch-up decision failed), catchup (forward-only catch-up failed), reorg (reorg failed; an expected ErrBlockAssemblyReset is logged as a warning and deliberately not counted), get_block (fetching the announced block failed), moveforward (subtree processor MoveForwardBlock failed), skipped_fork_mark_unfinished (a mark of skipped fork transactions did not finish and is retried), skipped_fork_mark_record (a move, reset or replay give-up was refused because its pending mark could not be recorded), skipped_fork_subtree_lost (a fork block owed that mark has no subtree in the subtree store; its transactions are given up)
teranode_blockassembly_catchupCounterNumber of forward-only catch-ups (moveBack=0) handled in block assembler
teranode_blockassembly_coinbase_divergence_totalCounterVecCoinbase-divergence events by outcome (label: outcome); every detection records exactly one follow-up outcome, so detected == repaired + no_gap + escalated + aborted
teranode_blockassembly_conflict_intents_pendingGaugeNumber of pending conflict-resolution WAL intents found at startup (interrupted ProcessConflicting/ReverseProcessConflicting operations awaiting replay)
teranode_blockassembly_conflict_intent_replay_totalCounterVecTotal conflict-resolution WAL intent replays at startup, by result (label: result)

Blockchain Service Metrics

Metric NameTypeDescription
teranode_blockchain_healthCounterNumber of calls to the health endpoint of the blockchain service
teranode_blockchain_add_blockHistogramHistogram of block added to the blockchain service
teranode_blockchain_get_blockHistogramHistogram of Get block calls to the blockchain service
teranode_blockchain_get_block_statsHistogramHistogram of Get block stats calls to the blockchain service
teranode_blockchain_get_block_graph_dataHistogramHistogram of Get block graph data calls to the blockchain service
teranode_blockchain_get_last_n_blockHistogramHistogram of GetLastNBlocks calls to the blockchain service
teranode_blockchain_get_suitable_blockHistogramHistogram of GetSuitableBlock calls to the blockchain service
teranode_blockchain_get_hash_of_ancestor_blockHistogramHistogram of GetHashOfAncestorBlock calls to the blockchain service
teranode_blockchain_get_next_work_requiredHistogramHistogram of GetNextWorkRequired calls to the blockchain service
teranode_blockchain_get_block_existsHistogramHistogram of GetBlockExists calls to the blockchain service
teranode_blockchain_get_get_best_block_headerHistogramHistogram of GetBestBlockHeader calls to the blockchain service
teranode_blockchain_check_block_is_in_current_chainHistogramHistogram of CheckBlockIsInCurrentChain calls to the blockchain service
teranode_blockchain_get_chain_tipsHistogramHistogram of GetChainTips calls to the blockchain service
teranode_blockchain_get_get_block_headerHistogramHistogram of GetBlockHeader calls to the blockchain service
teranode_blockchain_get_get_block_headersHistogramHistogram of GetBlockHeaders calls to the blockchain service
teranode_blockchain_get_get_block_headers_from_heightHistogramHistogram of GetBlockHeadersFromHeight calls to the blockchain service
teranode_blockchain_get_get_block_headers_by_heightHistogramHistogram of GetBlockHeadersByHeight calls to the blockchain service
teranode_blockchain_get_block_is_minedHistogramHistogram of GetBlockIsMined calls to the blockchain service
teranode_blockchain_subscribeHistogramHistogram of Subscribe calls to the blockchain service
teranode_blockchain_get_stateHistogramHistogram of GetState calls to the blockchain service
teranode_blockchain_set_stateHistogramHistogram of SetState calls to the blockchain service
teranode_blockchain_get_block_header_idsHistogramHistogram of GetBlockHeaderIDs calls to the blockchain service
teranode_blockchain_invalidate_blockHistogramHistogram of InvalidateBlock calls to the blockchain service
teranode_blockchain_revalidate_blockHistogramHistogram of RevalidateBlock calls to the blockchain service
teranode_blockchain_send_notificationHistogramHistogram of SendNotification calls to the blockchain service
teranode_blockchain_set_block_mined_setHistogramHistogram of SetBlockMinedSet calls to the blockchain service
teranode_blockchain_get_blocks_mined_not_setHistogramHistogram of GetBlocksMinedNotSet calls to the blockchain service
teranode_blockchain_set_block_subtrees_setHistogramHistogram of SetBlockSubtreesSet calls to the blockchain service
teranode_blockchain_get_blocks_subtrees_not_setHistogramHistogram of GetBlocksSubtreesNotSet calls to the blockchain service
teranode_blockchain_fsm_current_stateGaugeCurrent state of the blockchain FSM
teranode_blockchain_get_fsm_current_stateHistogramHistogram of GetFSMCurrentState calls to the blockchain service
teranode_blockchain_get_block_locatorHistogramHistogram of GetBlockLocator calls to the blockchain service
teranode_blockchain_locate_block_headersHistogramHistogram of LocateBlockHeaders calls to the blockchain service
teranode_blockchain_get_latest_block_header_from_block_locatorHistogramHistogram of GetLatestBlockHeaderFromBlockLocator calls to the blockchain service
teranode_blockchain_get_block_headers_from_oldestHistogramHistogram of GetBlockHeadersFromOldest calls to the blockchain service
teranode_blockchain_check_block_is_ancestor_of_blockHistogramHistogram of CheckBlockIsAncestorOfBlock calls to the blockchain service
teranode_blockchain_get_blocks_by_heightHistogramHistogram of GetBlocksByHeight calls to the blockchain service
teranode_blockchain_set_block_persisted_atHistogramHistogram of SetBlockPersistedAt calls to the blockchain service
teranode_blockchain_get_blocks_not_persistedHistogramHistogram of GetBlocksNotPersisted calls to the blockchain service

Block Persister Service Metrics

Metric NameTypeDescription
teranode_blockpersister_validate_subtreeHistogramHistogram of subtree validation
teranode_blockpersister_validate_subtree_retryCounterNumber of retries when subtrees validated
teranode_blockpersister_validate_subtree_handlerHistogramHistogram of subtree handler
teranode_blockpersister_persist_blockHistogramHistogram of PersistBlock in the blockpersister service
teranode_blockpersister_bless_missing_transactionHistogramHistogram of bless missing transaction
teranode_blockpersister_set_tx_meta_cache_kafkaHistogramHistogram of setting tx meta cache from kafka
teranode_blockpersister_del_tx_meta_cache_kafkaHistogramDuration of deleting tx meta cache from kafka
teranode_blockpersister_set_tx_meta_cache_kafka_errorsCounterNumber of errors setting tx meta cache from kafka
teranode_blockpersister_blocks_durationHistogramDuration of block processing by the block persister service
teranode_blockpersister_subtrees_durationHistogramDuration of subtree processing by the block persister service
teranode_blockpersister_subtree_batch_durationHistogramDuration of a subtree batch processing by the block persister service

Block Validation Service Metrics

Metric NameTypeDescription
teranode_blockvalidation_healthCounterNumber of health checks
teranode_blockvalidation_block_found_chGaugeNumber of blocks found buffered in the block found channel
teranode_blockvalidation_block_foundHistogramHistogram of calls to BlockFound method
teranode_blockvalidation_catchup_chGaugeNumber of catchups buffered in the catchup channel
teranode_blockvalidation_catchupHistogramHistogram of catchup events
teranode_blockvalidation_process_block_foundHistogramHistogram of process block found
teranode_blockvalidation_validate_blockHistogramHistogram of calls to ValidateBlock method
teranode_blockvalidation_revalidate_blockHistogramHistogram of re-validate block
teranode_blockvalidation_revalidate_block_errHistogramNumber of blocks revalidated with error
teranode_blockvalidation_last_validated_blocks_cacheGaugeNumber of blocks in the last validated blocks cache
teranode_blockvalidation_block_exists_cacheGaugeNumber of blocks in the block exists cache
teranode_blockvalidation_subtree_exists_cacheGaugeNumber of subtrees in the subtree exists cache
teranode_blockvalidation_recovered_panics_totalCounterVecPanics recovered in block validation, by path label: worker, catchup, kafka or block_found, or for an errgroup goroutine those start, catchup_fetch, block_validation or service. Each one is a bug in the node. The stack, and on most paths the block hash, are in the PANIC log line

Catchup Operation Metrics

CounterVec and HistogramVec metrics use labels: peer_id, success, error_type, priority, result, fork_id, reason.

Metric NameTypeDescription
teranode_blockvalidation_catchup_duration_secondsHistogramVecDuration of catchup operations in seconds
teranode_blockvalidation_catchup_blocks_fetched_totalCounterVecTotal number of blocks fetched during catchup
teranode_blockvalidation_catchup_headers_fetched_totalCounterVecTotal number of headers fetched during catchup
teranode_blockvalidation_catchup_errors_totalCounterVecTotal number of errors during catchup operations
teranode_blockvalidation_catchup_activeGaugeNumber of active catchup operations (0 or 1)
teranode_blockvalidation_catchup_prefetch_budget_parked_totalCounterTotal times a prewarm worker parked waiting for prefetch budget
teranode_blockvalidation_catchup_prefetch_oversized_blocks_totalCounterTotal blocks declaring a size over the budget, parsed one subtree at a time
teranode_blockvalidation_catchup_prefetch_undeclared_size_blocks_totalCounterTotal blocks declaring no size, parsed one subtree at a time
teranode_blockvalidation_priority_queue_sizeGaugeVecCurrent size of the block priority queue by priority level
teranode_blockvalidation_priority_queue_added_totalCounterVecTotal number of blocks added to priority queue by priority level
teranode_blockvalidation_priority_queue_processed_totalCounterVecTotal number of blocks processed from priority queue
teranode_blockvalidation_fork_countGaugeCurrent number of active forks being tracked
teranode_blockvalidation_fork_processing_workersGaugeCurrent number of active fork processing workers
teranode_blockvalidation_fork_blocks_processed_totalCounterVecTotal number of blocks processed on forks by fork ID
teranode_blockvalidation_fork_created_totalCounterVecTotal number of forks created
teranode_blockvalidation_fork_lifetime_secondsHistogramLifetime of forks in seconds
teranode_blockvalidation_fork_depth_blocksHistogramDepth of forks in blocks
teranode_blockvalidation_fork_resolved_totalCounterVecTotal number of forks resolved
teranode_blockvalidation_fork_resolution_depth_blocksHistogramDepth at which forks were resolved in blocks
teranode_blockvalidation_fork_orphaned_totalCounterTotal number of forks marked as orphaned
teranode_blockvalidation_processing_blocks_stuckGaugeNumber of blocks stuck in processing state
teranode_blockvalidation_fork_average_depthGaugeAverage depth of active forks in blocks
teranode_blockvalidation_fork_longest_depthGaugeLongest active fork depth in blocks
teranode_blockvalidation_fork_average_lifetime_secondsGaugeAverage lifetime of active forks in seconds
teranode_blockvalidation_queue_skip_countHistogramNumber of times blocks were skipped before processing
teranode_blockvalidation_queue_wait_secondsHistogramTime blocks spend in queue before processing
teranode_blockvalidation_outpoint_only_blocks_totalCounterBlocks that entered the below-checkpoint outpoint-only fast path, counted once on entry; a block that later falls back to normal validation is still counted
teranode_blockvalidation_quick_commit_add_block_secondsHistogramAddBlock duration in the quick-validation commit tail
teranode_blockvalidation_quick_commit_unlock_secondsHistogramUnlock pass duration in the quick-validation commit tail
teranode_blockvalidation_quick_commit_block_exists_secondsHistogramSetBlockExists duration in the quick-validation commit tail

P2P Service Metrics

Metric NameTypeLabelsDescription
teranode_p2p_publish_blocked_totalCountertopic, fsm_state, stageOutbound P2P messages suppressed by the per-FSM-state allow-list; stage="precheck" is an expected skip, stage="chokepoint" is a publish that leaked past the pre-checks
teranode_p2p_websocket_notifications_dropped_totalCountertypeWebSocket notifications dropped because the shared notification channel was full, by notification type (block, subtree, node_status)
teranode_p2p_websocket_clients_evicted_totalCounterWebSocket clients evicted from the broadcast fan-out because their send buffer was full when a broadcast reached them
teranode_p2p_connected_peersGauge-Number of peers currently connected to this node over the p2p network (directly connected only, matching node_status.connected_peers_count)
teranode_p2p_ban_events_totalCounterreasonTotal number of peer ban events; reason is one of protocol_violation, invalid_subtree, invalid_block, spam, operator_ban (BanPeer RPC), or unknown
teranode_p2p_catchup_attempts_totalCounter-Total number of catchup attempts made against peers
teranode_p2p_catchup_successes_totalCounter-Total number of successful catchups completed from peers
teranode_p2p_catchup_failures_totalCounterkindTotal number of failed catchup attempts against peers; kind is one of generic or block_incomplete
teranode_p2p_websocket_connectionsGauge-Number of currently connected websocket notification subscribers

Legacy Peer Server Metrics

Each metric measures “The time taken to handle a specific legacy action handler”.

Metric NameTypeDescription
teranode_legacy_peer_server_OnVersionHistogramThe time taken to handle OnVersion
teranode_legacy_peer_server_OnProtoconfHistogramThe time taken to handle OnProtoconf
teranode_legacy_peer_server_OnMemPoolHistogramThe time taken to handle OnMemPool
teranode_legacy_peer_server_OnTxHistogramThe time taken to handle OnTx
teranode_legacy_peer_server_OnBlockOnDiskHistogramThe time taken to hand a streamed block to the sync manager’s queue
teranode_legacy_peer_server_OnInvHistogramThe time taken to handle OnInv
teranode_legacy_peer_server_OnHeadersHistogramThe time taken to handle OnHeaders
teranode_legacy_peer_server_OnGetDataHistogramThe time taken to handle OnGetData
teranode_legacy_peer_server_OnGetBlocksHistogramThe time taken to handle OnGetBlocks
teranode_legacy_peer_server_OnGetHeadersHistogramThe time taken to handle OnGetHeaders
teranode_legacy_peer_server_OnFeeFilterHistogramThe time taken to handle OnFeeFilter
teranode_legacy_peer_server_OnGetAddrHistogramThe time taken to handle OnGetAddr
teranode_legacy_peer_server_OnAddrHistogramThe time taken to handle OnAddr
teranode_legacy_peer_server_OnRejectHistogramThe time taken to handle OnReject
teranode_legacy_peer_server_OnNotFoundHistogramThe time taken to handle OnNotFound
teranode_legacy_peer_server_OnReadHistogramThe time taken to handle OnRead
teranode_legacy_peer_server_OnWriteHistogramThe time taken to handle OnWrite
teranode_legacy_peer_server_OnCreateStreamHistogramThe time taken to handle OnCreateStream
teranode_legacy_peer_server_OnStreamAckHistogramThe time taken to handle OnStreamAck

Legacy NetSync Service Metrics

The download gauges and the download_*_total counters are published from the 30-second download queue report, so they lag by up to 30 seconds and the counters move in steps.

Metric NameTypeDescription
teranode_legacy_netsync_block_heightGaugeThe height of the block being processed
teranode_legacy_netsync_handle_tx_msgHistogramThe time taken to handle a tx message
teranode_legacy_netsync_handle_tx_msg_validateHistogramThe time taken to validate a tx message
teranode_legacy_netsync_process_orphan_transactionsHistogramThe time taken to process orphan transactions
teranode_legacy_netsync_handle_block_directHistogramThe time taken to commit a converted block (HandleConvertedBlock); the name predates the converted-record route
teranode_legacy_netsync_process_blockHistogramThe time taken to process a block
teranode_legacy_netsync_orphansGaugeThe number of orphan transactions
teranode_legacy_netsync_orphans_abandoned_totalCounterOrphan transactions dropped for good: evicted from the orphan pool and still invalid on the final validation attempt
teranode_legacy_netsync_relayed_tx_dropped_totalCounterTransactions relayed by legacy peers and dropped unvalidated: the relay queue was full or over its byte budget, the peer left before it was registered, or the sync manager stopped
teranode_legacy_netsync_inv_dropped_totalCounterVecTransaction announcements from legacy peers dropped without being asked for, in hashes, by reason (label reason): peer_share when the peer was over its share, node_cap when the node was over legacy_invQueueMaxHashes, handlers_busy when every legacy_invHandlerSlots handler slot for transaction-only messages was taken, request_queue when the peer already had 50,000 missing transactions waiting for a getdata, lookup_failed when the store read failed
teranode_legacy_netsync_inv_queued_hashesGaugeTransaction announcements from legacy peers held from the read loop until a lookup worker finishes with them, in hashes; legacy_invQueueMaxHashes bounds it
teranode_legacy_netsync_orphan_timeHistogramThe time taken to process an orphan transaction
teranode_legacy_netsync_parked_blocksGaugeThe number of downloaded blocks held on disk waiting for their parent
teranode_legacy_netsync_parked_bytesGaugeBytes the park bills for the blocks it holds: a block parked off the wire at its wire size, a block recovered from disk after a restart at its converted record’s size; bytes_ahead is what the backstop reads
teranode_legacy_netsync_frontier_races_totalCounterBlocks asked of a second peer because the chain was about to wait on them from a slow one
teranode_legacy_netsync_blocks_owedGaugeBlocks asked of peers and not yet delivered, every peer together, including blocks arriving now
teranode_legacy_netsync_header_cache_heightsGaugeHeights the headers-first header cache names, the header run the download pass picks blocks from below the last checkpoint
teranode_legacy_netsync_bytes_aheadGaugeBlock bytes held ahead of the chain, parked plus arriving, which the download measures against park_backstop_bytes
teranode_legacy_netsync_park_backstop_bytesGaugeThe bytes_ahead figure at which no block further ahead is asked for until the chain catches up; a constant (100 GiB)
teranode_legacy_netsync_download_received_bytes_totalCounterBlock body bytes read off the wire
teranode_legacy_netsync_download_duplicate_copies_drained_totalCounterBlock copies drained unwritten because another copy of the same block was converting
teranode_legacy_netsync_download_duplicate_copies_converted_totalCounterBlock copies converted in full for a block already parked
teranode_legacy_netsync_download_local_fault_copies_drained_totalCounterBlock copies drained because this node failed to store the block; the peer was kept and the block asked for again
teranode_legacy_netsync_download_streams_cut_totalCounterBlock bodies cut part way through
teranode_legacy_netsync_download_bytes_wasted_totalCounterBytes of block bodies cut part way through and of every copy drained unwritten, duplicate or local fault
teranode_legacy_netsync_download_peers_dropped_owing_totalCounterPeers that left while still owing blocks
teranode_legacy_netsync_download_blocks_owed_at_drop_totalCounterBlocks the peers counted by download_peers_dropped_owing_total still owed when they left
teranode_legacy_netsync_download_blocks_reasked_totalCounterBlocks made askable of another peer because the peers owing them sent no block bytes for the retry window
teranode_legacy_netsync_park_dispositions_totalCounterVecEvery disposition applied to a parked block, by the park policy row’s name (label disposition): a commit, a read or commit failure, a record whose files are gone, and a sweep or recovery eviction. One series per row exists from start-up at zero

Propagation Service Metrics

Metric NameTypeDescription
teranode_propagation_healthHistogramHistogram of calls to the health endpoint of the propagation service
teranode_propagation_transactionsHistogramHistogram of transaction processing by the propagation service
teranode_propagation_transactions_batchHistogramHistogram of transaction processing by the propagation service
teranode_propagation_handle_single_txHistogramHistogram of transaction processing by the propagation service using HTTP
teranode_propagation_handle_multiple_txHistogramHistogram of multiple transaction processing by the propagation service using HTTP
teranode_propagation_transactions_sizeHistogramSize of transactions processed by the propagation service
teranode_propagation_invalid_transactionsCounterNumber of transactions found invalid by the propagation service

RPC Service Metrics

Metric NameTypeDescription
teranode_rpc_get_blockHistogramHistogram of calls to handleGetBlock in the rpc service
teranode_rpc_get_block_by_heightHistogramHistogram of calls to handleGetBlockByHeight in the rpc service
teranode_rpc_get_block_hashHistogramHistogram of calls to handleGetBlockHash in the rpc service
teranode_rpc_get_block_headerHistogramHistogram of calls to handleGetBlockHeader in the rpc service
teranode_rpc_get_best_block_hashHistogramHistogram of calls to handleGetBestBlockHash in the rpc service
teranode_rpc_get_raw_transactionHistogramHistogram of calls to handleGetRawTransaction in the rpc service
teranode_rpc_create_raw_transactionHistogramHistogram of calls to handleCreateRawTransaction in the rpc service
teranode_rpc_send_raw_transactionHistogramHistogram of calls to handleSendRawTransaction in the rpc service
teranode_rpc_generateHistogramHistogram of calls to handleGenerate in the rpc service
teranode_rpc_generate_to_addressHistogramHistogram of calls to handleGenerateToAddress in the rpc service
teranode_rpc_get_mining_candidateHistogramHistogram of calls to handleGetMiningCandidate in the rpc service
teranode_rpc_submit_mining_solutionHistogramHistogram of calls to handleSubmitMiningSolution in the rpc service
teranode_rpc_get_peer_infoHistogramHistogram of calls to handleGetpeerinfo in the rpc service
teranode_rpc_get_rawmempoolHistogramHistogram of calls to handleGetRawmempool in the rpc service
teranode_rpc_get_blockchain_infoHistogramHistogram of calls to handleGetblockchaininfo in the rpc service
teranode_rpc_get_infoHistogramHistogram of calls to handleGetinfo in the rpc service
teranode_rpc_get_difficultyHistogramHistogram of calls to handleGetDifficulty in the rpc service
teranode_rpc_invalidate_blockHistogramHistogram of calls to handleInvalidateBlock in the rpc service
teranode_rpc_reconsider_blockHistogramHistogram of calls to handleReconsiderBlock in the rpc service
teranode_rpc_helpHistogramHistogram of calls to handleHelp in the rpc service
teranode_rpc_set_banHistogramHistogram of calls to handleSetBan in the rpc service
teranode_rpc_is_bannedHistogramHistogram of calls to handleIsBanned in the rpc service
teranode_rpc_get_mining_infoHistogramHistogram of calls to handleGetMiningInfo in the rpc service
teranode_rpc_list_bannedHistogramHistogram of calls to handleListBanned in the rpc service
teranode_rpc_clear_bannedHistogramHistogram of calls to handleClearBanned in the rpc service
teranode_rpc_freezeHistogramHistogram of calls to handleFreeze in the rpc service
teranode_rpc_unfreezeHistogramHistogram of calls to handleUnfreeze in the rpc service
teranode_rpc_reassignHistogramHistogram of calls to handleReassign in the rpc service
teranode_rpc_get_chaintipsHistogramHistogram of calls to handleGetChainTips in the rpc service

Subtree Validation Service Metrics

Metric NameTypeDescription
teranode_subtreevalidation_healthHistogramHistogram of calls to health endpoint
teranode_subtreevalidation_check_subtreeHistogramDuration of calls to checkSubtree endpoint
teranode_subtreevalidation_validate_subtreeHistogramHistogram of subtrees validated
teranode_subtreevalidation_validate_subtree_retryCounterNumber of retries when subtrees validated
teranode_subtreevalidation_validate_subtree_handlerHistogramDuration of subtree handler
teranode_subtreevalidation_validate_subtree_durationHistogramDuration of validate subtree
teranode_subtreevalidation_bless_missing_transactionHistogramDuration of bless missing transaction
teranode_subtreevalidation_bless_unlock_failedCounterAccepted transactions blessed although their two-phase-commit unlock failed
teranode_subtreevalidation_bless_unlock_recoveredCounterSubset of bless_unlock_failed that the per-level unlock retry made spendable
teranode_subtreevalidation_conflicting_kept_asideCounterTransactions of peer subtrees kept aside because they conflict with accepted transactions, until a block references them, counted per validation pass
teranode_subtreevalidation_bless_unlock_retry_failed_batchesCounterUnlock retry calls that returned an error; counted per call, not per transaction, because the store reports no per-record result
teranode_subtreevalidation_set_tx_meta_cache_kafka_batchHistogramDuration of one SetCacheMulti batch from a Kafka txmeta message (per Kafka message, not per entry)
teranode_subtreevalidation_set_tx_meta_cache_kafka_countCounterNumber of ADD entries processed from Kafka (per-entry; mirrors the pre-batching histogram count)
teranode_subtreevalidation_del_tx_meta_cache_kafkaHistogramDuration of deleting tx meta cache from kafka
teranode_subtreevalidation_set_tx_meta_cache_kafka_errorsCounterNumber of errors setting tx meta cache from kafka
teranode_subtreevalidation_kafka_malformed_messages_totalCounterMalformed Kafka subtree messages dropped, labelled by reason (nil_message, too_short, unmarshal_failure, bad_hash, bad_url)
teranode_subtreevalidation_subtree_already_exists_skipped_totalCounterSubtree Kafka messages skipped because the subtree already exists. Benign individually; the per-message log line is at DEBUG, so watch this rate instead
teranode_subtreevalidation_cached_subtree_conflict_probes_totalCounterTransactions of already-cached subtrees whose conflicting flag was read from the UTXO store during block blessing. Scales with transactions per block
teranode_subtreevalidation_peer_subtree_admission_waitHistogramSeconds a peer subtree waited for a subtreevalidation_peerSubtreeConcurrency slot
teranode_subtreevalidation_peer_subtree_admission_waitingGaugePeer subtrees waiting for a subtreevalidation_peerSubtreeConcurrency slot

Validator Service Metrics

Metric NameTypeDescription
teranode_validator_healthCounterNumber of calls to the health endpoint
teranode_validator_invalid_transactionsCounterNumber of transactions found invalid by the validator service
teranode_validator_transactions_validate_totalHistogramHistogram of total transaction validation
teranode_validator_transactions_validateHistogramHistogram of transaction validation
teranode_validator_transactions_extendHistogramHistogram of transaction extension operations
teranode_validator_transactions_validate_scriptsHistogramHistogram of transaction script validation
teranode_validator_transactions_validate_batchHistogramHistogram of transaction batch validation
teranode_validator_transactions_spend_utxosHistogramHistogram of transaction spending utxos
teranode_validator_transactions_input_block_heightsHistogramHistogram of transaction input block heights
teranode_validator_transactions_2phase_commitHistogramHistogram of 2-phase commit operations
teranode_validator_transactionsHistogramHistogram of transaction processing by the validator service
teranode_validator_transactions_sizeHistogramSize of transactions processed by the validator service
teranode_validator_send_to_block_assemblyHistogramHistogram of sending transactions to block assembly
teranode_validator_send_to_blockvalidation_kafkaHistogramHistogram of sending transactions to block validation kafka
teranode_validator_send_to_p2p_kafkaHistogramHistogram of sending rejected transactions to p2p kafka
teranode_validator_set_tx_metaHistogramHistogram of validator set tx meta
teranode_validator_parent_commit_retriesCounterRetries spent waiting for a parent transaction to finish committing, by condition (TX_LOCKED, TX_CREATING)
teranode_validator_parent_commit_exhaustedCounterTransactions rejected because the parent-commit retry budget ran out, by condition (TX_LOCKED, TX_CREATING)
teranode_validator_unlock_failedCounterAccepted transactions whose two-phase-commit unlock failed or timed out; they stay locked until mined, so their descendants are answered TX_LOCKED until then. Subtree validation retries them (teranode_subtreevalidation_bless_unlock_*)
teranode_validator_shed_unwind_totalCounterNumber of times a queue-full shed’s store work was unwound (record deleted, then inputs unspent)
teranode_validator_shed_unwind_failures_totalCounterNumber of shed unwinds whose delete or unspend returned an error; pair with shed_unwind_residue_total, shed_unwind_aborted_total and shed_unwind_unverified_total to see what the failure left behind
teranode_validator_shed_unwind_aborted_totalCounterNumber of shed unwinds abandoned because the record was still readable after a delete the store reported as successful (a store-contract violation; fix the wiring)
teranode_validator_shed_unwind_unverified_totalCounterNumber of shed unwinds abandoned because the record’s deletion could not be confirmed after the bounded retry, leaving its inputs spent (reconcile the logged outpoints)
teranode_validator_shed_unwind_residue_totalCounterNumber of shed unwinds where the master record was deleted but the rest of the cascade failed. Nothing mineable survives and the inputs are then unspent unless the unspend itself fails, in which case the outpoints are logged for recovery (shed_unwind_failures_total counts the unwind once, whether one operation in it failed or both). What can remain is orphan pagination children — locked, unspendable, and cleared only when a create of the same transaction adopts them — and/or an external blob, which carries no UTXO semantics and is kept as-is by a later create
teranode_validator_shed_unwind_reappeared_totalCounterNumber of shed unwinds aborted because the record was conclusively readable again immediately before the unspend, so another submission of the same txid now owns those spends and clearing them would free the inputs of a live transaction. This is the signal that concurrent same-txid submissions are real in this deployment and that per-txid serialisation of create, handoff and unwind is needed. Distinct from shed_unwind_unverified_total, which is the inconclusive-read case rather than a record that was conclusively back
teranode_validator_shed_unwind_unowned_record_totalCounterNumber of spend-only (SkipUtxoCreation) shed unwinds aborted because a record this call did not create is present, so its inputs must not be freed. Kept apart from the reappearance counter: that one means a record came back after this call deleted it, this one means a record was there all along — a different operator problem
teranode_validator_shed_dropped_totalCounterNumber of transactions dropped after the bounded block assembly handoff retry on the Kafka ingest path. Propagation already returned success to the submitter, so these drops are silent to the client — alert on this
teranode_validator_shed_inblock_dropped_totalCounterNumber of block-context transactions kept Locked outside the block assembly template because the ingest queue was full. The transaction is valid and not mineable by this node, and an ordinary child of it is refused as locked, until a block unlocks it or the next unmined reload. The validator schedules a check five minutes after the first such shed and resets block assembly if any of them is still Locked and unmined, see teranode_validator_shed_inblock_recovery_resets_total. The reload reads Locked unmined records, adds them to the template and unlocks them
teranode_validator_shed_inblock_recovery_resets_totalCounterNumber of block assembly resets the validator requested because transactions an in-block shed kept Locked were still outside the template. A rising value means blocks are not unlocking those records, so look at block validation
teranode_validator_held_limit_shed_transactionsCounterNumber of Kafka transactions refused because block assembly holds blockassembly_maxTransactionsInMemory transactions, shed by the validator before the UTXO store or refused by block assembly. They are valid and are not counted as invalid; see teranode_validator_shed_dropped_total for the drops
teranode_validator_handoff_deadline_totalCounterNumber of block assembly handoffs that hit the validator’s own handoff deadline, so the shed was neither classified nor unwound. Recurring values suggest blockassembly_queueFullWaitTimeout differs between this process and block assembly
teranode_validator_existing_tx_locked_unmined_totalCounterNumber of resubmits that found an existing transaction record locked, unmined and not conflicting. Causes include a conflict resolution in flight, an interrupted submission and a block-context transaction left outside the template by a queue shed
teranode_validator_kafka_backpressure_pausedGauge1 while the backpressure controller has the tx Kafka consumer paused, 0 otherwise
teranode_validator_kafka_backpressure_pause_totalCounterNumber of pause transitions made by the backpressure controller
teranode_validator_kafka_backpressure_resume_totalCounterNumber of resume transitions, including fail-open and max-pause resumes
teranode_validator_kafka_backpressure_paused_seconds_totalCounterTotal seconds the tx consumer has spent paused by the controller
teranode_validator_kafka_backpressure_pause_secondsGaugeSeconds the current backpressure pause has run, with a cause label of age, fill or held, and 0 once the consumer resumes. A pause caused only by the held transaction count is exempt from validator_kafkaBackpressureMaxPause and ends at the retention of the validator transaction topic, so alert on a long held value well before that
teranode_validator_kafka_backpressure_read_errorsGaugeCurrent consecutive queue-stats read-error streak (resets to 0 on a good read)

TxMetaCache Service Metrics

Metric NameTypeDescription
teranode_tx_meta_cache_sizeGaugeNumber of items in the tx meta cache (commented out in code, not emitted)
teranode_tx_meta_cache_insertionsGaugeNumber of insertions into the tx meta cache
teranode_tx_meta_cache_hitsGaugeNumber of hits in the tx meta cache
teranode_tx_meta_cache_missesGaugeNumber of misses in the tx meta cache
teranode_tx_meta_cache_get_originGaugeNumber of get origins in the tx meta cache
teranode_tx_meta_cache_evictionsGaugeNumber of entries removed from the tx meta cache: deletes on every backend, plus an estimate of entries overwritten by ring wraps on the lockfree cache
teranode_tx_meta_cache_trimsGaugeNumber of trim operations in the tx meta cache
teranode_tx_meta_cache_map_sizeGaugeNumber of total elements in the improved cache’s bucket maps
teranode_tx_meta_cache_total_elements_addedGaugeNumber of total number of elements added to the txmetacache
teranode_tx_meta_cache_valid_entries_countGaugeNumber of valid (readable) entries in the tx meta cache
teranode_tx_meta_cache_current_gen_entriesGaugeNumber of entries in the current generation of the tx meta cache
teranode_tx_meta_cache_previous_gen_entriesGaugeNumber of valid entries from the previous generation in the tx meta cache
teranode_tx_meta_cache_lane_waitsGaugeNumber of times a lockfree cache writer found every lane busy; should stay 0
teranode_tx_meta_cache_tombstoned_writes_evicted_totalCounterWrites of a tx demoted to conflicting that were dropped, or evicted right after landing when the demotion arrived during the write: a txmeta ADD consumed after the demotion’s DELETE (they travel on different partitions), or a read-through write whose store read raced the demotion. Non-zero means the race happened and was absorbed

Aerospike Service Metrics

CounterVec metrics use labels: function (function name), error (error type).

Metric NameTypeDescription
teranode_aerospike_txmeta_getCounterNumber of txmeta get calls done to aerospike
teranode_aerospike_utxo_storeCounterNumber of Create calls done to aerospike
teranode_aerospike_txmeta_errorsCounterVecNumber of txmeta map errors
teranode_aerospike_txmeta_get_multiCounterNumber of txmeta get_multi calls done to aerospike map
teranode_aerospike_txmeta_get_multi_nCounterNumber of txmeta get_multi txs done to aerospike map
teranode_aerospike_txmeta_set_mined_batchCounterNumber of txmeta set_mined_batch calls done to aerospike map
teranode_aerospike_txmeta_set_mined_batch_nCounterNumber of txmeta set_mined_batch txs done to aerospike map
teranode_aerospike_txmeta_set_mined_batch_err_nCounterNumber of txmeta set_mined_batch txs errors to aerospike map
teranode_aerospike_expired_preservation_restore_failedCounterExpired preservations whose master could not be restored after a failed child DAH follow-up
teranode_aerospike_longest_chain_restore_failedCounterLongest-chain DAH updates whose master could not be restored after a failed pagination-record write
teranode_aerospike_utxo_getCounterNumber of utxo get calls done to aerospike
teranode_aerospike_utxo_spendCounterNumber of utxo spend calls done to aerospike
teranode_aerospike_utxo_resetCounterNumber of utxo reset calls done to aerospike
teranode_aerospike_utxo_deleteCounterNumber of utxo delete calls done to aerospike
teranode_aerospike_utxo_errorsCounterVecNumber of utxo errors
teranode_aerospike_utxo_create_batchHistogramDuration of utxo create batch
teranode_aerospike_utxo_create_batch_sizeHistogramSize of utxo create batch
teranode_aerospike_utxo_spend_batchHistogramDuration of utxo spend batch
teranode_aerospike_utxo_spend_batch_sizeHistogramSize of utxo spend batch
teranode_aerospike_get_externalHistogramDuration of getting an external transaction from the blob store
teranode_aerospike_set_externalHistogramDuration of setting an external transaction to the blob store
teranode_aerospike_txmeta_get_counter_conflictingHistogramDuration of getting counter conflicting transactions from the blob store
teranode_aerospike_txmeta_get_conflictingHistogramDuration of getting conflicting transactions from the blob store

SQL Service Metrics

CounterVec metrics use labels: function (function name), error (error type).

Metric NameTypeDescription
teranode_sql_utxo_getCounterNumber of utxo get calls done to sql
teranode_sql_utxo_spendCounterNumber of utxo spend calls done to sql
teranode_sql_utxo_resetCounterNumber of utxo reset calls done to sql
teranode_sql_utxo_deleteCounterNumber of utxo delete calls done to sql
teranode_sql_utxo_errorsCounterVecNumber of utxo errors
teranode_sql_utxo_get_counter_conflictingHistogramHistogram of utxo get counter conflicting calls done to sql
teranode_sql_utxo_get_conflictingHistogramHistogram of utxo get conflicting calls done to sql

Subtree Processor Service Metrics

Metric NameTypeDescription
teranode_subtreeprocessor_add_txCounterNumber of times a tx is added in subtree processor
teranode_subtreeprocessor_move_forwardCounterNumber of times a block is moved up in subtree processor
teranode_subtreeprocessor_move_forward_durationHistogramHistogram of moving up block in subtree processor
teranode_subtreeprocessor_move_backCounterNumber of times a block is moved down in subtree processor
teranode_subtreeprocessor_move_back_durationHistogramHistogram of moving down block in subtree processor
teranode_subtreeprocessor_process_coinbase_txCounterNumber of times a coinbase tx is processed in subtree processor
teranode_subtreeprocessor_process_coinbase_tx_durationHistogramDuration of processing coinbase tx in subtree processor
teranode_subtreeprocessor_transaction_mapCounterNumber of times a transaction map is created in subtree processor
teranode_subtreeprocessor_transaction_map_durationHistogramDuration of creating transaction map in subtree processor
teranode_subtreeprocessor_remove_txHistogramDuration of removing tx in subtree processor
teranode_subtreeprocessor_resetHistogramDuration of resetting subtree processor
teranode_subtreeprocessor_dynamic_subtree_sizeGaugeSize of the dynamic subtree in the subtree processor
teranode_subtreeprocessor_blockmaxsize_resizesCounterNumber of times the queued subtrees were re-chunked because the first one exceeded blockmaxsize. A resize lasts for one block, so a count that rises every block means the queue is rebuilt and stored again each block
teranode_subtreeprocessor_current_stateGaugeCurrent state of the subtree processor
teranode_subtreeprocessor_oversize_batch_admitted_totalCounterNumber of client batches that exceeded the normalized blockassembly_maxQueueItems and were admitted alone onto an empty queue, so the ceiling is transiently exceeded by one batch rather than that client wedging forever. Any non-zero value means a client’s batch size is above this pod’s normalized cap and the cross-pod sizing guidance in the blockassembly_maxQueueItems longdesc has not been applied
teranode_subtreeprocessor_txmeta_delete_dropped_totalCounterTxmeta cache DELETEs for txs demoted to conflicting that block assembly dropped because the txmeta producer buffer was full or stopped. Each one can leave subtree validation caching that tx as non-conflicting until the cache rotates it out, so alert on any increase