BSV blockchainBSV blockchain

Run an overlay with @bsv/overlay-express

Use @bsv/overlay-express to run an overlay serviceA service that indexes an app's transactions. Topic managers decide which outputs it admits, and lookup services answer queries about them for your own app on an Express server. The package wires the overlay engine to the storage, the network settings and the HTTP routes. You register your topics and lookup services, then start the server. For the full walkthrough, with a transaction sent to the host and looked up, follow Track transactions with overlays.

Install

The latest version on npm is 2.8.0, with the status Stable. It needs Node.js 22 or newer and @bsv/sdk. This page was written against @bsv/overlay-express 2.8.0, @bsv/overlay-topics 2.0.3 and @bsv/sdk 4.0.1. For a local host, add the topics package, which is Experimental, and the storage packages. In a project folder with "type": "module" in its package.json, run:

npm install @bsv/overlay-express@2.8.0 @bsv/overlay-topics@2.0.3 @bsv/sdk@4.0.1 mongodb-memory-server better-sqlite3 knex

What the package includes

The package connects the overlay engine to storage, HTTP routes and the network:

PieceWhat it does
OverlayExpressCreates the server and exposes the configure methods and start
Topic managers and lookup servicesDecide which transaction outputs the host admits and answer queries about them. Prebuilt ones, such as tm_helloworld, come from @bsv/overlay-topics
StorageA SQL database through Knex for the host’s own data, and MongoDB for the lookup services
HTTP routes/submit and /lookup for apps, health routes, a web UI, and /admin routes for the operator
Network servicesGASPGraph Aware Sync Protocol, how overlay hosts copy each other's data synchronization with other hosts, and ARC or BSV BroadcasterServices that broadcast transactions and return proofs that they were mined for broadcasting and proofs

Example on testnet

The example runs a local host on testnet with the tm_helloworld topic and the ls_helloworld lookup service. It keeps everything in memory, so it starts empty each time.

Write the host

Create overlay-host.mjs in your project folder.

The script generates a new host key each time it starts, which suits this local example. For a host you keep running, load the key from an environment variable and never paste it into your code.

The host script is:

overlay-host.mjs
import OverlayExpress from '@bsv/overlay-express'
import { HelloWorldTopicManager, createHelloWorldLookupService } from '@bsv/overlay-topics'
import { MongoMemoryServer } from 'mongodb-memory-server'
import { PrivateKey } from '@bsv/sdk'

const mongo = await MongoMemoryServer.create()
const server = new OverlayExpress(
  'my-overlay',
  PrivateKey.fromRandom().toHex(),
  'localhost:8080',
  process.env.ADMIN_TOKEN
)

server.configurePort(8080)
server.configureNetwork('test')
await server.configureKnex({
  client: 'better-sqlite3',
  connection: { filename: ':memory:' },
  useNullAsDefault: true
})
await server.configureMongo(mongo.getUri())

server.configureTopicManager('tm_helloworld', new HelloWorldTopicManager())
await server.configureLookupServiceWithMongo('ls_helloworld', db => createHelloWorldLookupService(db))

await server.configureEngine()
await server.start()

What the key lines do:

  • OverlayExpress takes a name, the key that signs the host’s advertisements, the address it announces without https://, and the optional admin token.
  • configureNetwork('test') runs the host on testnet. Without it the host runs on Mainnet.
  • better-sqlite3 keeps the SQL database in memory, and mongodb-memory-server starts a temporary MongoDB for development.

Start the host

Generate an admin token of at least 32 bytes and start the host with it. The command works in Git Bash or any Unix shell:

export ADMIN_TOKEN=$(node -p "require('crypto').randomBytes(24).toString('hex')")
node overlay-host.mjs

The host prints a long startup log. It is ready when the last line is:

my-overlay is ready and listening on local port 8080

The log also has a line that starts with Advertiser not initialized for FQDN localhost:8080. This is expected on localhost, where the host does not announce itself to other hosts. Next, the log shows Failed to GASP sync and No competent mainnet hosts found by the SLAP trackers, because a local host has no peers to synchronize with. Both messages are harmless, and the host makes no outside connection while idle on testnet.

Check the host

Open http://localhost:8080/health. The answer starts with {"status":"ok","live":true,"ready":true, and names the network, test.

Configuration options

The sections below cover the settings of the host, the order of the calls and the admin routes.

Settings and defaults

These settings control the network, the port, the admin access and the synchronization of the host:

OptionDefaultEffect
configureNetworkmainChooses main, test or ttn. With the default, the host contacts public Mainnet hosts at startup and synchronizes from them
configurePort3000The port the server listens on
Admin tokenA random UUIDThe fourth argument of the constructor, between 32 and 16,384 bytes. Read the generated one with server.getAdminToken()
configureEnableGASPSynctrueCopies data from other overlay hosts when the host starts
configureArcApiKey, configureArcadeNot setBroadcasting and proofs through ARC or BSV Broadcaster, which configureArcade configures. Without a provider the host disables the ingest route for their callbacks

Order of the calls and admin routes

Call the methods in this order:

  1. configureKnex and configureMongo
  2. configureTopicManager and configureLookupServiceWithMongo
  3. configureEngine
  4. start

Each step fails with its own message when an earlier one is missing, for example You must configure your SQL database with the .configureKnex() method first!. The host needs both databases, even when none of your own services uses MongoDB.

Every /admin route needs the admin token, except /admin/config, which answers without one and returns the adminIdentityKey and the nodeName of the host. Send the token as a bearer token, Authorization: Bearer YOUR_ADMIN_TOKEN, and replace YOUR_ADMIN_TOKEN with your token. The host answers like this:

RequestAnswer
/health, /listTopicManagers or /admin/config, with no token200
Any other /admin route, with no token401
Any other /admin route, with a wrong token403
Any other /admin route, with the right token200

Limitations

Check these two limits before you run an app on testnet:

  • No public testnet overlay host or tracker works, so a testnet app runs its own host, as in this example.
  • Other users cannot reach a host on localhost. To announce itself to other hosts through SHIPServices Host Interconnect Protocol and SLAPServices Lookup Availability Protocol, a host needs a public address instead of localhost.

To learn more about overlays on the BSV blockchain, review these articles: