Why is reliability important for running a Lightning node?

In Bitcoin, hardware is not particularly important unless one is specifically running a mining node. The Bitcoin Core node software can be run on any machine that meets its minimum requirements and does not need to be online to receive payments; only to send them. If a Bitcoin node goes down for an extended period of time, the user can simply reboot the node and once it connects to the rest of the network, it will re-sync the blockchain.

In Lightning, however, the user needs to be online both to send and to receive payments. If the Lightning node is offline, it cannot receive any payments from anyone and thus its open invoices cannot be fulfilled. Furthermore, the open channels of an offline node cannot be used to route payments. Your channel partners will notice that you are offline and cannot contact you to route a payment. If you are offline too often, they may consider the bitcoin locked up in their channels with you to be underutilized capacity, and may close those channels. We already discussed the case of a protocol attack where your channel partner tries to cheat you by submitting an earlier commitment transaction. If you are offline and your channels aren’t being monitored, then the attempted theft could succeed and you will have no recourse once the timelock expires. Hence, node reliability is extremely important for a Lightning node.

There are also the issues of hardware failure and loss of data. In Bitcoin, a hardware failure can be a trivial problem if the user has a backup of their mnemonic phrase or private keys. The Bitcoin wallet and the bitcoin inside the wallet can be easily restored from the private keys on a new computer. Most information can be re-downloaded from the blockchain.

In contrast, in Lightning the information about the user’s channels, including the commitment transactions and revocation secrets, are not publicly known and are only stored on the individual user’s hardware. Thus, software and hardware failures in the Lightning Network can easily result in loss of funds.

Types of hardware Lightning nodes

There are three main types of hardware Lightning nodes:

• General-purpose computers: A Lightning Network node can be run on a home computer or laptop running Windows, Mac OS, or Linux. Typically this is run alongside a Bitcoin node.

• Dedicated hardware: A Lightning node can also be run on dedicated hardware like a Raspberry Pi, Rock64, or mini-PC. This setup would usually run a software stack including a Bitcoin node and other applications. This setup is popular as the hardware is dedicated to running and maintaining the Lightning node only and is usually set up with an installation “helper”.

• Pre-configured hardware: A Lightning Network node can also be run on purpose-built hardware specifically selected and configured for it. This would include “out-of-the-box” Lightning node solutions that can be purchased as a kit or a turn-key system.

Running in the “cloud”

Virtual Private Server (VPS) and “cloud computing” services such as Microsoft Azure, Google Cloud, Amazon Web Services (AWS), or DigitalOcean are quite affordable and can be set up very quickly. A Lightning node can be hosted for between $20 and $40 per month on such a service.

However, as the saying goes, "Cloud is just other people’s computers”. Using these services means running your node on other people’s computers. This brings along the corresponding advantages and disadvantages. The key advantages are convenience, efficiency, uptime, and possibly even cost. The cloud operator manages and runs the node to a high degree automatically providing you with convenience and efficiency. They provide excellent uptime and availability, often much better than what an individual can achieve at home. If you consider that just the electricity cost of running a server in many western countries is around $10 per month, then add to that the cost of network bandwidth and the hardware itself, the VPS offering becomes financially competitive. Lastly, with a VPS you need no space for a PC at home, and don’t have any issues with PC noise or heat. On the other hand there are several notable disadvantages. A Lightning node running in the “cloud” will always be less secure and less private than one running on your own computer. Additionally, these cloud computing services are very centralized. The vast majority of Bitcoin and Lightning nodes running on such services are located in a handful of data centers in Virginia, Sunnyvale, Seattle, London, and Frankfurt. When the networks or data centers of these providers have service problems, it affects thousands of nodes on so-called “decentralized” networks.

If you have the possibility and capacity of running a node on your own computer at home or in your office, then this might be preferable to running it in the cloud. Nonetheless, if running your own server is not an option, by all means consider running one on a VPS.

Running a node at home

If you have a reasonable capacity Internet connection at home or in your office, you can certainly run a Lightning node there. Any “broadband” connection is sufficient for the purpose of running a lightweight node, and a fast connection will allow you to run a Bitcoin full node too.

While you can run a Lightning node (and even a Bitcoin node) on your laptop, it will become annoying quite fast. These programs consume your computer’s resources and need to run 24/7. Your user applications like your browser or your spreadsheet will find themselves competing against the Lightning background services for your computer’s resources. In other words, your browser and other desktop workloads will be slowed down. And when your word-processing app freezes up your laptop, your Lightning node will go down as well leaving you unable to receive transactions and potentially vulnerable to attacks. Furthermore, you should never turn off your laptop. All this combined together results in a set-up that is not ideal. The same will apply to your daily-use personal desktop PC.

Instead, most users will choose to run a node on a dedicated computer. Fortunately, you don’t need a “server” class computer to do this. You can run a Lightning node on a single-board computer, such as a Raspberry Pi or on a Mini PC (usually marketed as home theater PC’s). These are simple computers which are commonly used as a home automation hub or a media server. They are relatively inexpensive, when compared to a PC or a laptop. The advantage of a dedicated device as a platform for Lightning and Bitcoin nodes is that it can run continuously, silently, and unobtrusively on your home network, tucked behind your router or TV. No one will even know that this little box is actually part of a global banking system!

What hardware is required to run a Lightning node?

At a minimum, the following will be required to run a Lightning node:

• CPU: Sufficient processing power will be required to run a Bitcoin node, which will continuously download and validate new blocks. The user also needs to consider the Initial Block Download (IBD) when setting up a new Bitcoin node, which can take anywhere from several hours to several days. A 2-core or 4-core CPU is recommended.

• RAM: A system with 2GB of RAM will barely run both Bitcoin and Lightning nodes. It will perform much better with at least 4GB of RAM. The Initial Block Download will be especially challenging with less than 4GB of RAM. More than 8GB of RAM is unnecessary, because the CPU is the greater bottleneck for these types of services, due to cryptographic operations such as signature validation.

• Storage drive: This can be a Hard Disk Drive (HDD) or a Solid State Drive (SSD). An SSD will be significantly quicker (but more expensive) for running a node. Most of the storage is used for the Bitcoin blockchain, which is hundreds of gigabytes in size. A fair tradeoff (cost for complexity) is to buy a small SSD to boot the OS from and a larger HDD to store large data objects (mostly databases).

• Internet connection: A reliable Internet connection will be required to download new Bitcoin blocks, as well as to communicate with other Lightning peers. During operation the estimated data use ranges from 10GB to 100GB per month, depending on configuration. At startup, a Bitcoin full node downloads the full blockchain.

• Power supply: A reliable power supply is required as Lightning nodes need to be online at all times. A power failure will cause in-progress payments to fail. For heavy duty routing nodes, a backup or uninterruptible power supply (UPS) is useful in the event of power outages. Ideally, you should connect your Internet router to this UPS as well.

• Backup: Backup is crucial as a failure can result in loss of data and hence in loss of funds. The user will want to consider some kind of data backup solution. This could be a cloud-based automated backup to a server or web service the user controls. Alternatively, it could be an automated local hardware backup, such as a second hard drive. For the best results, both local and remote backup can be combined.

Switching server configuration in the cloud

When renting a cloud server, it is often cost effective to change the configuration between two phases of operation. A faster CPU and faster storage will be needed during the Initial Block Download (e.g. the first day). After the blockchain has synced, the CPU and storage speed requirements are much less, so the performance can be downgraded to a more cost-effective level.

For example, on Amazon’s cloud, we would use a 8-16GB RAM, 8-core CPU (e.g. t3-large or m3.large) and faster 400GB SSD (1000+ provisioned IOPS) for the Initial Block Download, reducing its time to just 6-8 hours. Once that is complete, we would switch the server instance to a 2GB RAM, 2-core CPU (e.g. t3.small) and storage to a general purpose 1TB HDD. This will cost about the same as if you ran it on the slower server the entire time, but it will get you up and running in less than a day instead of having to wait almost a week for the IBD.

RaspiBlitz

One of the most popular and complete “helpers” is RaspiBlitz, a project built by Christian Rotzoll. It is intended to be installed on a Raspberry Pi 4. RaspiBlitz comes with a recommended hardware “kit” that you can build in a matter of hours or at most a weekend. If you attend a Lightning “hackathon” in your city, you are likely to see many people working on their RaspiBlitz setup, swapping tips, and helping each other. You can find the RaspiBlitz project here:

https://github.com/rootzoll/raspiblitz

In addition to a Bitcoin and Lightning node, RaspiBlitz can install a number of additional services, such as:

• TOR (run as hidden service)

• ElectRS (Electrum server in Rust)

• BTCPayServer (cryptocurrency payment processor)

• BTC-RPC-Explorer (Bitcoin blockchain explorer)

• Ride The Lightning (Lightning node management GUI)

• LNbits (Lightning wallet/accounts system)

• SpecterDesktop (multisig Trezor, Ledger, Coldcard wallet & Specter-DIY)

• LNDmanage (command-line interface for advanced channel management)

• Loop (submarine swaps service)

• JoinMarket (CoinJoin service)

MyNode

MyNode is another popular open source “helper” project including a lot of Bitcoin related software. It is easy to install: you “flash” the installer onto an SD card and boot your mini-PC from the SD card. You do not need any monitor to use myNode as the administrative tools are accessible remotely from a browser. If your mini-PC has no monitor, mouse, or keyboard; you can manage it from another computer or even from your smartphone. Once installed, go to http://mynode.local/ and create a Lightning wallet and node in two clicks.

You can find the MyNode project here:

https://mynodebtc.com/

In addition to a Bitcoin and Lightning node, MyNode can optionally install a variety of additional services, such as:

• Ride The Lightning (Lightning node management GUI)

• OpenVPN (VPN Support for remote management or wallet)

• LNDmanage (command-line interface for advanced channel management)

• btc-rpc-explorer (A Bitcoin blockchain explorer)

Umbrel

Famous for their UX/UI, Umbrel provides a very easy and accessible way to get your Bitcoin and Lightning node up and running in no time, especially for beginners. A very distinctive feature is that Umbrel utilizes Neutrino/SPV during the Initial Blockchain Download (IBD) so you can instantly start using your node. Once Bitcoin Core is fully synced in the background it automatically switches over and disables SPV mode. Umbrel OS supports the Raspberry Pi 4 and can also be installed on any Linux-based OS or on a virtual machine on macOS or Windows. You can also connect any wallet that supports Bitcoin Core P2P, Bitcoin Core RPC, the Electrum protocol or lndconnect.

There’s no need to wait for a rainy day - you can find the project here: https://getumbrel.com

In addition to a Bitcoin and Lightning node, Umbrel introduced the Umbrel App Store, where you can easily install additional services, such as:

• Lightning Terminal (interface for managing channel liquidity, loop-in & loop-out)

• Ride The Lightning (Lightning node management GUI)

• Specter Desktop (watch-only coordinator for multi-signature and single-key Bitcoin wallets)

• BTCPayServer (cryptocurrency payment processor)

• BTC-RPC-Explorer (Bitcoin blockchain explorer)

• ThunderHub (monitor and manage your node)

• Sphinx Relay (handling connectivity and storage for Sphinx chat)

• mempool.space (mempool visualizer and block explorer)

• LNbits (Lightning wallet/accounts system)

Umbrel is currently still in beta and not considered secure.

BTCPay Server

While not initially designed as an installation “helper”, the e-commerce and payment platform BTCPay Server has an incredibly easy installation system that uses Docker containers and docker-compose to install a Bitcoin node, Lightning node, and payment gateway, among many other services. It can be installed on a variety of hardware platforms, from a simple Raspberry Pi 4 (4GB recommended) to a mini-PC, old laptop, desktop or server.

BTCPay Server is a fully-featured self-hosted self-custody e-commerce platform that can be integrated with many e-commerce platforms such as Wordpress Woocommerce and others. The installation of the full node is only a step of the e-commerce platform installation. While originally developed as a feature-for-feature replacement of the Bitpay commercial payment service and API, it has evolved past that to be a complete platform for BTC and Lighting services related to e-commerce. For many sellers or shops it is a one-shop turn-key solution to e-commerce.

More information can be found at:

https://btcpayserver.org/

In addition to a Bitcoin and Lightning node, BTCPay Server can also install a variety of services, including:

• c-lightning or LND Lightning node

• Litecoin support

• Monero support

• Spark server (c-lightning web wallet)

• Charge server (c-lightning e-commerce API)

• Ride The Lightning (Lightning node management web GUI)

• Many BTC forks

• BTCTransmuter (event-action automation service supporting currency exchange)

The number of additional services and features is growing rapidly, so the list above is only a small subset of what is available on the BTCPay Server platform.

Bitcoin node or lightweight Lightning

One critical choice for your setup will be the choice of the Bitcoin node and its configuration. Bitcoin Core, the reference implementation, is the most common choice but not the only choice available. One alternative choice is btcd, which is a Go-language implementation of a Bitcoin node. Btcd supports some features that are useful for running an LND Lightning node and are not available in Bitcoin Core.

A second consideration is whether you will run an archival Bitcoin node with a full copy of the blockchain (some 350GB in mid-2021) or a pruned blockchain that only keeps the most recent blocks. A pruned blockchain can save you some disk space, but you will still need to download the full blockchain at least once (during the Initial Block Download). Hence it won’t save you any network traffic. Using a pruned node to run a Lightning node is still an experimental capability and might not support all the functionality. However, many people are running a node like that successfully.

Finally, you also have the option of not running a Bitcoin node at all. Instead you can operate the LND Lightning node in “lightweight” mode, using the neutrino protocol to retrieve blockchain information from public Bitcoin nodes operated by others. Running like this means that you are taking resources from the Bitcoin network without offering any in return. Instead, you are offering your resources and contributing to the Lightning Network community. For smaller Lightning nodes this will generally reduce network traffic in comparison to running a full Bitcoin node.

Keep in mind that operating a Bitcoin node allows you to support other services, besides and on top of a Lightning node. These other services may require an archival (not pruned) Bitcoin node and often can’t run without a Bitcoin node. Consider upfront what other services you may want to run now or in the future to make an informed decision on the type of Bitcoin node you select.

The bottom line for this decision is: If you can afford a disk larger than 500GB, run a full archival Bitcoin node. You will be contributing resources to the Bitcoin system and helping others who cannot afford to do so. If you can’t afford such a big disk, run a pruned node. If you can’t afford the disk or the bandwidth for even a pruned node, run a lightweight LND node over neutrino.

Operating system choice

The next step is to select an operating system for your node. The vast majority of Internet servers run on some variant of Linux. Linux is the platform of choice for the Internet because it is a powerful, open-source operating system. Linux, however, has a steep learning curve and requires familiarity with a command-line environment. It is often intimidating for new users.

Ultimately, most of the services can be run on any modern POSIX operating system, which includes Mac OS, Windows, and of course Linux. Your choice should be driven more by your familiarity and comfort with an operating system and your learning objectives. If you want to expand your knowledge and learn how to operate a Linux system, this is a great opportunity to do so with a specific project and a clear goal. If you just want to get a node up and running, go with what you know.

Nowadays, many services are also delivered in the form of containers, usually based on the Docker system. These containers can be deployed on a variety of operating systems, abstracting the underlying OS. You may need to learn some Linux CLI commands nonetheless, as most of the containers run some variant of Linux inside.

Background services

For those accustomed to running applications on their desktop or smartphone, an application always has a graphical user interface even if it may sometimes run in the background. The Bitcoin and Lightning node applications, however, are very different. These applications do not have a graphical user interface built in. Instead, they run as headless background services, meaning they are always operating in the background and do not interact with the user directly.

This can create some confusion for users who are not used to running background services. How do you know if such a service is currently running? How do you start and stop it? How do you interact with it? The answers to these questions depend on the operating system you are using. For now we will assume you are using some Linux variant and answer them in that context.

Process isolation

Background services usually run under a specific user account in order to isolate them from the operating system and each other. For example, Bitcoin Core is configured to run as user bitcoin. You will need to use the command-line to create a user for each of the services you run.

In addition, if you have connected an external drive, you will need to tell the operating system to relocate the user’s home directory to that drive. That’s because a service like Bitcoin Core will create files under the user’s home directory. If you are setting it up to download the full Bitcoin blockchain, these files will take up several hundred Gigabytes. Here, we assume you have connected the external drive and it is located on the /external_drive/ path of the operating system.

On most Linux systems you can create a new user with the useradd command, like this:

$ sudo useradd -m -d /external_drive/bitcoin -s /dev/null bitcoin

The m and d flags create the user’s home directory as specified by /external_drive/bitcoin in this case. The s flag assigns the user’s interactive shell. In this case, we set it to /dev/null to disable interactive shell use. The last argument is the new user’s username bitcoin.

Node startup

For both Bitcoin and Lightning node services, “installation” also involves creating a so called startup script to make sure that the node starts when the computer boots. Startup and shutdown of background services is handled by an operating system process, which in Linux is called init or systemd. You can usually find a system startup script in the contrib sub-directory of each project. For example, if you are on a modern Linux OS that uses systemd, you would find a script called bitcoind.service, that can start and stop the Bitcoin Core node service.

Here’s an example of what a Bitcoin node’s startup script looks like, taken from the Bitcoin Core code repository:

[Unit]
Description=Bitcoin daemon
After=network.target

[Service]
ExecStart=/usr/bin/bitcoind -daemon 
                            -pid=/run/bitcoind/bitcoind.pid 
                            -conf=/etc/bitcoin/bitcoin.conf 
                            -datadir=/var/lib/bitcoind

# Make sure the config directory is readable by the service user
PermissionsStartOnly=true
ExecStartPre=/bin/chgrp bitcoin /etc/bitcoin

# Process management
####################

Type=forking
PIDFile=/run/bitcoind/bitcoind.pid
Restart=on-failure
TimeoutStopSec=600

# Directory creation and permissions
####################################

# Run as bitcoin:bitcoin
User=bitcoin
Group=bitcoin

# /run/bitcoind
RuntimeDirectory=bitcoind
RuntimeDirectoryMode=0710

# /etc/bitcoin
ConfigurationDirectory=bitcoin
ConfigurationDirectoryMode=0710

# /var/lib/bitcoind
StateDirectory=bitcoind
StateDirectoryMode=0710

[...]

[Install]
WantedBy=multi-user.target

As the root user, install the script by copying it into the systemd service folder /lib/systemd/system/ and then reload systemd:

$ sudo systemctl daemon-reload

Next, enable the service:

$ sudo systemctl enable bitcoind

You can now start and stop the service. Don’t start it yet, as we haven’t configured the Bitcoin node.

$ sudo systemctl start bitcoind
$ sudo systemctl stop bitcoind

Node configuration

To configure your node, you need to create and reference a configuration file. By convention, this file is usually created in /etc, under a directory with the name of the program. For example, Bitcoin Core and LND configurations would usually be stored in:

/etc/bitcoin/bitcoin.conf /etc/lnd/lnd.conf

These configuration files are text files with each line expressing one configuration option and its value. Default values are assumed for anything not defined in the configuration file. You can see what options can be set in the configuration in two ways. First, running the node application with a help argument will show the options that can be defined on the command-line. These same options can be defined in the configuration file. Second, you can usually find an example configuration file, with all the default options, in the code repository of the software.

You can find one example of a configuration file in each of the Docker images we used in Chapter 4. For example, the file code/docker/bitcoind/bitcoind/bitcoin.conf:

regtest=1
server=1
debuglogfile=debug.log
debug=1
txindex=1
printtoconsole=0

[regtest]
fallbackfee=0.000001
port=18444
noconnect=1
dnsseed=0
dns=0
upnp=0
onlynet=ipv4
rpcport=18443
rpcbind=0.0.0.0
rpcallowip=0.0.0.0/0
rpcuser=regtest
rpcpassword=regtest
zmqpubrawblock=tcp://0.0.0.0:12005
zmqpubrawtx=tcp://0.0.0.0:12006

That particular configuration file configures Bitcoin Core for operation as a regtest node and provides a weak username and password for remote access, so you shouldn’t use it for your node configuration. However, it serves to illustrate the syntax of a configuration file and you can make adjustments to it in the docker container to experiment with different options. See if you can use the bitcoind -help command to understand what each of the options does in the context of the docker network we build in Chapter 4.

Often, the defaults suffice, and with a few modifications your node software can be configured quickly. To get a Bitcoin Core node running with minimal customization, you only need four lines of configuration:

server=1
daemon=1
txindex=1
rpcuser=USERNAME
rpcpassword=PASSWORD

Even the txindex option is not strictly necessary, though it will ensure your Bitcoin node creates an index of all transactions, which is required for some applications. The txindex option is not required to run a Lightning node.

A c-lightning Lightning node running on the same server also requires only a few lines in the configuration:

network=mainnet
bitcoin-rpcuser=USERNAME
bitcoin-rpcpassword=PASSWORD

In general, it is a good idea to minimize the amount of customization of these systems. The default configuration is carefully designed to support the most common deployments. If you modify a default value, it may cause problems later on or reduce the performance of your node. In short, modify only when necessary!

Network configuration

Network configuration is normally not an issue when configuring a new application. However, peer-to-peer networks like Bitcoin and the Lightning network present some unique challenges for network configuration.

In a centralized service, your computer connects to the “big servers” of some corporation, and not vice-versa. Your home Internet connection is actually configured on the assumption that you are simply a consumer of services provided by others. But in a peer-to-peer system, every peer both consumes from and provides services to other nodes. If you’re running a Bitcoin or Lightning node at your home, you’re providing a service to other computers on the Internet. Your Internet service by default is not configured to allow you to run servers and may need some additional configuration to enable others to reach your node.

If you want to run a Bitcoin or Lightning node, you need to make it possible for other nodes on the Internet to connect to you. That means enabling incoming TCP connections to the Bitcoin port (port 8333 by default) or Lightning port (port 9735 by default). While you can run a Bitcoin node without incoming connectivity, you can’t do that with a Lightning node. A Lightning node must be accessible to others from outside your network.

By default, your home Internet router does not expect incoming connections from the outside, and in fact incoming connections are blocked. Your Internet router IP address is the only externally accessible IP address, and all the computers you run inside your home network share that single IP address. This is achieved by a mechanism called Network Address Translation (NAT) which allows your Internet router to act as an intermediary for all outbound connections. If you want to allow an inbound connection you have to set up Port Forwarding, which tells your Internet router that incoming connections on specific ports should be forwarded to specific computers inside the network. You can do this manually by changing your Internet router configuration or, if your router supports it, through an automatic port forwarding mechanism called Universal Plug and Play (UPNP).

An alternative mechanism to port forwarding is to enable The Onion Router (TOR), which provides a kind of virtual private network overlay that allows incoming connections to an onion address. If you run TOR, you don’t need to do port forwarding nor enable incoming connections to Bitcoin or Lightning ports. If you run your nodes using TOR, all traffic goes through TOR and no other ports are used.

Let’s look at different ways you can make it possible for others to connect to your node. We’ll look at these alternatives in order, from easiest to most difficult.

It just works!

There is a possibility that your Internet service provider or router is configured to support UPNP by default and everything just works automatically. Let’s try this approach first, just in case we are lucky.

Assuming you already have a Bitcoin or Lightning node running, we will try and see if they are accessible from the outside.

Note

For this test to work, you have to have either a Bitcoin or Lightning node (or both) up and running on your home network. If your router supports UPNP, the incoming traffic will automatically be forwarded to the corresponding ports on the computer running the node.

You can use some very popular and useful websites to find out what is your external IP address and whether it allows and forwards incoming connections to a known port. Here are two that are reliable:

https://canyouseeme.org/

https://www.whatismyip.com/port-scanner/

By default, these services only allow you to check incoming connections to the IP address from which you are connecting. This is done to prevent you from using the service to scan other people’s networks and computers. You will see your router’s external IP address and a field for entering a port number. If you haven’t changed the default ports in your node configuration, try port 8333 (Bitcoin) and/or 9735 (Lightning).

Figure 5-1. Checking for incoming port 9735

In Figure 5-1 you can see the result of checking port 9735 on a server running Lightning, using the whatismyip.org port scanner tool. It shows that the server is accepting incoming connections to the Lightning port. If you see a result like this, you are all set!

sudo apt install tor

[Tor]
tor.active=true
tor.v3=true
tor.streamisolation=true
listen=localhost

curl --socks5 localhost:9050 --socks5-hostname localhost:9050 -s https://check.torproject.org/ | cat | grep -m 1 Congratulations | xargs

Operating system security

Securing an operating system is a vast topic that is beyond the scope of this book. However, we can establish some basic principles.

To secure your operating system, here are some of the top items to consider:

1. Provenance: Start by ensuring that you are downloading the correct operating system image and verify any signatures or checksums before installing it. Extend this to any software that you install. Double-check any source or URL from where you download. Verify the integrity and correctness of the downloaded software via signature and checksum verification.

2. Maintenance: Make sure that you keep your operating system up to date. Enable automated daily or weekly installation of security updates.

3. Least privilege: Set up users for specific processes and give them the least access needed to run a service. Do not run processes with admin privileges (e.g. root).

4. Process isolation: Use the operating system features to isolate processes from each other.

5. File system permissions: Configure the file system carefully, on the least-privilege principle. Do not make files readable or writable by everyone.

6. Strong authentication: Use strong randomly generated passwords or, whenever possible, public-key authentication. E.g. it is safer to use Secure Shell (SSH) with a cryptographic key pair instead of a password.

7. Two-factor authentication (2FA): Use two-factor authentication wherever possible, including Universal 2-Factor (U2F) with hardware security keys. This applies to all external services you might be using such as your cloud service provider. You can apply this also to your own set-up such as your own SSH configuration. Use 2FA also for indirect services. For exmaple, say you are using a cloud service. You gave your cloud service provider an email address, so you should also protect your email address with 2FA.

8. Backup: Make backups of your system, and make sure you protect the backups with encryption too. Perform these backups periodically. At least once test if you can restore your backup and that your backup is complete and accessible. If possible, keep one copy of your backups on a different disk to avoid that a single hard disk failure destroys both your active node as well as your backup copies.

9. Vulnerability and exposure management: Use remote scanning to ensure you have minimized the attack surface of your system. Close any unnecessary services or ports.

   • Minimize: Install only software and packages that you really need and use. Uninstall packages that you no longer use. It is recommended that you do not use your node computer for non-node activities that you can perform on another of your computers. Especially, if you can, do not use your node computer for browsing, surfing the Internet, or reading your email.

This is a list of the most basic security measures. It is by no means exhaustive.

Node access

Your Lightning node will expose a Remote Procedure Call (RPC) Application Programming Interface (API). This means that your node can be controlled remotely by commands sent to a specific TCP port. Access control to that RPC API is achieved by some form of user authentication. Depending on the type of Lightning node you set up, this will either be done by username/password authentication or by a mechanism called an authentication macaroon. As the name implies, a macaroon is a more sophisticated type of cookie. Unlike a cookie, it is cryptographically signed and can express a set of access capabilities.

For example, LND uses macaroons to grant access to the RPC API. By default, the LND software creates three macaroons with different levels of access, called admin, invoice, and readonly. Depending on which macaroon you copy and use in your RPC client, you either have “readonly” access, “invoice” access (which includes the “readonly” capabilities), or “admin” access which gives you full control. There is also a macaroon “bakery” function in LND that can construct macaroons with any combination of capabilities with very fine-grained control.

If you use a username/password authentication model, make sure you select a long and random password. You will not have to type this password often as it will be stored in the configuration files. You should therefore pick one that cannot be guessed. Many of the examples you will see include poorly chosen passwords, and often people copy these into their own systems, providing easy access to anyone. Don’t do that! Use a password manager to generate a long random alpha-numeric password. Since certain special characters such as $?/!*&%‘"’ can interfere with the command line, it is best to avoid these for passwords that will be used in a shell environment. To avoid problems stick with long random alpha-numeric passwords.

A plain alphanumeric sequence that is longer than 12 characters and randomly generated is usually sufficient. If you plan to store large amounts of money on your Lightning node and are concerned about remote brute-force attacks, select a password length of more than 20 characters to make such attacks practically infeasible.

Static Channel Backups (SCB)

At the time of writing this book, only LND offers a built-in mechanism for static channel backups. Eclair has no backup on the server side, although Eclair mobile does offer optional backup to a Google Drive. C-lightning recently merged the necessary interfaces for a plugin to implement channel backups. Unfortunately, there is no consistent, agreed upon backup mechanism across different node implementations.

File-based backups of the Lightning node databases are at best a partial solution because you run the risk of backing up an inconsistent database state. In addition, you may not reliably catch the latest state commitments. It is much better to have a backup mechanism that is triggered every time there is a state change in a channel, thereby ensuring data consistency.

To set up static channel backups in LND, set the backupfilepath parameter either on the command-line or in the configuration file. LND will then save an SCB file in that directory path. Of course, that’s only the first step of the solution. Now, you have to setup a mechanism that monitors this file for changes. Each time the file changes the backup mechanism must copy this file to another, preferably “off-site” disk. Such backup mechanisms are beyond the scope of this book. Nonetheless, any sophisticated backup solution should be able to handle this scenario. Recall, the backup files should be encrypted too.

Hot wallet risk

As we’ve discussed previously, the Lightning Network consists of a network of hot wallets. The funds you store in a Lightning wallet are online all the time. This makes them vulnerable. Hence, you should not store large amounts in a Lightning wallet. Large amounts should be kept in a cold wallet that is not online and which can transact only on-chain.

Even if you start small, as time passes you may still find you have a significant amount of money in a Lightning wallet. This is a typical scenario for store owners. If you use a Lightning node for an e-commerce operation, your wallet will likely receive funds often, but send funds rarely. You will therefore end up having two problems simultaneously: First, your channels will be imbalanced with large local balances outweighing small remote balances. Secondly, you will have too much money in the wallet. Fortunately, you can also solve both of these problems simultaneously.

Let’s look at some of the solutions you can use to reduce the funds exposed in a hot wallet.

Sweeping funds

If your Lightning wallet balance becomes too large for your risk tolerance, you will need to “sweep” funds out of the wallet. You can do so in three ways: on-chain, off-chain, and loop-out. Let’s look at each one of these options in the next few sections.

loop out --amt 501000 --conf_target 400
Max swap fees for 501000 sat Loop Out: 25716 sat
Regular swap speed requested, it might take up to 30m0s for the swap to be executed.
CONTINUE SWAP? (y/n), expand fee detail (x): x

Estimated on-chain sweep fee:        149 sat
Max on-chain sweep fee:              14900 sat
Max off-chain swap routing fee:      10030 sat
Max no show penalty (prepay):        1337 sat
Max off-chain prepay routing fee:    36 sat
Max swap fee:                        750 sat
CONTINUE SWAP? (y/n): y
Swap initiated

Run `loop monitor` to monitor progress.

Tolerate faults and automate

If you have the time and skills you should test some basic fault scenarios on the Lightning testnet. On the testnet you will learn valuable lessons without risking any funds. Any step you perform to automate your system will improve your availability.

• Automatic computer server restart: What happens when your server or the operating system crashes? What happens when there is a power outage? Simulate this fault by pressing the “reset” button on your PC or by unplugging the power cable. After crash, reset, or power failure the computer should automatically restart itself. Some computers have a setting in their BIOS to specify how the computer should react on power failures. Test it to make sure the computer really restarts automatically on power failure without human intervention.

• Automatic node restart: What happens when your node or one of your nodes crashes? Simulate this fault by killing the corresponding node processes. If a node crashes, it should automatically restart itself. Test it to make sure the node or nodes really restart automatically on failure without human intervention. If this is not the case, most likely your node is not set up correctly as an operating system service.

• Automatic network reconnection: What happens if your network goes down? What happens when your ISP goes temporarily down? What happens when your ISP assigns a new IP address to your router or your computer? When the network comes back, do the nodes your are running automatically reconnect to the network? Simulate this fault by unplugging and later re-plugging the Ethernet cable from the device hosting your nodes. The nodes should automatically reconnect and continue operation without human intervention.

• Configure your log files: All of the above failures should leave textual entries behind in the corresponding log files. Turn up the verbosity of logging if needed. Find these error entries in the log files and use them for monitoring.

Monitoring node availability

Monitoring your node is an important part of keeping it running. You need to monitor not only the availability of the computer itself, but also the availability and correct operation of the Lightning node software.

There are a number of ways to do this, but most require some customization. You can use generic infrastructure monitoring or application monitoring tools, but you have to customize them specifically to query the Lightning node API to ensure the node is running, synchronized to the blockchain, and connected to channel peers.

There is a specialized service that offers Lightning node monitoring. It uses a Telegram bot to notify you of any interruptions in service. This is a free service, though you can pay (over Lightning of course) to get faster alerts. Find more information at:

https://lightning.watch

Over time, we expect more third-party services to provide specialized Lightning node monitoring payable via micro-payments. Perhaps such services and their APIs will become standardized and will one day be directly supported by Lightning node software.

Watchtowers

Watchtowers are a mechanism for outsourcing the monitoring and penalty resolution of Lightning protocol violations.

As we mentioned in previous chapters, the Lightning protocol maintains security through a penalty mechanism. If one of your channel partners broadcasts an old commitment transaction, your node will need to exercise the revocation clause and broadcast a penalty transaction to avoid losing money. But if your node is down during the protocol violation, you might lose money.

To solve this problem, we can use one or more watchtowers to outsource the job of monitoring protocol violations and issuing penalty transactions. There are two parts to a watchtower setup: a watchtower server (or simply “watchtower”) that monitors the blockchain, and a watchtower client that asks the watchtower server for this monitoring service.

Watchtower technology is still in the early stages of development and is not widely supported. However, below we list some experimental implementations that you can try.

LND software includes both a watchtower server and a watchtower client. You can activate the watchtower server by adding the following configuration options:

[watchtower]
watchtower.active=1
watchtower.towerdir=/path_to_watchtower_data_directory

You can use LND’s watchtower client by activating it in the configuration and then using the command-line to connect it to a watchtower server. The configuration is:

[wtclient]
wtclient.active=1

LND’s command-line client lncli shows the following options for managing the watchtower client:

$ lncli wtclient

NAME:
   lncli wtclient - Interact with the watchtower client.

USAGE:
   lncli wtclient command [command options] [arguments...]

COMMANDS:
     add     Register a watchtower to use for future sessions/backups.
     remove  Remove a watchtower to prevent its use for future sessions/backups.
     towers  Display information about all registered watchtowers.
     tower   Display information about a specific registered watchtower.
     stats   Display the session stats of the watchtower client.
     policy  Display the active watchtower client policy configuration.

OPTIONS:
   --help, -h  show help

C-lightning has the API hooks necessary for a watchtower client plugin, though no such plugin has been implemented yet.

Finally, a popular standalone watchtower server is The Eye of Satoshi (TEOS). It can be found here:

https://github.com/talaia-labs/python-teos

Opening outbound channels

As soon as you get your Lightning node up and running, you can fund its Bitcoin wallet and then start opening channels with those funds.

You must choose channel partners carefully as your node’s ability to send payments depends on who your channel partners are and how well connected they are to the rest of the Lightning Network. You also want to have more than one channel to avoid being susceptible to a single point of failure. Since Lightning now supports multi-path payments, you can split your initial funds into several channels and route bigger payments by combining their capacity. At the same time, avoid making your channels too small. Since you need to pay Bitcoin transaction fees to open and close a channel, the channel balance should not be so small that the on-chain fees consume a significant portion. It’s all about balance!

To summarize:

• Connect to a few well connected nodes.

• Open more than one channel.

• Don’t open too many channels.

• Don’t make the channels too small.

One way to find well connected nodes is to open a channel to a popular merchant selling products on the Lightning Network. These nodes tend to be well funded and well connected. So, when you are ready to buy something online via Lightning, you can open a channel directly to the merchant’s node. The merchant’s node ID will be in the invoice you will receive when you try to buy something. That makes it easy.

Another way to find well-connected nodes is to use a Lightning Explorer (see “Lightning Explorer”) such as 1ml.com and browse the list of nodes sorted by channel capacity and number of channels. Don’t go for the biggest nodes as that encourages centralization. Go for a node in the middle of the list so that you can help them grow. Another factor to consider might be the time span a node has been in operation. Nodes established for more than a year are likely to be more reputable and less risky than nodes that started operation a week ago.

[lnd-autopilot]
autopilot.active=1
autopilot.maxchannels=40
autopilot.allocation=0.70
autopilot.minchansize=500000
autopilot.maxchansize=5000000
autopilot.heuristic=top_centrality:1.0

[c-lightning-autopilot]
autopilot-percent=75
autopilot-num-channels=10
autopilot-min-channel-size-msat=100000000msat

Getting inbound liquidity

In the current design of the Lightning Network, it is more typical for users to obtain outbound liquidity before obtaining inbound liquidity. They will do so by opening a channel with another node, and more often they’ll be able to spend bitcoin before they can receive it. There are three typical ways of getting inbound liquidity:

• Open a channel with outbound liquidity and then spend some of those funds. Now the balance is on the other end of the channel, which means that you can receive payments.

• Ask someone to open a channel to your node. Offer to reciprocate, so that both of your nodes become better connected and balanced.

• Use a submarine swap (e.g. Loop-In) to exchange on-chain BTC for an inbound channel to your node.

• Pay a third party service to open a channel with you. Several such services exist. Some charge a fee to provide liquidity, some are free.

Here is a list of currently available liquidity providers who will open a channel to your node for a fee:

• Bitrefill’s Thor service at https://www.bitrefill.com/thor-lightning-network-channels/

• Lightning To Me at https://lightningto.me/

• LNBig at https://lnbig.com/

• Lightning Conductor at https://lightningconductor.net/channels

Creating inbound liquidity is challenging from both a practical and user experience perspective. Inbound liquidity does not happen automatically, so you have to find ways to build it for your node. This asymmetry of payment channels is also not intuitive. In most other payment systems you get paid first (inbound) before you pay others (outbound).

The challenge of creating inbound liquidity is most noticeable if you are a merchant or sell your services for Lightning payments. In that case, you need to be vigilant to ensure that you have enough inbound liquidity to be able to continue to receive payments. What if there is a surge of buyers on your store, but they can’t actually pay you because there is no more inbound capacity?

In the future these challenges can be partially mitigated by the implementation of dual-funded channels, which are funded from both sides and offer balanced inbound and outbound capacity. The burden could also be mitigated by more sophisticated autopilot software, which could request and pay for inbound capacity as needed.

Ultimately, Lightning users need to be strategic and proactive about channel management to ensure that sufficient inbound capacity is available to meet their needs.

Closing channels

As discussed earlier in the book, a mutual close is the preferred way of closing a channel. However, there are instances where a force close is necessary.

Some examples:

• Your channel partner is offline and cannot be contacted to initiate a mutual close.

• Your channel partner is online, but is not responding to requests to initiate a mutual close.

• Your channel partner is online and your nodes are negotiating a mutual close, but they become stuck and cannot reach a resolution.

Re-balancing channels

In the course of transacting and routing payments on Lightning, the combination of inbound and outbound capacities can become unbalanced.

For example, if one of your channel partners is frequently routing payments through your node, you will exhaust the inbound capacity on that channel, while also exhausting the outbound capacity on the outgoing channels. Once that happens, you can no longer route payments through that route.

There are many ways to re-balance channels, each with different advantages and disadvantages. One way is to use a submarine swap (e.g. Loop-Out) as described previously in this chapter. Another way to re-balance is to simply wait for routed payments that flow in the opposite direction. If your node is well connected, when a specific route becomes exhausted in one direction, the same route becomes available in the opposite direction. Other nodes may “discover” that route in the opposite direction and start using it as part of their payment path, thereby re-balancing the funds again.

A third way to re-balance channels is to purposefully create a circular route that sends a payment from your node back to your node, via the Lightning Network. By sending a payment out on a channel with large local capacity and arranging the path so that it returns to your node on a channel with large remote capacity, both of those channels will become more balanced. An example of a circular route re-balancing strategy can be seen in Figure 5-2.

Figure 5-2. Circular route re-balancing

Circular re-balancing is supported by most Lightning node implementations and can be done on the command-line or via one of the web management interfaces such as Ride the Lightning (RTL) (see XREF HERE).

Channel rebalancing is a complex issue that is the subject of active research and covered in more detail in XREF HERE.

Ride The Lightning (RTL)

RTL is a graphical web user interface to help users manage Lightning node operations for the three main Lightning node implementations (LND, c-lightning, and Eclair). RTL is an open source project developed by Suheb, Shahana Farooqui, and many other contributors. You can find the RTL software here:

https://github.com/Ride-The-Lightning/RTL

Here is an example screenshot of RTL’s web interface, as provided on the project repository:

Figure 5-3. Example RTL web interface

LNDMon

Lightning Labs, the makers of LND, provide a web-based graphical user interface called lndmon to monitor the various metrics of an LND Lightning node. lndmon only works with LND nodes. It is a read-only interface for monitoring and as such does not allow you to actively manage the node. It cannot open channels or make payments. Find lndmon here:

https://github.com/lightninglabs/lndmon