01-messaging.md (18495B)
1 # BOLT #1: Base Protocol 2 3 ## Overview 4 5 This protocol assumes an underlying authenticated and ordered transport mechanism that takes care of framing individual messages. 6 [BOLT #8](08-transport.md) specifies the canonical transport layer used in Lightning, though it can be replaced by any transport that fulfills the above guarantees. 7 8 The default TCP port depends on the network used. The most common networks are: 9 10 - Bitcoin mainnet with port number 9735 or the corresponding hexadecimal `0x2607`; 11 - Bitcoin testnet with port number 19735 (`0x4D17`); 12 - Bitcoin signet with port number 39735 (`0x9B37`). 13 14 The Unicode code point for LIGHTNING <sup>[1](#reference-1)</sup>, and the port convention try to follow the Bitcoin Core convention. 15 16 All data fields are unsigned big-endian unless otherwise specified. 17 18 ## Table of Contents 19 20 * [Connection Handling and Multiplexing](#connection-handling-and-multiplexing) 21 * [Lightning Message Format](#lightning-message-format) 22 * [Type-Length-Value Format](#type-length-value-format) 23 * [Fundamental Types](#fundamental-types) 24 * [Setup Messages](#setup-messages) 25 * [The `init` Message](#the-init-message) 26 * [The `error` and `warning` Messages](#the-error-and-warning-messages) 27 * [Control Messages](#control-messages) 28 * [The `ping` and `pong` Messages](#the-ping-and-pong-messages) 29 * [Peer Storage](#peer-storage) 30 * [The `peer_storage` and `peer_storage_retrieval` Messages](#the-peer_storage-and-peer_storage_retrieval-messages) 31 * [Appendix A: BigSize Test Vectors](#appendix-a-bigsize-test-vectors) 32 * [Appendix B: Type-Length-Value Test Vectors](#appendix-b-type-length-value-test-vectors) 33 * [Appendix C: Message Extension](#appendix-c-message-extension) 34 * [Appendix D: Signed Integers Test Vectors](#appendix-d-signed-integers-test-vectors) 35 * [Acknowledgments](#acknowledgments) 36 * [References](#references) 37 * [Authors](#authors) 38 39 ## Connection Handling and Multiplexing 40 41 Implementations MUST use a single connection per peer; channel messages (which include a channel ID) are multiplexed over this single connection. 42 43 ## Lightning Message Format 44 45 After decryption, all Lightning messages are of the form: 46 47 1. `type`: a 2-byte big-endian field indicating the type of message 48 2. `payload`: a variable-length payload that comprises the remainder of 49 the message and that conforms to a format matching the `type` 50 3. `extension`: an optional [TLV stream](#type-length-value-format) 51 52 The `type` field indicates how to interpret the `payload` field. 53 The format for each individual type is defined by a specification in this repository. 54 The type follows the _it's ok to be odd_ rule, so nodes MAY send _odd_-numbered types without ascertaining that the recipient understands it. 55 56 The messages are grouped logically into five groups, ordered by the most significant bit that is set: 57 58 - Setup & Control (types `0`-`31`): messages related to connection setup, control, supported features, and error reporting (described below) 59 - Channel (types `32`-`127`): messages used to setup and tear down micropayment channels (described in [BOLT #2](02-peer-protocol.md)) 60 - Commitment (types `128`-`255`): messages related to updating the current commitment transaction, which includes adding, revoking, and settling HTLCs as well as updating fees and exchanging signatures (described in [BOLT #2](02-peer-protocol.md)) 61 - Routing (types `256`-`511`): messages containing node and channel announcements, as well as any active route exploration (described in [BOLT #7](07-routing-gossip.md)) 62 - Custom (types `32768`-`65535`): experimental and application-specific messages 63 64 The size of the message is required by the transport layer to fit into a 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes. 65 66 A sending node: 67 - MUST NOT send an evenly-typed message not listed here without prior negotiation. 68 - MUST NOT send evenly-typed TLV records in the `extension` without prior negotiation. 69 - that negotiates an option in this specification: 70 - MUST include all the fields annotated with that option. 71 - When defining custom messages: 72 - SHOULD pick a random `type` to avoid collision with other custom types. 73 - SHOULD pick a `type` that doesn't conflict with other experiments listed in [this issue](https://github.com/lightningnetwork/lightning-rfc/issues/716). 74 - SHOULD pick an odd `type` identifiers when regular nodes should ignore the 75 additional data. 76 - SHOULD pick an even `type` identifiers when regular nodes should reject 77 the message and close the connection. 78 79 A receiving node: 80 - upon receiving a message of _odd_, unknown type: 81 - MUST ignore the received message. 82 - upon receiving a message of _even_, unknown type: 83 - MUST close the connection. 84 - MAY fail the channels. 85 - upon receiving a known message with insufficient length for the contents: 86 - MUST close the connection. 87 - MAY fail the channels. 88 - upon receiving a message with an `extension`: 89 - MAY ignore the `extension`. 90 - Otherwise, if the `extension` is invalid: 91 - MUST close the connection. 92 - MAY fail the channels. 93 94 ### Rationale 95 96 By default `SHA2` and Bitcoin public keys are both encoded as 97 big endian, thus it would be unusual to use a different endian for 98 other fields. 99 100 Length is limited to 65535 bytes by the cryptographic wrapping, and 101 messages in the protocol are never more than that length anyway. 102 103 The _it's ok to be odd_ rule allows for future optional extensions 104 without negotiation or special coding in clients. The _extension_ field 105 similarly allows for future expansion by letting senders include additional 106 TLV data. Note that an _extension_ field can only be added when the message 107 `payload` doesn't already fill the 65535 bytes maximum length. 108 109 Implementations may prefer to have message data aligned on an 8-byte 110 boundary (the largest natural alignment requirement of any type here); 111 however, adding a 6-byte padding after the type field was considered 112 wasteful: alignment may be achieved by decrypting the message into 113 a buffer with 6-bytes of pre-padding. 114 115 ## Type-Length-Value Format 116 117 Throughout the protocol, a TLV (Type-Length-Value) format is used to allow for 118 the backwards-compatible addition of new fields to existing message types. 119 120 A `tlv_record` represents a single field, encoded in the form: 121 122 * [`bigsize`: `type`] 123 * [`bigsize`: `length`] 124 * [`length`: `value`] 125 126 A `tlv_stream` is a series of (possibly zero) `tlv_record`s, represented as the 127 concatenation of the encoded `tlv_record`s. When used to extend existing 128 messages, a `tlv_stream` is typically placed after all currently defined fields. 129 130 The `type` is encoded using the BigSize format. It functions as a 131 message-specific, 64-bit identifier for the `tlv_record` determining how the 132 contents of `value` should be decoded. `type` identifiers below 2^16 are 133 reserved for use in this specification. `type` identifiers greater than or equal 134 to 2^16 are available for custom records. Any record not defined in this 135 specification is considered a custom record. This includes experimental and 136 application-specific messages. 137 138 The `length` is encoded using the BigSize format signaling the size of 139 `value` in bytes. 140 141 The `value` depends entirely on the `type`, and should be encoded or decoded 142 according to the message-specific format determined by `type`. 143 144 ### Requirements 145 146 The sending node: 147 - MUST order `tlv_record`s in a `tlv_stream` by strictly-increasing `type`, 148 hence MUST not produce more than a single TLV record with the same `type` 149 - MUST minimally encode `type` and `length`. 150 - When defining custom record `type` identifiers: 151 - SHOULD pick random `type` identifiers to avoid collision with other 152 custom types. 153 - SHOULD pick odd `type` identifiers when regular nodes should ignore the 154 additional data. 155 - SHOULD pick even `type` identifiers when regular nodes should reject the 156 full tlv stream containing the custom record. 157 - SHOULD NOT use redundant, variable-length encodings in a `tlv_record`. 158 159 The receiving node: 160 - if zero bytes remain before parsing a `type`: 161 - MUST stop parsing the `tlv_stream`. 162 - if a `type` or `length` is not minimally encoded: 163 - MUST fail to parse the `tlv_stream`. 164 - if decoded `type`s are not strictly-increasing (including situations when 165 two or more occurrences of the same `type` are met): 166 - MUST fail to parse the `tlv_stream`. 167 - if `length` exceeds the number of bytes remaining in the message: 168 - MUST fail to parse the `tlv_stream`. 169 - if `type` is known: 170 - MUST decode the next `length` bytes using the known encoding for `type`. 171 - if `length` is not exactly equal to that required for the known encoding 172 for `type`: 173 - MUST fail to parse the `tlv_stream`. 174 - if variable-length fields within the known encoding for `type` are not 175 minimal: 176 - MUST fail to parse the `tlv_stream`. 177 - otherwise, if `type` is unknown: 178 - if `type` is even: 179 - MUST fail to parse the `tlv_stream`. 180 - otherwise, if `type` is odd: 181 - MUST discard the next `length` bytes. 182 183 ### Rationale 184 185 The primary advantage in using TLV is that a reader is able to ignore new fields 186 that it does not understand, since each field carries the exact size of the 187 encoded element. Without TLV, even if a node does not wish to use a particular 188 field, the node is forced to add parsing logic for that field in order to 189 determine the offset of any fields that follow. 190 191 The strict monotonicity constraint ensures that all `type`s are unique and can 192 appear at most once. Fields that map to complex objects, e.g. vectors, maps, or 193 structs, should do so by defining the encoding such that the object is 194 serialized within a single `tlv_record`. The uniqueness constraint, among other 195 things, enables the following optimizations: 196 - canonical ordering is defined independent of the encoded `value`s. 197 - canonical ordering can be known at compile-time, rather than being determined 198 dynamically at the time of encoding. 199 - verifying canonical ordering requires less state and is less-expensive. 200 - variable-size fields can reserve their expected size up front, rather than 201 appending elements sequentially and incurring double-and-copy overhead. 202 203 The use of a bigsize for `type` and `length` permits a space savings for small 204 `type`s or short `value`s. This potentially leaves more space for application 205 data over the wire or in an onion payload. 206 207 All `type`s must appear in increasing order to create a canonical encoding of 208 the underlying `tlv_record`s. This is crucial when computing signatures over a 209 `tlv_stream`, as it ensures verifiers will be able to recompute the same message 210 digest as the signer. Note that the canonical ordering over the set of fields 211 can be enforced even if the verifier does not understand what the fields 212 contain. 213 214 Writers should avoid using redundant, variable-length encodings in a 215 `tlv_record` since this results in encoding the length twice and complicates 216 computing the outer length. As an example, when writing a variable length byte 217 array, the `value` should contain only the raw bytes and forgo an additional 218 internal length since the `tlv_record` already carries the number of bytes that 219 follow. On the other hand, if a `tlv_record` contains multiple, variable-length 220 elements then this would not be considered redundant, and is needed to allow the 221 receiver to parse individual elements from `value`. 222 223 ## Fundamental Types 224 225 Various fundamental types are referred to in the message specifications: 226 227 * `byte`: an 8-bit byte 228 * `s8`: an 8-bit signed integer 229 * `u16`: a 2 byte unsigned integer 230 * `s16`: a 2 byte signed integer 231 * `u32`: a 4 byte unsigned integer 232 * `s32`: a 4 byte signed integer 233 * `u64`: an 8 byte unsigned integer 234 * `s64`: an 8 byte signed integer 235 236 Signed integers use standard big-endian two's complement representation 237 (see test vectors [below](#appendix-d-signed-integers-test-vectors)). 238 239 For the final value in TLV records, truncated integers may be used. Leading 240 zeros in truncated integers MUST be omitted: 241 242 * `tu16`: a 0 to 2 byte truncated unsigned integer 243 * `tu32`: a 0 to 4 byte truncated unsigned integer 244 * `tu64`: a 0 to 8 byte truncated unsigned integer 245 246 When used to encode amounts, the previous fields MUST comply with the upper 247 bound of 21 million BTC: 248 249 * satoshi amounts MUST be at most `0x000775f05a074000` 250 * milli-satoshi amounts MUST be at most `0x1d24b2dfac520000` 251 252 The following convenience types are also defined: 253 254 * `chain_hash`: a 32-byte chain identifier (see [BOLT #0](00-introduction.md#glossary-and-terminology-guide)) 255 * `channel_id`: a 32-byte channel_id (see [BOLT #2](02-peer-protocol.md#definition-of-channel-id)) 256 * `sha256`: a 32-byte SHA2-256 hash 257 * `signature`: a 64-byte bitcoin Elliptic Curve signature 258 * `bip340sig`: a 64-byte bitcoin Elliptic Curve Schnorr signature as per [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki) 259 * `point`: a 33-byte Elliptic Curve point (compressed encoding as per [SEC 1 standard](http://www.secg.org/sec1-v2.pdf#subsubsection.2.3.3)) 260 * `short_channel_id`: an 8 byte value identifying a channel (see [BOLT #7](07-routing-gossip.md#definition-of-short_channel_id)) 261 * `sciddir_or_pubkey`: either 9 or 33 bytes referencing or identifying a node 262 * `bigsize`: a variable-length, unsigned integer similar to Bitcoin's CompactSize encoding, but big-endian. Described in [BigSize](#appendix-a-bigsize-test-vectors). 263 * `utf8`: a byte as part of a UTF-8 string. 264 265 ## Setup Messages 266 267 ### The `init` Message 268 269 Once authentication is complete, the first message reveals the features 270 supported or required by this node, even if this is a reconnection. 271 272 [BOLT #9](09-features.md) specifies lists of features. Each feature is 273 generally represented by 2 bits. The least-significant bit is numbered 0, 274 which is _even_, and the next most significant bit is numbered 1, which is 275 _odd_. For historical reasons, features are divided into global and local 276 feature bitmasks. 277 278 A feature is *offered* if a peer set it in the `init` message for the current 279 connection (as either even or odd). A feature is *negotiated* if either both 280 peers offered it, or the local node offered it as even: it can assume the peer 281 supports it, as it did not disconnect as it would be required to do. 282 283 The `features` field MUST be padded to bytes with 0s. 284 285 1. type: 16 (`init`) 286 2. data: 287 * [`u16`:`gflen`] 288 * [`gflen*byte`:`globalfeatures`] 289 * [`u16`:`flen`] 290 * [`flen*byte`:`features`] 291 * [`init_tlvs`:`tlvs`] 292 293 1. `tlv_stream`: `init_tlvs` 294 2. types: 295 1. type: 1 (`networks`) 296 2. data: 297 * [`...*chain_hash`:`chains`] 298 1. type: 3 (`remote_addr`) 299 2. data: 300 * [`...*byte`:`data`] 301 302 ### The `error` and `warning` Messages 303 304 For simplicity of diagnosis, it's often useful to tell a peer that something 305 is incorrect. 306 307 1. type: 17 (`error`) 308 2. data: 309 * [`channel_id`:`channel_id`] 310 * [`u16`:`len`] 311 * [`len*byte`:`data`] 312 313 1. type: 1 (`warning`) 314 2. data: 315 * [`channel_id`:`channel_id`] 316 * [`u16`:`len`] 317 * [`len*byte`:`data`] 318 319 ## Control Messages 320 321 ### The `ping` and `pong` Messages 322 323 In order to allow for the existence of long-lived TCP connections, at times it 324 may be required that both ends keep alive the TCP connection at the application 325 level. Such messages also allow obfuscation of traffic patterns. 326 327 1. type: 18 (`ping`) 328 2. data: 329 * [`u16`:`num_pong_bytes`] 330 * [`u16`:`byteslen`] 331 * [`byteslen*byte`:`ignored`] 332 333 1. type: 19 (`pong`) 334 2. data: 335 * [`u16`:`byteslen`] 336 * [`byteslen*byte`:`ignored`] 337 338 ## Peer Storage 339 340 ### The `peer_storage` and `peer_storage_retrieval` Messages 341 342 Nodes that advertise the `option_provide_storage` feature offer storing 343 arbitrary data for their peers. 344 345 1. type: 7 (`peer_storage`) 346 2. data: 347 * [`u16`:`length`] 348 * [`length*byte`:`blob`] 349 350 1. type: 9 (`peer_storage_retrieval`) 351 2. data: 352 * [`u16`:`length`] 353 * [`length*byte`:`blob`] 354 355 ## Appendix A: BigSize Test Vectors 356 357 Values encoded with BigSize will produce an encoding of either 1, 3, 5, or 9 358 bytes depending on the size of the integer. The encoding is a piece-wise 359 function that takes a `uint64` value `x` and produces: 360 361 ``` 362 uint8(x) if x < 0xfd 363 0xfd + be16(uint16(x)) if x < 0x10000 364 0xfe + be32(uint32(x)) if x < 0x100000000 365 0xff + be64(x) otherwise. 366 ``` 367 368 ### BigSize Decoding Tests 369 370 ```json 371 [ 372 { "name": "zero", "value": 0, "bytes": "00" }, 373 { "name": "one byte high", "value": 252, "bytes": "fc" }, 374 { "name": "two byte low", "value": 253, "bytes": "fd00fd" }, 375 { "name": "two byte high", "value": 65535, "bytes": "fdffff" }, 376 { "name": "four byte low", "value": 65536, "bytes": "fe00010000" }, 377 { "name": "four byte high", "value": 4294967295, "bytes": "feffffffff" }, 378 { "name": "eight byte low", "value": 4294967296, 379 "bytes": "ff0000000100000000" }, 380 { "name": "eight byte high", "value": 18446744073709551615, 381 "bytes": "ffffffffffffffffff" } 382 ] 383 ``` 384 385 ## Appendix D: Signed Integers Test Vectors 386 387 ```json 388 [ 389 { "value": 0, "bytes": "00" }, 390 { "value": 42, "bytes": "2a" }, 391 { "value": -42, "bytes": "d6" }, 392 { "value": 127, "bytes": "7f" }, 393 { "value": -128, "bytes": "80" }, 394 { "value": 128, "bytes": "0080" }, 395 { "value": -129, "bytes": "ff7f" }, 396 { "value": 15000, "bytes": "3a98" }, 397 { "value": -15000, "bytes": "c568" }, 398 { "value": 32767, "bytes": "7fff" }, 399 { "value": -32768, "bytes": "8000" }, 400 { "value": 32768, "bytes": "00008000" }, 401 { "value": -32769, "bytes": "ffff7fff" }, 402 { "value": 21000000, "bytes": "01406f40" }, 403 { "value": -21000000, "bytes": "febf90c0" }, 404 { "value": 2147483647, "bytes": "7fffffff" }, 405 { "value": -2147483648, "bytes": "80000000" }, 406 { "value": 2147483648, "bytes": "0000000080000000" }, 407 { "value": -2147483649, "bytes": "ffffffff7fffffff" }, 408 { "value": 500000000000, "bytes": "000000746a528800" }, 409 { "value": -500000000000, "bytes": "ffffff8b95ad7800" }, 410 { "value": 9223372036854775807, "bytes": "7fffffffffffffff" }, 411 { "value": -9223372036854775808, "bytes": "8000000000000000" } 412 ] 413 ``` 414 415 ## References 416 417 1. <a id="reference-1">http://www.unicode.org/charts/PDF/U2600.pdf</a> 418 419  420 <br> 421 This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).