01-messaging.md (43961B)
1 # BOLT #1: Base Protocol 2 3 ## Overview 4 5 This protocol assumes an underlying authenticated and ordered transport mechanism that takes care of framing individual messages. 6 [BOLT #8](08-transport.md) specifies the canonical transport layer used in Lightning, though it can be replaced by any transport that fulfills the above guarantees. 7 8 The default TCP port depends on the network used. The most common networks are: 9 10 - Bitcoin mainnet with port number 9735 or the corresponding hexadecimal `0x2607`; 11 - Bitcoin testnet with port number 19735 (`0x4D17`); 12 - Bitcoin signet with port number 39735 (`0x9B37`). 13 14 The Unicode code point for LIGHTNING <sup>[1](#reference-1)</sup>, and the port convention try to follow the Bitcoin Core convention. 15 16 All data fields are unsigned big-endian unless otherwise specified. 17 18 ## Table of Contents 19 20 * [Connection Handling and Multiplexing](#connection-handling-and-multiplexing) 21 * [Lightning Message Format](#lightning-message-format) 22 * [Type-Length-Value Format](#type-length-value-format) 23 * [Fundamental Types](#fundamental-types) 24 * [Setup Messages](#setup-messages) 25 * [The `init` Message](#the-init-message) 26 * [The `error` and `warning` Messages](#the-error-and-warning-messages) 27 * [Control Messages](#control-messages) 28 * [The `ping` and `pong` Messages](#the-ping-and-pong-messages) 29 * [Peer Storage](#peer-storage) 30 * [The `peer_storage` and `peer_storage_retrieval` Messages](#the-peer_storage-and-peer_storage_retrieval-messages) 31 * [Appendix A: BigSize Test Vectors](#appendix-a-bigsize-test-vectors) 32 * [Appendix B: Type-Length-Value Test Vectors](#appendix-b-type-length-value-test-vectors) 33 * [Appendix C: Message Extension](#appendix-c-message-extension) 34 * [Appendix D: Signed Integers Test Vectors](#appendix-d-signed-integers-test-vectors) 35 * [Acknowledgments](#acknowledgments) 36 * [References](#references) 37 * [Authors](#authors) 38 39 ## Connection Handling and Multiplexing 40 41 Implementations MUST use a single connection per peer; channel messages (which include a channel ID) are multiplexed over this single connection. 42 43 ## Lightning Message Format 44 45 After decryption, all Lightning messages are of the form: 46 47 1. `type`: a 2-byte big-endian field indicating the type of message 48 2. `payload`: a variable-length payload that comprises the remainder of 49 the message and that conforms to a format matching the `type` 50 3. `extension`: an optional [TLV stream](#type-length-value-format) 51 52 The `type` field indicates how to interpret the `payload` field. 53 The format for each individual type is defined by a specification in this repository. 54 The type follows the _it's ok to be odd_ rule, so nodes MAY send _odd_-numbered types without ascertaining that the recipient understands it. 55 56 The messages are grouped logically into five groups, ordered by the most significant bit that is set: 57 58 - Setup & Control (types `0`-`31`): messages related to connection setup, control, supported features, and error reporting (described below) 59 - Channel (types `32`-`127`): messages used to setup and tear down micropayment channels (described in [BOLT #2](02-peer-protocol.md)) 60 - Commitment (types `128`-`255`): messages related to updating the current commitment transaction, which includes adding, revoking, and settling HTLCs as well as updating fees and exchanging signatures (described in [BOLT #2](02-peer-protocol.md)) 61 - Routing (types `256`-`511`): messages containing node and channel announcements, as well as any active route exploration (described in [BOLT #7](07-routing-gossip.md)) 62 - Custom (types `32768`-`65535`): experimental and application-specific messages 63 64 The size of the message is required by the transport layer to fit into a 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes. 65 66 A sending node: 67 - MUST NOT send an evenly-typed message not listed here without prior negotiation. 68 - MUST NOT send evenly-typed TLV records in the `extension` without prior negotiation. 69 - that negotiates an option in this specification: 70 - MUST include all the fields annotated with that option. 71 - When defining custom messages: 72 - SHOULD pick a random `type` to avoid collision with other custom types. 73 - SHOULD pick a `type` that doesn't conflict with other experiments listed in [this issue](https://github.com/lightningnetwork/lightning-rfc/issues/716). 74 - SHOULD pick an odd `type` identifiers when regular nodes should ignore the 75 additional data. 76 - SHOULD pick an even `type` identifiers when regular nodes should reject 77 the message and close the connection. 78 79 A receiving node: 80 - upon receiving a message of _odd_, unknown type: 81 - MUST ignore the received message. 82 - upon receiving a message of _even_, unknown type: 83 - MUST close the connection. 84 - MAY fail the channels. 85 - upon receiving a known message with insufficient length for the contents: 86 - MUST close the connection. 87 - MAY fail the channels. 88 - upon receiving a message with an `extension`: 89 - MAY ignore the `extension`. 90 - Otherwise, if the `extension` is invalid: 91 - MUST close the connection. 92 - MAY fail the channels. 93 94 ### Rationale 95 96 By default `SHA2` and Bitcoin public keys are both encoded as 97 big endian, thus it would be unusual to use a different endian for 98 other fields. 99 100 Length is limited to 65535 bytes by the cryptographic wrapping, and 101 messages in the protocol are never more than that length anyway. 102 103 The _it's ok to be odd_ rule allows for future optional extensions 104 without negotiation or special coding in clients. The _extension_ field 105 similarly allows for future expansion by letting senders include additional 106 TLV data. Note that an _extension_ field can only be added when the message 107 `payload` doesn't already fill the 65535 bytes maximum length. 108 109 Implementations may prefer to have message data aligned on an 8-byte 110 boundary (the largest natural alignment requirement of any type here); 111 however, adding a 6-byte padding after the type field was considered 112 wasteful: alignment may be achieved by decrypting the message into 113 a buffer with 6-bytes of pre-padding. 114 115 ## Type-Length-Value Format 116 117 Throughout the protocol, a TLV (Type-Length-Value) format is used to allow for 118 the backwards-compatible addition of new fields to existing message types. 119 120 A `tlv_record` represents a single field, encoded in the form: 121 122 * [`bigsize`: `type`] 123 * [`bigsize`: `length`] 124 * [`length`: `value`] 125 126 A `tlv_stream` is a series of (possibly zero) `tlv_record`s, represented as the 127 concatenation of the encoded `tlv_record`s. When used to extend existing 128 messages, a `tlv_stream` is typically placed after all currently defined fields. 129 130 The `type` is encoded using the BigSize format. It functions as a 131 message-specific, 64-bit identifier for the `tlv_record` determining how the 132 contents of `value` should be decoded. `type` identifiers below 2^16 are 133 reserved for use in this specification. `type` identifiers greater than or equal 134 to 2^16 are available for custom records. Any record not defined in this 135 specification is considered a custom record. This includes experimental and 136 application-specific messages. 137 138 The `length` is encoded using the BigSize format signaling the size of 139 `value` in bytes. 140 141 The `value` depends entirely on the `type`, and should be encoded or decoded 142 according to the message-specific format determined by `type`. 143 144 ### Requirements 145 146 The sending node: 147 - MUST order `tlv_record`s in a `tlv_stream` by strictly-increasing `type`, 148 hence MUST not produce more than a single TLV record with the same `type` 149 - MUST minimally encode `type` and `length`. 150 - When defining custom record `type` identifiers: 151 - SHOULD pick random `type` identifiers to avoid collision with other 152 custom types. 153 - SHOULD pick odd `type` identifiers when regular nodes should ignore the 154 additional data. 155 - SHOULD pick even `type` identifiers when regular nodes should reject the 156 full tlv stream containing the custom record. 157 - SHOULD NOT use redundant, variable-length encodings in a `tlv_record`. 158 159 The receiving node: 160 - if zero bytes remain before parsing a `type`: 161 - MUST stop parsing the `tlv_stream`. 162 - if a `type` or `length` is not minimally encoded: 163 - MUST fail to parse the `tlv_stream`. 164 - if decoded `type`s are not strictly-increasing (including situations when 165 two or more occurrences of the same `type` are met): 166 - MUST fail to parse the `tlv_stream`. 167 - if `length` exceeds the number of bytes remaining in the message: 168 - MUST fail to parse the `tlv_stream`. 169 - if `type` is known: 170 - MUST decode the next `length` bytes using the known encoding for `type`. 171 - if `length` is not exactly equal to that required for the known encoding for `type`: 172 - MUST fail to parse the `tlv_stream`. 173 - if variable-length fields within the known encoding for `type` are not minimal: 174 - MUST fail to parse the `tlv_stream`. 175 - otherwise, if `type` is unknown: 176 - if `type` is even: 177 - MUST fail to parse the `tlv_stream`. 178 - otherwise, if `type` is odd: 179 - MUST discard the next `length` bytes. 180 181 ### Rationale 182 183 The primary advantage in using TLV is that a reader is able to ignore new fields 184 that it does not understand, since each field carries the exact size of the 185 encoded element. Without TLV, even if a node does not wish to use a particular 186 field, the node is forced to add parsing logic for that field in order to 187 determine the offset of any fields that follow. 188 189 The strict monotonicity constraint ensures that all `type`s are unique and can 190 appear at most once. Fields that map to complex objects, e.g. vectors, maps, or 191 structs, should do so by defining the encoding such that the object is 192 serialized within a single `tlv_record`. The uniqueness constraint, among other 193 things, enables the following optimizations: 194 - canonical ordering is defined independent of the encoded `value`s. 195 - canonical ordering can be known at compile-time, rather than being determined 196 dynamically at the time of encoding. 197 - verifying canonical ordering requires less state and is less-expensive. 198 - variable-size fields can reserve their expected size up front, rather than 199 appending elements sequentially and incurring double-and-copy overhead. 200 201 The use of a bigsize for `type` and `length` permits a space savings for small 202 `type`s or short `value`s. This potentially leaves more space for application 203 data over the wire or in an onion payload. 204 205 All `type`s must appear in increasing order to create a canonical encoding of 206 the underlying `tlv_record`s. This is crucial when computing signatures over a 207 `tlv_stream`, as it ensures verifiers will be able to recompute the same message 208 digest as the signer. Note that the canonical ordering over the set of fields 209 can be enforced even if the verifier does not understand what the fields 210 contain. 211 212 Writers should avoid using redundant, variable-length encodings in a 213 `tlv_record` since this results in encoding the length twice and complicates 214 computing the outer length. As an example, when writing a variable length byte 215 array, the `value` should contain only the raw bytes and forgo an additional 216 internal length since the `tlv_record` already carries the number of bytes that 217 follow. On the other hand, if a `tlv_record` contains multiple, variable-length 218 elements then this would not be considered redundant, and is needed to allow the 219 receiver to parse individual elements from `value`. 220 221 ## Fundamental Types 222 223 Various fundamental types are referred to in the message specifications: 224 225 * `byte`: an 8-bit byte 226 * `s8`: an 8-bit signed integer 227 * `u16`: a 2 byte unsigned integer 228 * `s16`: a 2 byte signed integer 229 * `u32`: a 4 byte unsigned integer 230 * `s32`: a 4 byte signed integer 231 * `u64`: an 8 byte unsigned integer 232 * `s64`: an 8 byte signed integer 233 234 Signed integers use standard big-endian two's complement representation 235 (see test vectors [below](#appendix-d-signed-integers-test-vectors)). 236 237 For the final value in TLV records, truncated integers may be used. Leading zeros in 238 truncated integers MUST be omitted: 239 240 * `tu16`: a 0 to 2 byte truncated unsigned integer 241 * `tu32`: a 0 to 4 byte truncated unsigned integer 242 * `tu64`: a 0 to 8 byte truncated unsigned integer 243 244 When used to encode amounts, the previous fields MUST comply with the upper 245 bound of 21 million BTC: 246 247 * satoshi amounts MUST be at most `0x000775f05a074000` 248 * milli-satoshi amounts MUST be at most `0x1d24b2dfac520000` 249 250 The following convenience types are also defined: 251 252 * `chain_hash`: a 32-byte chain identifier (see [BOLT #0](00-introduction.md#glossary-and-terminology-guide)) 253 * `channel_id`: a 32-byte channel_id (see [BOLT #2](02-peer-protocol.md#definition-of-channel-id)) 254 * `sha256`: a 32-byte SHA2-256 hash 255 * `signature`: a 64-byte bitcoin Elliptic Curve signature 256 * `bip340sig`: a 64-byte bitcoin Elliptic Curve Schnorr signature as per [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki) 257 * `point`: a 33-byte Elliptic Curve point (compressed encoding as per [SEC 1 standard](http://www.secg.org/sec1-v2.pdf#subsubsection.2.3.3)) 258 * `short_channel_id`: an 8 byte value identifying a channel (see [BOLT #7](07-routing-gossip.md#definition-of-short-channel-id)) 259 * `sciddir_or_pubkey`: either 9 or 33 bytes referencing or identifying a node, respectively 260 * if the first byte is 0 or 1, then an 8-byte `short_channel_id` follows for a total of 9 bytes 261 * 0 for the first byte indicates this refers to `node_id_1` in the `channel_announcement` for `short_channel_id` 262 * 1 for the first byte indicates this refers to `node_id_2` in the `channel_announcement` for `short_channel_id` 263 (see [BOLT #7](07-routing-gossip.md#the-channel_announcement-message) 264 * if the first byte is 2 or 3, then the value is a 33-byte `point` 265 * `bigsize`: a variable-length, unsigned integer similar to Bitcoin's CompactSize encoding, but big-endian. Described in [BigSize](#appendix-a-bigsize-test-vectors). 266 * `utf8`: a byte as part of a UTF-8 string. A writer MUST ensure an array of these is a valid UTF-8 string, a reader MAY reject any messages containing an array of these which is not a valid UTF-8 string. 267 268 ## Setup Messages 269 270 ### The `init` Message 271 272 Once authentication is complete, the first message reveals the features supported or required by this node, even if this is a reconnection. 273 274 [BOLT #9](09-features.md) specifies lists of features. Each feature is generally represented by 2 bits. The least-significant bit is numbered 0, which is _even_, and the next most significant bit is numbered 1, which is _odd_. For historical reasons, features are divided into global and local feature bitmasks. 275 276 A feature is *offered* if a peer set it in the `init` message for the current connection (as either even or odd). A feature is *negotiated* if either both peers offered it, or the local node offered it as even: it can assume the peer supports it, as it did not disconnect as it would be required to do. 277 278 The `features` field MUST be padded to bytes with 0s. 279 280 1. type: 16 (`init`) 281 2. data: 282 * [`u16`:`gflen`] 283 * [`gflen*byte`:`globalfeatures`] 284 * [`u16`:`flen`] 285 * [`flen*byte`:`features`] 286 * [`init_tlvs`:`tlvs`] 287 288 1. `tlv_stream`: `init_tlvs` 289 2. types: 290 1. type: 1 (`networks`) 291 2. data: 292 * [`...*chain_hash`:`chains`] 293 1. type: 3 (`remote_addr`) 294 2. data: 295 * [`...*byte`:`data`] 296 297 The optional `networks` indicates the chains the node is interested in. 298 The optional `remote_addr` can be used to circumvent NAT issues. 299 300 #### Requirements 301 302 The sending node: 303 - MUST send `init` as the first Lightning message for any connection. 304 - MUST set feature bits as defined in [BOLT #9](09-features.md). 305 - MUST set any undefined feature bits to 0. 306 - SHOULD NOT set features greater than 13 in `globalfeatures`. 307 - SHOULD use the minimum length required to represent the `features` field. 308 - SHOULD set `networks` to all chains it will gossip or open channels for. 309 - SHOULD set `remote_addr` to reflect the remote IP address (and port) of an 310 incoming connection, if the node is the receiver and the connection was done 311 via IP. 312 - if it sets `remote_addr`: 313 - MUST set it to a valid `address descriptor` (1 byte type and data) as described in [BOLT 7](07-routing-gossip.md#the-node_announcement-message). 314 - SHOULD NOT set private addresses as `remote_addr`. 315 316 The receiving node: 317 - MUST wait to receive `init` before sending any other messages. 318 - MUST combine (bitwise OR) the two feature bitmaps into one logical `features` map. 319 - MUST respond to known feature bits as specified in [BOLT #9](09-features.md). 320 - upon receiving unknown _odd_ feature bits that are non-zero: 321 - MUST ignore the bit. 322 - upon receiving unknown _even_ feature bits that are non-zero: 323 - MUST close the connection. 324 - upon receiving `networks` containing no common chains 325 - MAY close the connection. 326 - if the feature vector does not set all known, transitive dependencies: 327 - MUST close the connection. 328 - MAY use the `remote_addr` to update its `node_announcement` 329 330 #### Rationale 331 332 There used to be two feature bitfields here, but for backwards compatibility they're now 333 combined into one. 334 335 This semantic allows both future incompatible changes and future backward compatible changes. Bits should generally be assigned in pairs, in order that optional features may later become compulsory. 336 337 Nodes wait for receipt of the other's features to simplify error 338 diagnosis when features are incompatible. 339 340 Since all networks share the same port, but most implementations only 341 support a single network, the `networks` fields avoids nodes 342 erroneously believing they will receive updates about their preferred 343 network, or that they can open channels. 344 345 ### The `error` and `warning` Messages 346 347 For simplicity of diagnosis, it's often useful to tell a peer that something is incorrect. 348 349 1. type: 17 (`error`) 350 2. data: 351 * [`channel_id`:`channel_id`] 352 * [`u16`:`len`] 353 * [`len*byte`:`data`] 354 355 1. type: 1 (`warning`) 356 2. data: 357 * [`channel_id`:`channel_id`] 358 * [`u16`:`len`] 359 * [`len*byte`:`data`] 360 361 #### Requirements 362 363 The channel is referred to by `channel_id`, unless `channel_id` is 0 (i.e. all bytes are 0), in which case it refers to all channels. 364 365 The funding node using channel establishment v1 (`open_channel`): 366 - for all error messages sent before (and including) the `funding_created` message: 367 - MUST use `temporary_channel_id` in lieu of `channel_id`. 368 369 The fundee node using channel establishment v1 (`accept_channel`): 370 - for all error messages sent before (and not including) the `funding_signed` message: 371 - MUST use `temporary_channel_id` in lieu of `channel_id`. 372 373 The opener node using channel establishment v2 (`open_channel2`): 374 - for all error messages sent before the `accept_channel2` message is received: 375 - MUST use `temporary_channel_id` in lieu of `channel_id`. 376 377 The accepter node using channel establishment v2 (`open_channel2`): 378 - for all error messages sent before (and including) the `accept_channel2` message: 379 - MUST use `temporary_channel_id` in lieu of `channel_id`. 380 381 A sending node: 382 - SHOULD send `error` for protocol violations or internal errors that make channels unusable or that make further communication unusable. 383 - SHOULD send `error` with the unknown `channel_id` in reply to messages of type `32`-`255` related to unknown channels. 384 - when sending `error`: 385 - MUST fail the channel(s) referred to by the error message. 386 - MAY set `channel_id` to all zero to indicate all channels. 387 - when sending `warning`: 388 - MAY set `channel_id` to all zero if the warning is not related to a specific channel. 389 - MAY send an empty `data` field. 390 - when failure was caused by an invalid signature check: 391 - SHOULD include the raw, hex-encoded transaction in reply to a `funding_created`, `funding_signed`, `closing_signed`, or `commitment_signed` message. 392 393 The receiving node: 394 - upon receiving `error`: 395 - if `channel_id` is all zero: 396 - MUST fail all channels with the sending node. 397 - otherwise: 398 - MUST fail the channel referred to by `channel_id`, if that channel is with the sending node. 399 - upon receiving `warning`: 400 - SHOULD log the message for later diagnosis. 401 - MAY attempt `shutdown` if permitted at this point. 402 - if no existing channel is referred to by `channel_id`: 403 - MUST ignore the message. 404 - if `data` is not composed solely of printable ASCII characters (For reference: the printable character set includes byte values 32 through 126, inclusive): 405 - SHOULD NOT print out `data` verbatim. 406 407 #### Rationale 408 409 There are unrecoverable errors that require an abort of conversations; 410 if the connection is simply dropped, then the peer may retry the 411 connection. It's also useful to describe protocol violations for 412 diagnosis, as this indicates that one peer has a bug. 413 414 On the other hand, overuse of error messages has lead to 415 implementations ignoring them (to avoid an otherwise expensive channel 416 break), so the "warning" message was added to allow some degree of 417 retry or recovery for spurious errors. 418 419 It may be wise not to distinguish errors in production settings, lest 420 it leak information — hence, the optional `data` field. 421 422 ## Control Messages 423 424 ### The `ping` and `pong` Messages 425 426 In order to allow for the existence of long-lived TCP connections, at 427 times it may be required that both ends keep alive the TCP connection at the 428 application level. Such messages also allow obfuscation of traffic patterns. 429 430 1. type: 18 (`ping`) 431 2. data: 432 * [`u16`:`num_pong_bytes`] 433 * [`u16`:`byteslen`] 434 * [`byteslen*byte`:`ignored`] 435 436 The `pong` message is to be sent whenever a `ping` message is received. It 437 serves as a reply and also serves to keep the connection alive, while 438 explicitly notifying the other end that the receiver is still active. Within 439 the received `ping` message, the sender will specify the number of bytes to be 440 included within the data payload of the `pong` message. 441 442 1. type: 19 (`pong`) 443 2. data: 444 * [`u16`:`byteslen`] 445 * [`byteslen*byte`:`ignored`] 446 447 #### Requirements 448 449 A node sending a `ping` message: 450 - SHOULD set `ignored` to 0s. 451 - MUST NOT set `ignored` to sensitive data such as secrets or portions of initialized 452 memory. 453 - if it doesn't receive a corresponding `pong`: 454 - MAY close the network connection, 455 - and MUST NOT fail the channels in this case. 456 457 A node sending a `pong` message: 458 - SHOULD set `ignored` to 0s. 459 - MUST NOT set `ignored` to sensitive data such as secrets or portions of initialized 460 memory. 461 462 A node receiving a `ping` message: 463 - if `num_pong_bytes` is less than 65532: 464 - MUST respond by sending a `pong` message, with `byteslen` equal to `num_pong_bytes`. 465 - otherwise (`num_pong_bytes` is **not** less than 65532): 466 - MUST ignore the `ping`. 467 468 A node receiving a `pong` message: 469 - if `byteslen` does not correspond to any `ping`'s `num_pong_bytes` value it has sent: 470 - MAY close the connection. 471 472 ### Rationale 473 474 The largest possible message is 65535 bytes; thus, the maximum sensible `byteslen` 475 is 65531 — in order to account for the type field (`pong`) and the `byteslen` itself. This allows 476 a convenient cutoff for `num_pong_bytes` to indicate that no reply should be sent. 477 478 Connections between nodes within the network may be long lived, as payment 479 channels have an indefinite lifetime. However, it's likely that 480 no new data will be 481 exchanged for a 482 significant portion of a connection's lifetime. Also, on several platforms it's possible that Lightning 483 clients will be put to sleep without prior warning. Hence, a 484 distinct `ping` message is used, in order to probe for the liveness of the connection on 485 the other side, as well as to keep the established connection active. 486 487 Additionally, the ability for a sender to request that the receiver send a 488 response with a particular number of bytes enables nodes on the network to 489 create _synthetic_ traffic. Such traffic can be used to partially defend 490 against packet and timing analysis — as nodes can fake the traffic patterns of 491 typical exchanges without applying any true updates to their respective 492 channels. 493 494 When combined with the onion routing protocol defined in 495 [BOLT #4](04-onion-routing.md), 496 careful statistically driven synthetic traffic can serve to further bolster the 497 privacy of participants within the network. 498 499 Limited precautions are recommended against `ping` flooding, however some 500 latitude is given because of network delays. Note that there are other methods 501 of incoming traffic flooding (e.g. sending _odd_ unknown message types, or padding 502 every message maximally). 503 504 Finally, the usage of periodic `ping` messages serves to promote frequent key 505 rotations as specified within [BOLT #8](08-transport.md). 506 507 ## Peer storage 508 509 ### The `peer_storage` and `peer_storage_retrieval` Messages 510 511 Nodes that advertise the `option_provide_storage` feature offer storing 512 arbitrary data for their peers. The data stored must not exceed 65531 bytes, 513 which lets it fit in lightning messages. 514 515 Nodes can verify that their `option_provide_storage` peers correctly store 516 their data at each reconnection, by comparing the contents of the 517 retrieved data with the last one they sent. However, nodes should not expect 518 their peers to always have their latest data available. 519 520 Nodes ask their peers to store data using the `peer_storage` message and expect 521 peers to return the latest data to them using the `peer_storage_retrieval` message: 522 523 1. type: 7 (`peer_storage`) 524 2. data: 525 * [`u16`:`length`] 526 * [`length*byte`:`blob`] 527 528 529 1. type: 9 (`peer_storage_retrieval`) 530 2. data: 531 * [`u16`:`length`] 532 * [`length*byte`:`blob`] 533 534 535 Requirements: 536 537 The sender of `peer_storage`: 538 - MAY send `peer_storage` whenever necessary. 539 - MUST limit its `blob` to 65531 bytes. 540 - MUST encrypt the data in a manner that ensures its integrity upon receipt. 541 - SHOULD pad the `blob` to ensure its length is always exactly 65531 bytes. 542 543 The receiver of `peer_storage`: 544 - If it offered `option_provide_storage`: 545 - if it has an open channel with the sender: 546 - MUST store the message. 547 - MAY store the message anyway. 548 549 - If it does store the message: 550 - MAY delay storage to ratelimit peer to no more than one update per minute. 551 - MUST replace the old `blob` with the latest received. 552 - MUST send `peer_storage_retrieval` again after reconnection, after exchanging `init` messages. 553 554 555 The sender of `peer_storage_retrieval`: 556 - MUST include the last `blob` it stored for that peer. 557 - when all channels with that peer are closed: 558 - SHOULD wait at least 2016 blocks before deleting the `blob`. 559 560 The receiver of `peer_storage_retrieval`: 561 - when it receives `peer_storage_retrieval` with an outdated or irrelevant data: 562 - MAY send a warning. 563 564 #### Rationale: 565 The `peer_storage` and `peer_storage_retrieval` messages enable nodes to securely store 566 and share data with other nodes in the network, serving as a backup mechanism for important 567 information. By utilizing them, nodes can safeguard crucial data, enhancing the network's 568 resilience and reliability. Additionally, even if we don't have an open channel, some nodes 569 might provide this service in exchange for some sats so they may store `peer_storage`. 570 571 Nodes should pad the `blob` to obscure its actual size, enhancing privacy by making size-based 572 analysis more difficult for the receiver. 573 574 `peer_storage_retrieval` should not be sent after `channel_reestablish` because then the user 575 wouldn't have an option to recover the node and update its state in case they lost data. 576 577 Nodes should send a `peer_storage` message whenever they wish to update the `blob` stored with their peers. 578 This `blob` can be used to distribute encrypted data, which could be helpful in restoring the node. 579 580 ## Appendix A: BigSize Test Vectors 581 582 The following test vectors can be used to assert the correctness of a BigSize 583 implementation used in the TLV format. The format is identical to the 584 CompactSize encoding used in bitcoin, but replaces the little-endian encoding of 585 multi-byte values with big-endian. 586 587 Values encoded with BigSize will produce an encoding of either 1, 3, 5, or 9 588 bytes depending on the size of the integer. The encoding is a piece-wise 589 function that takes a `uint64` value `x` and produces: 590 ``` 591 uint8(x) if x < 0xfd 592 0xfd + be16(uint16(x)) if x < 0x10000 593 0xfe + be32(uint32(x)) if x < 0x100000000 594 0xff + be64(x) otherwise. 595 ``` 596 597 Here `+` denotes concatenation and `be16`, `be32`, and `be64` produce a 598 big-endian encoding of the input for 16, 32, and 64-bit integers, respectively. 599 600 A value is said to be _minimally encoded_ if it could not be encoded using 601 fewer bytes. For example, a BigSize encoding that occupies 5 bytes 602 but whose value is less than 0x10000 is not minimally encoded. All values 603 decoded with BigSize should be checked to ensure they are minimally encoded. 604 605 ### BigSize Decoding Tests 606 607 The following is an example of how to execute the BigSize decoding tests. 608 ```golang 609 func testReadBigSize(t *testing.T, test bigSizeTest) { 610 var buf [8]byte 611 r := bytes.NewReader(test.Bytes) 612 val, err := tlv.ReadBigSize(r, &buf) 613 if err != nil && err.Error() != test.ExpErr { 614 t.Fatalf("expected decoding error: %v, got: %v", 615 test.ExpErr, err) 616 } 617 618 // If we expected a decoding error, there's no point checking the value. 619 if test.ExpErr != "" { 620 return 621 } 622 623 if val != test.Value { 624 t.Fatalf("expected value: %d, got %d", test.Value, val) 625 } 626 } 627 ``` 628 629 A correct implementation should pass against these test vectors: 630 ```json 631 [ 632 { 633 "name": "zero", 634 "value": 0, 635 "bytes": "00" 636 }, 637 { 638 "name": "one byte high", 639 "value": 252, 640 "bytes": "fc" 641 }, 642 { 643 "name": "two byte low", 644 "value": 253, 645 "bytes": "fd00fd" 646 }, 647 { 648 "name": "two byte high", 649 "value": 65535, 650 "bytes": "fdffff" 651 }, 652 { 653 "name": "four byte low", 654 "value": 65536, 655 "bytes": "fe00010000" 656 }, 657 { 658 "name": "four byte high", 659 "value": 4294967295, 660 "bytes": "feffffffff" 661 }, 662 { 663 "name": "eight byte low", 664 "value": 4294967296, 665 "bytes": "ff0000000100000000" 666 }, 667 { 668 "name": "eight byte high", 669 "value": 18446744073709551615, 670 "bytes": "ffffffffffffffffff" 671 }, 672 { 673 "name": "two byte not canonical", 674 "value": 0, 675 "bytes": "fd00fc", 676 "exp_error": "decoded bigsize is not canonical" 677 }, 678 { 679 "name": "four byte not canonical", 680 "value": 0, 681 "bytes": "fe0000ffff", 682 "exp_error": "decoded bigsize is not canonical" 683 }, 684 { 685 "name": "eight byte not canonical", 686 "value": 0, 687 "bytes": "ff00000000ffffffff", 688 "exp_error": "decoded bigsize is not canonical" 689 }, 690 { 691 "name": "two byte short read", 692 "value": 0, 693 "bytes": "fd00", 694 "exp_error": "unexpected EOF" 695 }, 696 { 697 "name": "four byte short read", 698 "value": 0, 699 "bytes": "feffff", 700 "exp_error": "unexpected EOF" 701 }, 702 { 703 "name": "eight byte short read", 704 "value": 0, 705 "bytes": "ffffffffff", 706 "exp_error": "unexpected EOF" 707 }, 708 { 709 "name": "one byte no read", 710 "value": 0, 711 "bytes": "", 712 "exp_error": "EOF" 713 }, 714 { 715 "name": "two byte no read", 716 "value": 0, 717 "bytes": "fd", 718 "exp_error": "unexpected EOF" 719 }, 720 { 721 "name": "four byte no read", 722 "value": 0, 723 "bytes": "fe", 724 "exp_error": "unexpected EOF" 725 }, 726 { 727 "name": "eight byte no read", 728 "value": 0, 729 "bytes": "ff", 730 "exp_error": "unexpected EOF" 731 } 732 ] 733 ``` 734 735 ### BigSize Encoding Tests 736 737 The following is an example of how to execute the BigSize encoding tests. 738 ```golang 739 func testWriteBigSize(t *testing.T, test bigSizeTest) { 740 var ( 741 w bytes.Buffer 742 buf [8]byte 743 ) 744 err := tlv.WriteBigSize(&w, test.Value, &buf) 745 if err != nil { 746 t.Fatalf("unable to encode %d as bigsize: %v", 747 test.Value, err) 748 } 749 750 if bytes.Compare(w.Bytes(), test.Bytes) != 0 { 751 t.Fatalf("expected bytes: %v, got %v", 752 test.Bytes, w.Bytes()) 753 } 754 } 755 ``` 756 757 A correct implementation should pass against the following test vectors: 758 ```json 759 [ 760 { 761 "name": "zero", 762 "value": 0, 763 "bytes": "00" 764 }, 765 { 766 "name": "one byte high", 767 "value": 252, 768 "bytes": "fc" 769 }, 770 { 771 "name": "two byte low", 772 "value": 253, 773 "bytes": "fd00fd" 774 }, 775 { 776 "name": "two byte high", 777 "value": 65535, 778 "bytes": "fdffff" 779 }, 780 { 781 "name": "four byte low", 782 "value": 65536, 783 "bytes": "fe00010000" 784 }, 785 { 786 "name": "four byte high", 787 "value": 4294967295, 788 "bytes": "feffffffff" 789 }, 790 { 791 "name": "eight byte low", 792 "value": 4294967296, 793 "bytes": "ff0000000100000000" 794 }, 795 { 796 "name": "eight byte high", 797 "value": 18446744073709551615, 798 "bytes": "ffffffffffffffffff" 799 } 800 ] 801 ``` 802 803 ## Appendix B: Type-Length-Value Test Vectors 804 805 The following tests assume that two separate TLV namespaces exist: n1 and n2. 806 807 The n1 namespace supports the following TLV types: 808 809 1. `tlv_stream`: `n1` 810 2. types: 811 1. type: 1 (`tlv1`) 812 2. data: 813 * [`tu64`:`amount_msat`] 814 1. type: 2 (`tlv2`) 815 2. data: 816 * [`short_channel_id`:`scid`] 817 1. type: 3 (`tlv3`) 818 2. data: 819 * [`point`:`node_id`] 820 * [`u64`:`amount_msat_1`] 821 * [`u64`:`amount_msat_2`] 822 1. type: 254 (`tlv4`) 823 2. data: 824 * [`u16`:`cltv_delta`] 825 826 The n2 namespace supports the following TLV types: 827 828 1. `tlv_stream`: `n2` 829 2. types: 830 1. type: 0 (`tlv1`) 831 2. data: 832 * [`tu64`:`amount_msat`] 833 1. type: 11 (`tlv2`) 834 2. data: 835 * [`tu32`:`cltv_expiry`] 836 837 ### TLV Decoding Failures 838 839 The following TLV streams in any namespace should trigger a decoding failure: 840 841 1. Invalid stream: 0xfd 842 2. Reason: type truncated 843 844 1. Invalid stream: 0xfd01 845 2. Reason: type truncated 846 847 1. Invalid stream: 0xfd0001 00 848 2. Reason: not minimally encoded type 849 850 1. Invalid stream: 0xfd0101 851 2. Reason: missing length 852 853 1. Invalid stream: 0x0f fd 854 2. Reason: (length truncated) 855 856 1. Invalid stream: 0x0f fd26 857 2. Reason: (length truncated) 858 859 1. Invalid stream: 0x0f fd2602 860 2. Reason: missing value 861 862 1. Invalid stream: 0x0f fd0001 00 863 2. Reason: not minimally encoded length 864 865 1. Invalid stream: 0x0f fd0201 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 866 2. Reason: value truncated 867 868 The following TLV streams in either namespace should trigger a 869 decoding failure: 870 871 1. Invalid stream: 0x12 00 872 2. Reason: unknown even type. 873 874 1. Invalid stream: 0xfd0102 00 875 2. Reason: unknown even type. 876 877 1. Invalid stream: 0xfe01000002 00 878 2. Reason: unknown even type. 879 880 1. Invalid stream: 0xff0100000000000002 00 881 2. Reason: unknown even type. 882 883 The following TLV streams in namespace `n1` should trigger a decoding 884 failure: 885 886 1. Invalid stream: 0x01 09 ffffffffffffffffff 887 2. Reason: greater than encoding length for `n1`s `tlv1`. 888 889 1. Invalid stream: 0x01 01 00 890 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 891 892 1. Invalid stream: 0x01 02 0001 893 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 894 895 1. Invalid stream: 0x01 03 000100 896 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 897 898 1. Invalid stream: 0x01 04 00010000 899 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 900 901 1. Invalid stream: 0x01 05 0001000000 902 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 903 904 1. Invalid stream: 0x01 06 000100000000 905 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 906 907 1. Invalid stream: 0x01 07 00010000000000 908 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 909 910 1. Invalid stream: 0x01 08 0001000000000000 911 2. Reason: encoding for `n1`s `tlv1`s `amount_msat` is not minimal 912 913 1. Invalid stream: 0x02 07 01010101010101 914 2. Reason: less than encoding length for `n1`s `tlv2`. 915 916 1. Invalid stream: 0x02 09 010101010101010101 917 2. Reason: greater than encoding length for `n1`s `tlv2`. 918 919 1. Invalid stream: 0x03 21 023da092f6980e58d2c037173180e9a465476026ee50f96695963e8efe436f54eb 920 2. Reason: less than encoding length for `n1`s `tlv3`. 921 922 1. Invalid stream: 0x03 29 023da092f6980e58d2c037173180e9a465476026ee50f96695963e8efe436f54eb0000000000000001 923 2. Reason: less than encoding length for `n1`s `tlv3`. 924 925 1. Invalid stream: 0x03 30 023da092f6980e58d2c037173180e9a465476026ee50f96695963e8efe436f54eb000000000000000100000000000001 926 2. Reason: less than encoding length for `n1`s `tlv3`. 927 928 1. Invalid stream: 0x03 31 043da092f6980e58d2c037173180e9a465476026ee50f96695963e8efe436f54eb00000000000000010000000000000002 929 2. Reason: `n1`s `node_id` is not a valid point. 930 931 1. Invalid stream: 0x03 32 023da092f6980e58d2c037173180e9a465476026ee50f96695963e8efe436f54eb0000000000000001000000000000000001 932 2. Reason: greater than encoding length for `n1`s `tlv3`. 933 934 1. Invalid stream: 0xfd00fe 00 935 2. Reason: less than encoding length for `n1`s `tlv4`. 936 937 1. Invalid stream: 0xfd00fe 01 01 938 2. Reason: less than encoding length for `n1`s `tlv4`. 939 940 1. Invalid stream: 0xfd00fe 03 010101 941 2. Reason: greater than encoding length for `n1`s `tlv4`. 942 943 1. Invalid stream: 0x00 00 944 2. Reason: unknown even field for `n1`s namespace. 945 946 ### TLV Decoding Successes 947 948 The following TLV streams in either namespace should correctly decode, 949 and be ignored: 950 951 1. Valid stream: 0x 952 2. Explanation: empty message 953 954 1. Valid stream: 0x21 00 955 2. Explanation: Unknown odd type. 956 957 1. Valid stream: 0xfd0201 00 958 2. Explanation: Unknown odd type. 959 960 1. Valid stream: 0xfd00fd 00 961 2. Explanation: Unknown odd type. 962 963 1. Valid stream: 0xfd00ff 00 964 2. Explanation: Unknown odd type. 965 966 1. Valid stream: 0xfe02000001 00 967 2. Explanation: Unknown odd type. 968 969 1. Valid stream: 0xff0200000000000001 00 970 2. Explanation: Unknown odd type. 971 972 The following TLV streams in `n1` namespace should correctly decode, 973 with the values given here: 974 975 1. Valid stream: 0x01 00 976 2. Values: `tlv1` `amount_msat`=0 977 978 1. Valid stream: 0x01 01 01 979 2. Values: `tlv1` `amount_msat`=1 980 981 1. Valid stream: 0x01 02 0100 982 2. Values: `tlv1` `amount_msat`=256 983 984 1. Valid stream: 0x01 03 010000 985 2. Values: `tlv1` `amount_msat`=65536 986 987 1. Valid stream: 0x01 04 01000000 988 2. Values: `tlv1` `amount_msat`=16777216 989 990 1. Valid stream: 0x01 05 0100000000 991 2. Values: `tlv1` `amount_msat`=4294967296 992 993 1. Valid stream: 0x01 06 010000000000 994 2. Values: `tlv1` `amount_msat`=1099511627776 995 996 1. Valid stream: 0x01 07 01000000000000 997 2. Values: `tlv1` `amount_msat`=281474976710656 998 999 1. Valid stream: 0x01 08 0100000000000000 1000 2. Values: `tlv1` `amount_msat`=72057594037927936 1001 1002 1. Valid stream: 0x02 08 0000000000000226 1003 2. Values: `tlv2` `scid`=0x0x550 1004 1005 1. Valid stream: 0x03 31 023da092f6980e58d2c037173180e9a465476026ee50f96695963e8efe436f54eb00000000000000010000000000000002 1006 2. Values: `tlv3` `node_id`=023da092f6980e58d2c037173180e9a465476026ee50f96695963e8efe436f54eb `amount_msat_1`=1 `amount_msat_2`=2 1007 1008 1. Valid stream: 0xfd00fe 02 0226 1009 2. Values: `tlv4` `cltv_delta`=550 1010 1011 ### TLV Stream Decoding Failure 1012 1013 Any appending of an invalid stream to a valid stream should trigger 1014 a decoding failure. 1015 1016 Any appending of a higher-numbered valid stream to a lower-numbered 1017 valid stream should not trigger a decoding failure. 1018 1019 In addition, the following TLV streams in namespace `n1` should 1020 trigger a decoding failure: 1021 1022 1. Invalid stream: 0x02 08 0000000000000226 01 01 2a 1023 2. Reason: valid TLV records but invalid ordering 1024 1025 1. Invalid stream: 0x02 08 0000000000000231 02 08 0000000000000451 1026 2. Reason: duplicate TLV type 1027 1028 1. Invalid stream: 0x1f 00 0f 01 2a 1029 2. Reason: valid (ignored) TLV records but invalid ordering 1030 1031 1. Invalid stream: 0x1f 00 1f 01 2a 1032 2. Reason: duplicate TLV type (ignored) 1033 1034 The following TLV stream in namespace `n2` should trigger a decoding 1035 failure: 1036 1037 1. Invalid stream: 0xffffffffffffffffff 00 00 00 1038 2. Reason: valid TLV records but invalid ordering 1039 1040 ## Appendix C: Message Extension 1041 1042 This section contains examples of valid and invalid extensions on the `init` 1043 message. The base `init` message (without extensions) for these examples is 1044 `0x001000000000` (all features turned off). 1045 1046 The following `init` messages are valid: 1047 1048 - `0x001000000000`: no extension provided 1049 - `0x001000000000c9012acb0104`: the extension contains two unknown _odd_ TLV records (with types `0xc9` and `0xcb`) 1050 1051 The following `init` messages are invalid: 1052 1053 - `0x00100000000001`: the extension is present but truncated 1054 - `0x001000000000ca012a`: the extension contains unknown _even_ TLV records (assuming that TLV type `0xca` is unknown) 1055 - `0x001000000000c90101c90102`: the extension TLV stream is invalid (duplicate TLV record type `0xc9`) 1056 1057 Note that when messages are signed, the _extension_ is part of the signed bytes. 1058 Nodes should store the _extension_ bytes even if they don't understand them to 1059 be able to correctly verify signatures. 1060 1061 ## Appendix D: Signed Integers Test Vectors 1062 1063 The following test vector show how signed integers (`s8`, `s16`, `s32` 1064 and `s64`) are encoded using big-endian two's complement. 1065 1066 ```json 1067 [ 1068 { 1069 "value": 0, 1070 "bytes": "00" 1071 }, 1072 { 1073 "value": 42, 1074 "bytes": "2a" 1075 }, 1076 { 1077 "value": -42, 1078 "bytes": "d6" 1079 }, 1080 { 1081 "value": 127, 1082 "bytes": "7f" 1083 }, 1084 { 1085 "value": -128, 1086 "bytes": "80" 1087 }, 1088 { 1089 "value": 128, 1090 "bytes": "0080" 1091 }, 1092 { 1093 "value": -129, 1094 "bytes": "ff7f" 1095 }, 1096 { 1097 "value": 15000, 1098 "bytes": "3a98" 1099 }, 1100 { 1101 "value": -15000, 1102 "bytes": "c568" 1103 }, 1104 { 1105 "value": 32767, 1106 "bytes": "7fff" 1107 }, 1108 { 1109 "value": -32768, 1110 "bytes": "8000" 1111 }, 1112 { 1113 "value": 32768, 1114 "bytes": "00008000" 1115 }, 1116 { 1117 "value": -32769, 1118 "bytes": "ffff7fff" 1119 }, 1120 { 1121 "value": 21000000, 1122 "bytes": "01406f40" 1123 }, 1124 { 1125 "value": -21000000, 1126 "bytes": "febf90c0" 1127 }, 1128 { 1129 "value": 2147483647, 1130 "bytes": "7fffffff" 1131 }, 1132 { 1133 "value": -2147483648, 1134 "bytes": "80000000" 1135 }, 1136 { 1137 "value": 2147483648, 1138 "bytes": "0000000080000000" 1139 }, 1140 { 1141 "value": -2147483649, 1142 "bytes": "ffffffff7fffffff" 1143 }, 1144 { 1145 "value": 500000000000, 1146 "bytes": "000000746a528800" 1147 }, 1148 { 1149 "value": -500000000000, 1150 "bytes": "ffffff8b95ad7800" 1151 }, 1152 { 1153 "value": 9223372036854775807, 1154 "bytes": "7fffffffffffffff" 1155 }, 1156 { 1157 "value": -9223372036854775808, 1158 "bytes": "8000000000000000" 1159 } 1160 ] 1161 ``` 1162 1163 ## Acknowledgments 1164 1165 [ TODO: (roasbeef); fin ] 1166 1167 ## References 1168 1169 1. <a id="reference-1">http://www.unicode.org/charts/PDF/U2600.pdf</a> 1170 1171 ## Authors 1172 1173 [ FIXME: Insert Author List ] 1174 1175  1176 <br> 1177 This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).