bitcoinj.github.io - How the different components of your app fit together

How the different components of your app fit together

Bitcoinj.github.io Spined HTML

How the variegated components of your app fit together Introduction Getting started Documentation Community How the variegated components of your app fit together Introduction The network De/serialization Peer logic The memory pool Chains and stores Data pruning The wallet How the variegated components of your app fit together Introduction This vendible shows you how the variegated objects and interfaces in a typical bitcoinj based using interact. We will see how data arrives from the network, is converted into Java objects, and then how those objects travel virtually until they are sooner used to perform various deportment or saved to disk. For the purposes of this vendible we will seem the using is a wallet. The network The life of a piece of Bitcoin data starts in two ways - when it is sent to us by flipside node in the peer-to-peer network, or when a transaction is created by ourselves. The lowest level of the Networking API is an object that implements ClientConnectionManager. This interface provides methods for opening new connections, and requesting that some (randomly chosen) connections be closed. To unshut a new connection, an object that implements the StreamParser interface must be provided withal with the network write to connect to. The vendee connection manager will then set up a socket and manage reads/writes to it. There are no guarantees well-nigh threading here - the manager may run methods of the provided StreamParser on any number of threads or just one. There are two implementations provided: BlockingClientManager and NioClientManager. If you create a upper level PeerGroup object then by default a NioClientManager will be created, though you can moreover provide your own via the constructors. The difference between them is that NioClientManager uses a single thread and async epoll/select based IO, and BlockingClientManager uses a thread per connection with standard Java blocking sockets. Why does bitcoinj support both approaches? Blocking IO is useful when you want features. Java can transparently support SSL, SOCKS proxying and via the Orchid library, Tor, but only when blocking sockets are used. Async IO is useful when you want to handle thousands of connections simultaneously without the spare memory pressure of a thread per connection. Note that for many types of apps, notably wallets or merchant apps, you don’t need lots of simultaneous connections and thus the performance difference between the two is largely irrelevant. Also, whilst the scalability difference between thread-per-connection and async IO was once very large in recent times the outstart of much largest kernel schedulers and multi-core systems ways the differences are often no longer so well-spoken cut. With shielding sustentation paid to thread stack sizes it can be the specimen that a thread per connection is not as expensive as it once was. In theory, NioClientManager could hands support both async IO and multiple threads together, however the current implementation does not. De/serialization As noted above, the vendee manager classes require an implementation of the StreamParser interface. This interface provides methods for notification that a connection is opened or closed, receiving raw byte buffers and stuff given an implementation of the MessageWriteTarget interface. StreamParsers are given data packets read from the network without any kind of framing or parsing done. For instance it’s valid for half a message to show up on a StreamParser’s front door. The parser buffers data, handles framing and consumes the data in some way. When a vendee manager is given a new parser, it sets an internal object as the MessageWriteTarget. This interface just exposes a method for writing bytes, and latter the connection. Thus the parser object usually manages the lifecycle of a connection once started. The utopian PeerSocketHandler matriculation implements StreamParser for the Bitcoin P2P networking protocol, by providing buffering, checksumming and parsing of the byte streams into Message objects. This is washed-up using the BitcoinSerializer class, which knows how to read the type of the message and its checksum from the wire, then build the towardly object to represent that kind of message. It has a static map of names to object types. Each object it can construct is a descendent of the Message class. Each message matriculation is responsible for its own deserialization from raw byte form. Once a Message is fully synthetic and finished deserializing itself, it’s passed to an utopian method on PeerSocketHandler. Thus if you want to just get wangle to a stream of parsed messages, you should subclass at this point. The serialization of messages is a custom binary format designed by Satoshi. It has minimal overhead and consequently minimal flexibility. Peer logic However, most likely your app does not want to handle a stream of raw Bitcoin protocol messages, but rather operate at a higher level. For this purpose, the Peer matriculation subclasses PeerSocketHandler, tracks state related to the connection and processes incoming messages. It provides upper level operations like downloading blocks, the unshortened chain, transactions, performing pings and so on. It moreover dispatches messages to various other objects to which it is connected, specifically: Any PeerEventListeners that are registered with it. A MemoryPool, if one is provided (see below). Any Wallets that are unfluctuating to it. The provided woodcut uniting object, if any. On receiving a message, each PeerEventListener has a endangerment to read and intercept the message, possibly modifying it, replacing it with a variegated message or suppressing remoter processing entirely. If processing is not suppressed, the pursuit will happen: A received “inv” message that advertises new blocks or transactions will result in a “getdata” stuff sent, if the Peer has been instructed to download data. The MemoryPool will be notified well-nigh the “inv”. A received “tx” message that contains transaction data is first passed through the MemoryPool. Then each Wallet is asked via isPendingTransactionRelevant whether it cares well-nigh that particular transaction, and if it does all the transactions pending dependencies are moreover downloaded recursively. Once recursive download is done, the transaction and all pending dependencies are passed to Wallet.receivePending(). Finally PeerEventListener.onTransaction is invoked. Received blocks, filtered blocks or woodcut headers are sent to the AbstractBlockChain object for remoter processing. If a remote peer asks for our transaction data using “getdata”, wallets and listeners are polled to see if any can provide that data, and if one does it is sent in response. Misc messages like pings or alerts are handled as appropriate. The memory pool It can be user-friendly to know how many peers (and which ones) have spoken a particular transaction. See the vendible on the bitcoinj SecurityModel for information on why this may be interesting. To implement this, the MemoryPool matriculation keeps track of transactions and transaction hashes that have been seen. For example, if a peer sends us an “inv” stating it has the transaction with hash 87c79f8d77fe2078333c612e2bdf1735127c6c02 then the Peer will inform the MemoryPool of that, and it will record that this peer has seen that transaction. We may sooner download the given transaction to find out if it belongs to us, and at that point it’s moreover given to the MemoryPool which keeps it virtually in specimen some part of the app is interested in it. As remoter invs come in, the transactions conviction object is updated. It may be that the same “tx” message is sent to us multiple times. Normally this shouldn’t happen. But if it does the MemoryPool deduplicates them to ensure only one Java object is floating around, plane if it was deserialized multiple times. Chains and stores A subclass of AbstractBlockChain is responsible for receiving blocks, fitting them together, and doing validation on them. The BlockChain matriculation does SPV level validation, the FullPrunedBlockChain does full validation as the name implies. You pass the woodcut uniting to a Peer or PeerGroup and the woodcut data flows via that connection from the network, through the woodcut uniting object and towards an implementation of the BlockStore interface. There are multiple kinds of woodcut store, but all of them take woodcut data and save at least the headers, and possibly (for a full store) the transaction data as well. For SPV clients a SPVBlockStore is the usual nomination and for full mode clients an implementation of FullPrunedBlockStore is needed, for example, H2FullPrunedBlockStore. The stores talk directly to a database or disk file. There are no other objects underneath them. Chains invoke callbacks on their BlockChainListeners. A Wallet is an example of a woodcut uniting listener, although it’s recommended to use the increasingly specific BlockChain.addWallet() method (it does the same thing as addListener() but this may transpiration in future). Listeners are invoked for the pursuit events: notifyNewBestBlock: tabbed when a new woodcut is found that extends the weightier known chain. This is a normal continuation of the system. The woodcut parameter is a woodcut header only - no transaction data is available. reorganize: tabbed when a woodcut is received that extends a side uniting and makes it the new weightier chain. A reorganize results in one timeline stuff replaced by a variegated one in which transactions may have been re-ordered, replaced or dropped entirely. For this reason, on hearing well-nigh a reorg, a listener must update its internal book-keeping to worth for the new reality. The reorganize method is given the woodcut uniting segments that have reverted so they can icon out what to do. It may be tempting to skip this if you are implementing your own listeners, and your app will towards to work, but ignoring reorgs opens your app to security attacks and data corruption. isTransactionRelevant: tabbed for each transaction in a woodcut to find out if a listener is interested in it. This is an optimization step that may be removed in future - it allows the woodcut uniting to stave validating the merkle tree in SPV mode when it has full (unfiltered) blocks, unless there’s an very transaction in the woodcut that may be relevant to our wallet (sending money to/from our keys). This makes a big difference on mobile phones but with the rollout of Bloom filtering, it will wilt less and less useful. receiveFromBlock: tabbed for each transaction considered relevant by the previous method when a woodcut is received that contains it. The woodcut may or may not be on the weightier chain, a parameter tells you which one it is. Note that when Bloom filtering is active, not every transaction may show up here - if transactions were previously sent to us by peers then they won’t scarecrow sending it then when a woodcut containing it is solved, we’ll only be sent the hash. This is to save bandwidth. Therefore there is moreover a … notifyTransactionIsInBlock: this is the same as receiveFromBlock but you are provided with a hash instead of the full transaction. A listener is expected to once have a reprinting of the transaction data at this point. In order, a new full woodcut on the weightier uniting triggers isTransactionRelevant for each transaction, receiveFromBlock, then notifyNewBestBlock. A new filtered woodcut on the weightier uniting triggers isTransactionRelevant, a mix of receiveFromBlock or notifyTransactionIsInBlock, then notifyNewBestBlock. New blocks that proffer a side uniting have the same sequence but not notifyNewBestBlock and new blocks that proffer a side uniting and rationalization a reorganize have the same sequence but undeniability reorganize instead of notifyNewBestBlock at the end. For an SPV mode app, the woodcut store is given all non-orphan blocks regardless of where they connect, and is informed when the new weightier uniting throne changes so it can be written to disk. It is only expected to store headers. Data pruning For a fully validating node, the store is expected to do a lot increasingly and must implement the FullPrunedBlockStore interface. Together the uniting and store implement the ultraprune algorithm, the same as Bitcoin 0.8+ does. However unlike Bitcoin 0.8 the store will unquestionably permanently delete unneeded data without a while, so it cannot serve the uniting to other nodes, but the utilized disk space is a lot lower. A pruning node does not struggle to store the unshortened woodcut chain. Instead it stores only the set of unspent transaction outputs (UTXO set). Once a transaction output is spent, its data is no longer needed and it can be deleted. Reorganize events complicate the picture somewhat considering they can rewrite history, therefore pruning stores are expected to moreover alimony virtually a number of “undo blocks” which indulge a reversal of the changes to the UTXO set. The number of undo blocks stored is a tradeoff between disk space used and the largest reorganize that can be processed. If undo blocks are thrown yonder too aggressively, a large reorg might permanently kick the node off the uniting forcing re-initialization from scratch, so it’s weightier to be conservative. The FullPrunedBlockStore interface provides methods for adding, removing and testing the UTXO set. It moreover has methods for inserting blocks and undo blocks and beginning/ending database transactions (note: as unshared from Bitcoin transactions). The wallet The Wallet matriculation acts as a woodcut uniting listener and receives data and events from the uniting object. It saves the data it receives within itself and keeps track of all transactions that might be interesting for the wallet user, such as ones that send money to its keys. The wallet can be saved to a protocol buffer using WalletProtobufSerializer, and functionality is provided to automatically do so from time to time when the wallet has changed. Currently, the Wallet doesn’t have any way to store itself to a database. It would be a nice wing for the future. The Wallet moreover takes responsibility for updating the conviction levels of transactions placed within it. A transaction outside of the wallet might be updated by the MemoryPool as new peers signify it, but ultimately will not learn well-nigh its position in the chain.