Subjective Billing and Lost Transactions: Background
The EOS network is losing transactions.
Specifically, nodes on the EOS network occasionally fail to pass a transaction across the network to the currently active block producer. Subjective billing is often the cause of this issue.
This article explores the subjective billing feature, why it exists, and the issues with subjective billing that ENF-sponsored working groups identified. We also include a quick reference for preventing these issues. In the future, we will explore existing fixes in more detail, and look at additional features in the Mandel 3.1 release.
What Is “Subjective Billing,” and Why Was It Introduced?
Subjective billing is a feature in EOSIO that lets nodes bill account resources locally in their own node without sharing the billing with the rest of the network. It has, since its introduction, reduced node CPU use by almost 90%, but it also sometimes results in persistently failing transactions or lost transactions. When smart contract code running on the blockchain uses a “check” function, like assert() or check(), to verify data, it can often lead to subjective billing issues.
It is helpful to understand the architecture of a transaction’s lifecycle in order to understand the specifics of these issues.
In EOSIO, chains use peer-to-peer (P2P) network connections between nodes to propagate transactions through the blockchain network. In this P2P network, node configuration settings and network latency influence the paths transactions take as they flow between nodes.
As illustrated in fig. 1, an EOSIO transaction’s lifecycle begins when a user submits a transaction to an API node (1.1). The API node passes the transaction internally to its P2P protocol (1.2), which passes it to another node via P2P, which passes it to another node, and so on (1.3) until the transaction eventually reaches the Block Producer (BP) that is producing the current block (1.4), after which the active BP includes the transaction in the block (1.5) and bills the account for resources. Each node in the network independently determines whether to pass the transaction along to other connected nodes or fail the transaction due to failed internal checks or resource overutilization.
Unlike some other blockchains, EOSIO chains natively do not bill accounts for the resources used when a validly signed transaction is unsuccessful for one reason or another. Subjective billing allows individual nodes to do just that.
“Subjective” means the billing applies only to an individual instance of the EOSIO application nodeos running on a node operator’s server. “Billing” refers to charging for the node resources used in processing a transaction. In short, subjective billing allows an individual node on the network to bill accounts for their failed transactions, keeping track locally, without including the billing in a produced block. When a transaction arrives at a node with subjective billing enabled, the node verifies the signature and identifies which account sent it. If the account is over its subjective billing limit, the node drops the transaction.
Subjective billing was introduced after an increase in node resource usage in late 2019 overwhelmed nodes. Specifically, nodes began receiving numerous incoming transactions that passed signature validation, only to fail internal checks during execution, such as assert failures. These failed transactions overwhelmed block producers (BPs) and API nodes, which still use their server resources when validating transactions that fail. At the time, BP nodes were so busy expending node resources to deal with the failing transactions that many valid transactions never made it into a block, and some blocks were completely empty.
The opt-in subjective billing tool reduces the load on these nodes by reducing the number of failed transactions the node must process. Fig. 2 illustrates resource charges from a node with subjective billing enabled.
Subjective billing is applied when a transaction fails (2.1) after signature validation. With small amounts of subjective billing, transactions often still succeed. For successful transactions (2.2), the temporary subjective billing of a new transaction is replaced with objective billing when it’s recorded on the blockchain.
If subjective billing is too much for account resources (2.3), the transaction will fail. For failed transactions that never make it into a block, the node charges subjective billing, but the on-chain billing does not increase. Each subsequently failed transaction (2.4) routed through a node increases subjective billing on that node.
Accounts must have adequate resources to cover subjective billing along with the new transaction (2.5). This temporarily increases the account resources required for all the account’s transactions routed through that node until the subjective billing decays over 24 hours (2.6).
Depending on a node’s performance, some may pass a transaction to the next node, while others might drop the transaction for exceeding the account’s allowed resources.
The default subjective billing decay parameter is set to 24 hours, much like regular resource billing and the PowerUp mechanism. New features in Mandel 3.1 will allow node operators to set their node’s subjective billing decay time parameter upon initialization. This option will adjust the time it takes to zero out an account’s subjective bill.
Many block producers and some API nodes have enabled subjective billing. This adoption has dropped node CPU usage by nearly 90%.
Challenges with Subjective Billing
While subjective billing is effective at preventing empty blocks and reducing the load on nodes, it has side effects that can lead to frustration and confusion among users.
A node subjectively billing an account will add the billing as an additional resource charge to any incoming transaction from the account. The next time the node receives a transaction from that account, the account must have enough extra resources to cover its previous failed transaction before covering the new transaction. If the account does not have enough extra resources for the previously failed transaction, the node will validate the signature and then drop the transaction without executing the rest of it.
As shown in fig. 3, some users receive repeated error messages when submitting transactions, despite block explorers showing adequate resources. This happens when, without their knowledge, the user has connected to an API node (3.1) that has subjective billing applied to their account. Their transactions fail, but at least the user knows something is wrong when the node returns an error message (3.2).
What’s worse is the situation where there is no error message, as shown in fig. 4. Some users successfully submit transactions (4.1) to an API node (4.2) but never see the transactions in any block.
This often happens when there are enough subjectively billing nodes on the relay path (4.3) to the active block producer node (4.4) that it prevents propagation through the network. These problems can persist for up to 24 hours as the billing balance decays.
Continually trying to re-submit failing transactions further increases the subjective billing balance and exacerbates the problem, as shown in fig. 2 (2.4).
Lost Transaction Tools
Because subjective billing is scoped locally to each node, and there is no protocol for sharing subjective billing data, it is effectively impossible for users to find out why their transactions are disappearing. Individual node operators may build and share their own subjective billing data-sharing tools to help with troubleshooting, but without a chain-wide solution, interoperability is limited.
One example of a prototype data-sharing tool was built by EOS Nation to look up failed transactions, but the siloing of data limits the tool to looking up transactions processed by their own nodes. You can find it at https://eos.api.eosnation.io/transaction_lookup. Widespread sharing of the kind of data feeding this tool could allow users and apps to identify why their transactions are failing, helping to pinpoint specific bugs and diagnose other issues with subjective billing.
The release of Mandel includes fixes for many of these issues, as soon as block producers, node operators, wallets, and dApps begin using its new features. However, development and adoption take time. Fortunately, developers and operators can integrate existing mitigation practices into their workflows until applications and nodes adopt the new updates.
In the next few weeks, we will outline the current recommendations for node operators and smart contract developers. We’ll also highlight some of the features slated for release in future versions of Mandel, including several transaction lifecycle improvements proposed in the API+ Blue Paper.
For now, we are providing the following as a quick reference that operators can use to test potential solutions. We will explore these suggestions more thoroughly in the next article.
| Quick References for Fixing Subjective Billing Issues and Lost Transactions |
|---|
Each node operator can determine which accounts are unlikely to send too many failed transactions, and can disable subjective billing on those accounts using: disable-subjective-account-billing = <accountname> An example might be Greymass Fuel, which a node operator might decide has enough rate limiting or Sybil resistance to be “trusted.” The operator would disable subjective billing for Greymass Fuel using: e.g. disable-subjective-account-billing = greymassfuel |
Due to known issues with the implementation in nodeos 2.x, nodes using the disable_subjective_api_billing and disable_subjective_p2p_billing optional parameters should ensure both are set to the same Boolean value, both true or both false. The Mandel 3.1 release is expected to resolve these issues. |
Alternatively, operators can use the disable-subjective-billing = true option to turn off subjective billing entirely. |
Cloud nodes can use database-map-mode = heap and mount swap partitions as tmpfs to reduce disk reads and resource usage. |
Assert() and check() functions can cause transaction failure and trigger subjective billing. Developers should be aware of this, and can assert or check earlier in a contract’s execution to reduce the billing applied. As long as the lack of an error message doesn’t affect user experience, some contracts may also benefit from replacing some asserts and checks with return statements to ensure their transactions succeed and are billed objectively on-chain. |
API+ Working Group Education Series
This series of articles will outline the lifecycle of an EOSIO transaction, highlighting the subjective billing feature and the lost transactions it can cause. First, we learned all about subjective billing. Second, we will outline best practices for current smart contract developers and node operators to avoid problems. Last, we will take a look at a few new features in Mandel that will help further mitigate these issues.
EOS Network
The EOS Network is a 3rd generation blockchain platform powered by the EOS VM, a low-latency, highly performant, and extensible WebAssembly engine for deterministic execution of near feeless transactions; purpose-built for enabling optimal web3 user and developer experiences. EOS is the flagship blockchain and financial center of the EOSIO protocol, serving as the driving force behind multi-chain collaboration and public goods funding for tools and infrastructure through the EOS Network Foundation (ENF).
EOS Network Foundation
The EOS Network Foundation (ENF) is a not-for-profit organization that coordinates financial and non-financial support to encourage the growth and development of the EOS Network. The ENF is the hub of the EOS Network, harnessing the power of decentralization as a force for positive global change to chart a coordinated future for EOS.
