Sending offline transactions to mainnet is "flaky." - ethereum

Question

This is the code that I'm using to send signed transactions to mainnet programmatically:
import Web3 from 'web3'
import EthereumTx from 'ethereumjs-tx'
const web3 = new Web3(new Web3.providers.HttpProvider(INFURA_URL))
const Contract = new web3.eth.Contract(ABI, CONTRACT_ADDRESS)
const createItem = (name, price, nonce, callback) => {
console.log(`Nonce: ${nonce}. Create an Item with Name: ${name}, Price: ${price}`)
const data = Contract.methods.createItem(name, price).encodeABI()
const tx = new EthereumTx({
nonce: nonce,
gasPrice: web3.utils.toHex(web3.utils.toWei('4', 'gwei')),
gasLimit: 400000,
to: CONTRACT_ADDRESS,
value: 0,
data: data,
})
tx.sign(new Buffer(MAINNET_PRIVATE_KEY, 'hex'))
const raw = '0x' + tx.serialize().toString('hex')
web3.eth.sendSignedTransaction(raw, callback)
}
export default createItem
I have to mass create (i.e. populate) items in my contract, and I want to do it programatically. However, while the code works well in ropsten, it fails to send all the transactions in mainnet; it only sends the first few transactions and doesn't send the rest. The errors are not helpful because this error is usually guaranteed to occur:
Unhandled rejection Error: Transaction was not mined within 50 blocks, please make sure your transaction was properly sent. Be aware that it might still be mined!
I wonder how other people do when they have to send a lot of transactions to Ethereum mainnet today. Is there anything I'm doing wrong?

Answer 1

You basically cannot reliably do what you guys are trying to do. Here's a set of problems that arises:
Transactions only succeed in nonce order. If an early nonce is pending (or worse, went missing), the other transaction will pend until the early nonce has been consumed (or it gets removed from the mempool).
There's no hard rule for when a transaction will drop from the mempool. This is scary when you've made a nonce error or an intermediary nonce has not reached the network for some reason because you don't know what is going to happen when you finally post that nonce.
Many transactions with the same nonce can be sent. They are very likely to be selected by gas price (because miners are incentivized to do exactly that). One useful trick when something strange has happened is to clear out your nonces by sending a bunch of high gas price, zero value transactions. You might call this an increment method. Remember this comes with a cost.
A lot of tools do one of two things to handle nonces: a live read from getTransactionCount() or a read from getTransactionCount() followed by incrementing for each additional transaction you send. Neither of these work reliably: for the first, transactions are sometimes pending but not visible in the pool yet. This seems to especially happen if gas price is below safemin, but I'm not entirely sure what is happening here. For the second, if any other system sends a transaction with the same address, it will not work.
So, how do we work around this?
A smart contract is a straightforward way to reduce a transaction from many sender nonces to few sender nonces. Write a contract that sends all your different transactions, then send the budget to that contract. This is a relatively high cost (in terms of time/effort/expertise) way to solve the problem.
Just do it anyway, batch style. When I've had to send many transactions manually, I've batched them in to sets of 10 or so and gone for it. Incrementing the nonce manually each time (because the transaction is usually not on the network yet) and then waiting sufficiently long for all the transactions to confirm. Don't rely on pending transactions on Etherscan or similar to determine whether this is working, as things often vanish unpredictably from this level. Never reuse a nonce for a different transaction that isn't a 0ing high gas transaction - you will screw it up and you'll end up sending the same transaction twice by mistake.
Serialize. One-by-one you post a transaction, wait for it to confirm, increment your nonce. This is probably the best of the easy-to-implement automated solutions. It will not fail. It might buffer forever if you have a constant stream of transactions. It also assures you can never have more than one transaction per block, limiting your throughput to 4 a minute or so.
Fire and retry. This is a little sketchy because it involves reusing nonces for different transactions. Send all (or some large batch) of your transactions to the network. If any fail, restart from the failure nonce and send again. There's possibly a more intelligent solution where you just try to swap out the missing nonce. You'll need to be very careful you never send a transaction that is secretly in the pending-but-not-visible pool.
New address for every transaction. A buffering step of distributing funds to your own addresses ensures you never screw it up for other people. It does double your transaction time and cost though.
I think some variant of fire and retry is what most of the big services (like pools and exchanges) do. Some of these can be improved by splitting the budget over a few addresses as available (reducing the collision frequency).