Now 99% of all interactions with the blockchain occur through Infuria or Alchemy (MetaMask - API Infuria).
Nobody raises their geth nodes.
Because of this, most people are skeptical about the word "decentralization", since the application still has a point of failure.
Why is it impossible to send transactions directly from the browser to the validator? What prevents this?
After all, this is the last obstacle before decentralization. If the browser/extensions stored hundreds of addresses of mining pools to which you can send a transaction, then such an application is almost fail-safe.
Generally, a signed transaction is sent from a wallet software to a node (in the peer-to-peer Ethereum network) that broadcasts it to the rest of the network as a "transaction waiting to be mined" (i.e. placed in a mempool).
Miners usually take transactions from the mempool and place them in blocks.
It is technically possible for a miner to accept a transaction from another source (or create and sign it themselves), and place it in a block.
But it comes with an inconvenience for the transaction sender - they need to wait until this specific miner mines a block containing their transaction. If they sent the transaction to the mempool instead, any miner could have picked it up and include in their block. And there is currently no standardized way of sending a transaction to the miner directly - so each might have a different channel and different rules.
So to answer your question:
Why can't transactions be sent directly to validators/mining pools?
They can. But it's just faster (for the transaction sender) to use the mempool and let the transaction be mined by anyone, instead of waiting for one specific mining pool to mine a block.
Related
Is there anyway that i can limit the calling of apply method to just 1 in the Transaction Processor? By default it is called twice.
I guess your question is based on the log traces you see.
Short answer: apply method also the core business logic in your transaction family is executed once for a input transaction. There is a different reason for you to see the logs appear twice. Well, in reality transaction execution happens and state transitions are defined with respect to the context. Read the long answer for detailed understanding.
Long answer: If you're observing logs, then you may need to go a little deeper into the way Hyperledger Sawtooth works. Let's get started.
Flow of events (at very high level):
Client sends the transaction embed in a batches.
Validator adds all the transactions in the pending queue.
Based on the consensus engine's request, the validator will start creating the block.
For the block creation, a current state context information is passed along with the transaction request to the Transaction Processor. Eventually send to the right Transaction Family's apply method.
The apply method's result either success or failure is recorded. The transaction is removed from the pending queue if is invalid or it is added to the block.
If the response of the apply method is internal error then that is resubmitted.
If a transaction is added to the block. Depending on the consensus algorithm, the created block is broadcasted to all the nodes.
Every node executes the transactions in the arriving block. The node that created the block will also execute. This probably is what you're talking about.
This is the code that I'm using to send signed transactions to mainnet programmatically:
import Web3 from 'web3'
import EthereumTx from 'ethereumjs-tx'
const web3 = new Web3(new Web3.providers.HttpProvider(INFURA_URL))
const Contract = new web3.eth.Contract(ABI, CONTRACT_ADDRESS)
const createItem = (name, price, nonce, callback) => {
console.log(`Nonce: ${nonce}. Create an Item with Name: ${name}, Price: ${price}`)
const data = Contract.methods.createItem(name, price).encodeABI()
const tx = new EthereumTx({
nonce: nonce,
gasPrice: web3.utils.toHex(web3.utils.toWei('4', 'gwei')),
gasLimit: 400000,
to: CONTRACT_ADDRESS,
value: 0,
data: data,
})
tx.sign(new Buffer(MAINNET_PRIVATE_KEY, 'hex'))
const raw = '0x' + tx.serialize().toString('hex')
web3.eth.sendSignedTransaction(raw, callback)
}
export default createItem
I have to mass create (i.e. populate) items in my contract, and I want to do it programatically. However, while the code works well in ropsten, it fails to send all the transactions in mainnet; it only sends the first few transactions and doesn't send the rest. The errors are not helpful because this error is usually guaranteed to occur:
Unhandled rejection Error: Transaction was not mined within 50 blocks, please make sure your transaction was properly sent. Be aware that it might still be mined!
I wonder how other people do when they have to send a lot of transactions to Ethereum mainnet today. Is there anything I'm doing wrong?
You basically cannot reliably do what you guys are trying to do. Here's a set of problems that arises:
Transactions only succeed in nonce order. If an early nonce is pending (or worse, went missing), the other transaction will pend until the early nonce has been consumed (or it gets removed from the mempool).
There's no hard rule for when a transaction will drop from the mempool. This is scary when you've made a nonce error or an intermediary nonce has not reached the network for some reason because you don't know what is going to happen when you finally post that nonce.
Many transactions with the same nonce can be sent. They are very likely to be selected by gas price (because miners are incentivized to do exactly that). One useful trick when something strange has happened is to clear out your nonces by sending a bunch of high gas price, zero value transactions. You might call this an increment method. Remember this comes with a cost.
A lot of tools do one of two things to handle nonces: a live read from getTransactionCount() or a read from getTransactionCount() followed by incrementing for each additional transaction you send. Neither of these work reliably: for the first, transactions are sometimes pending but not visible in the pool yet. This seems to especially happen if gas price is below safemin, but I'm not entirely sure what is happening here. For the second, if any other system sends a transaction with the same address, it will not work.
So, how do we work around this?
A smart contract is a straightforward way to reduce a transaction from many sender nonces to few sender nonces. Write a contract that sends all your different transactions, then send the budget to that contract. This is a relatively high cost (in terms of time/effort/expertise) way to solve the problem.
Just do it anyway, batch style. When I've had to send many transactions manually, I've batched them in to sets of 10 or so and gone for it. Incrementing the nonce manually each time (because the transaction is usually not on the network yet) and then waiting sufficiently long for all the transactions to confirm. Don't rely on pending transactions on Etherscan or similar to determine whether this is working, as things often vanish unpredictably from this level. Never reuse a nonce for a different transaction that isn't a 0ing high gas transaction - you will screw it up and you'll end up sending the same transaction twice by mistake.
Serialize. One-by-one you post a transaction, wait for it to confirm, increment your nonce. This is probably the best of the easy-to-implement automated solutions. It will not fail. It might buffer forever if you have a constant stream of transactions. It also assures you can never have more than one transaction per block, limiting your throughput to 4 a minute or so.
Fire and retry. This is a little sketchy because it involves reusing nonces for different transactions. Send all (or some large batch) of your transactions to the network. If any fail, restart from the failure nonce and send again. There's possibly a more intelligent solution where you just try to swap out the missing nonce. You'll need to be very careful you never send a transaction that is secretly in the pending-but-not-visible pool.
New address for every transaction. A buffering step of distributing funds to your own addresses ensures you never screw it up for other people. It does double your transaction time and cost though.
I think some variant of fire and retry is what most of the big services (like pools and exchanges) do. Some of these can be improved by splitting the budget over a few addresses as available (reducing the collision frequency).
I was wondering how ethereum blockchain works compared with the bitcoin blockchain.
I know that, in bitcoin, all nodes compete to mine blocks (and put to public transaction into them and thus make bitcoin as transaction processing fee), and that all nodes compete for the next block at one time with an equal chance of mining it.
But in ethereum, where you want a network of distributed apps that get executed according to the gas price they are willing to pay (and starting gas), are all nodes competing for the next block at one given time? Wouldn't this be a waste of computation?
Yes, all the nodes do compete for (pretty much) the same blocks, and yes - they do execute all the code in a block, even if this block is not going to be successfully mined.
Don't think of it as "waste," but rather as a mechanism to ensure proof of work.
In short, yes, there is a lot of wasted computation.
Ethereum's mining process is almost the same as bitcoin’s.
For each block of transactions, miners will run the block’s unique header metadata (including timestamp and software version) through a hash function. If the miner finds a hash that matches the current target, the miner will be awarded ether and broadcast the block across the network for each node to validate and add to their own copy of the ledger. If miner B finds the hash, miner A will stop work on the current block and repeat the process for the next block.
We have a requirement for an API, which allows asynchronous updates via a MSMQ message queue, that I'm putting together which will allow the developer consuming the API to specify different retry policies per message. So a high priority client system, e.g. for sales will submit all messages with 5 delivery attempts (retries) and 15 minutes between each attempt, whereas a low priority client system, e.g. back-end mail shot system will allow users to update their marketing preferences, submitting messages with 3 retries and an hour between each attempt.
Is there a way in the System.Messaging MSMQ (version 3 or 4) implementation to specify number of retries, retry delay and things like whether messages are sent to a dead letter queue or just deleted? (and if so, how?)
I would be open to using other messaging frameworks if they fulfilled this requirement.
Is there a way in the System.Messaging MSMQ (version 3 or 4) implementation to specify number of retries
Depending on which operating system/msmq version you're using, specifying retry semantics is highly sophisticated in WCF. The following is for Windows 2008 and MSMQ4 using a transactional queue.
The main setting on the binding is called MaxRetryCycles. One retry cycle is an attempt to successfully read a message from a queue and process it inside the handling method. This "attempt" can actually be made up of multiple attempts, as defined by the msmq binding property ReceiveRetryCount. ReceiveRetryCount is the number of times an application will try to read the message and process it before rolling back the de-queue transaction. This marks the end of one retry cycle.
You can also introduce a delay in between cycles with the RetryCycleDelay property.
A more complicated consideration is what to do with the messages which fail even after multiple retry cycles.
allow the developer consuming the API to specify different retry policies per message
I am not sure how you could do this with MSMQ - as far as I'm aware it's only possible to set retry semantics on a per-endpoint basis. If you're using transactions then you can't even allow API users to set the priority of the messages being sent (transactional queues guarantee delivery in order).
The only thing you could do is host a another instance of your API as high-priority and one for low priority. These could be hosted on different environments, and this has the added benefit that low priority messages won't be competing for system resources with high priority messages.
Hope this helps.
I have studied Message Queues System in my class but I still don't get it how these Message Queues System work in real time scenarios? Is there any tutorial which can help me to get the complete picture?
Can someone explain me how these systems work?
An example: My thread or process can send a message to your message queue, and having sent it, my code goes on to do something else. Your code, when it gets around to it, reads the next message from the message queue, and then decides what to do about that message. Message queues avoid needing to have a critical section or mutex shared between the two threads, or processes. The underlying message queue layer itself takes care of making sure that messages get into the queue without race conditions affecting the integrity of the queue.
Message queues can be used for both one-way and two-way, asynchronous messaging. For one-way use, my thread can use it to keep your thread appraised of key events in my thread, without acknowledgement back from your thread. For two-way use, after my thread sends a message to your thread, your thread may need to send data back to my thread via my message queue.
The message queue layer uses lower level synchronization schemes to insure that no two writers to the queue can write at the same time. It insures that all writes to the queue are atomic. It also insures that a reader of the queue cannot read a partially written message from the queue.
Most message queue APIs also offer support for reading messages from the queue based on a filter that you designate. Say for instance that you consider messages from a time critical thread to be more important that other messages. You can each time you check your queue for messages, first check for messages from the critical thread, and service those messages first. Your thread would then go onto to process the rest of the messages as normal, provided no more messages from the critical thread are found.
A C tutorial of the UNIX message queues
That's a complex topic but to put it simply:
Message Queues are one of the best ways, if not the best, to
implement distributed systems.
Now you might ask, what is a distributed system? It is an integrated system that spans multiple machines, clients or nodes which execute their tasks in parallel in a non-disruptive way. A distributed system should be robust enough to continue to operate when one or more nodes fail, stop working, lag or are taken down for maintenance.
Then you might ask, what is a message queue? It is a message-oriented middleware that enables the development of a distributed system by using asynchronous messages for inter-node communication through the network.
And finally you might ask, what is all that good for? This is good for implementing applications with a lot of moving parts called nodes which needs real-time monitoring and real-time reaction capabilities. To summarize they provide: parallelism (nodes can truly run in parallel), tight integration (all nodes see the same messages in the same order), decoupling (nodes can evolve independently), failover/redundancy (when a node fails, another one can be running and building state to take over immediately), scalability/load balancing (just add more nodes), elasticity (nodes can lag during activity peaks without affecting the system as a whole) and resiliency (nodes can fail / stop working without taking the whole system down).
Check this article which discusses a message queue infrastructure in detail.