I'm trying to listen for all the transactions an address does. I'm using my own geth light node and subscribed to pendingTransactions events. My doubt is if I could miss a transaction this way and if it would be better to listen for newHeads and rebuild (get block number out of it and then get the transactions) out of it.
Related
I am thinking of one thing for the transaction history data. Whether there is an API call, maybe RPC call or REST call, that we can query target wallet address transaction history? I am trying for making a wallet, I think if we can show users their transaction history, it is a good feature for the user experience.
For this question, there are maybe some ways:
Call Bcsscan or Ethereumscan API, directly.
Start your own server, subscribe to all events, and store them to DB and query events in DB.
Any other good ideas for this feature.
For method 2) Currently to extract the full address interaction history you need to walk through all blockchain blocks and then all transactions in those using EVM tracing.
For method 1) You can also get some details over Etherscan API, but please note that this is not an open-source product and you need to contact their paid product support to understand what data is available over their API and what are the API limitations.
Using OpenEthereum, which is the json rpc call allowing to do it?
I found the parity_pubsub module but I ve no idea of the parameter for getting not yet confirmed transactions.
I also found https://web3js.readthedocs.io/en/v1.2.11/web3-eth-subscribe.html#subscribe-pendingtransactions in the web3 module but it doesn t allows to filter transactions and the documentation doesn t explains how to use the full web3 api along OpenEthereum.
The purpose
The aim is to frontrun transactions for arbitrage by always maintinning the highest gas price until the next block is mined.
This involve detecting competiting transactions (those targetting the same Ethereum address as mine and with a higher gas price than me) with the lowest latency as possible in order to replace them as soon as possible (either for abortting the trade or changing the gas price again). So this exclude constantly polling for the answer as the requirement is to have the lowest number of context switches.
According to Etherscan i have 7.5 Ether, but when I execute eth.getBalance(eth.accounts[0]) inside the Javascript console it returns always "0"
this is how I am connecting geth to rinkeby (is running for more than 24 hours)
geth --rinkeby
this is the status of the sync
λ geth --rinkeby attach ipc:\\.\pipe\geth.ipc
Welcome to the Geth JavaScript console!
instance: Geth/v1.9.9-stable-01744997/windows-amd64/go1.13.4
coinbase: 0x7f9153e8fe06c4b3eb10e8457c20d0559921de5c
at block: 0 (Wed, 12 Apr 2017 16:59:06 CEST)
datadir: C:\Users\andre_000\AppData\Local\Ethereum\rinkeby
modules: admin:1.0 clique:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0
> eth.syncing
{
currentBlock: 5746334,
highestBlock: 5746402,
knownStates: 32641057,
pulledStates: 32636964,
startingBlock: 5746304
}
> eth.getBalance(eth.accounts[0])
0
> eth.getBalance(eth.coinbase)
0
> web3.fromWei(eth.getBalance(eth.coinbase));
0
> eth.getBalance("0x7f9153e8fe06c4b3eb10e8457c20d0559921de5c")
0
> eth.blockNumber
0
du -h
30G
It sounds like geth is not yet synced up.
Please type this into your geth console:
eth.getBlock("latest").number
As of this post, you should get 5757199 or higher.
"Syncing Ethereum is a pain point", as says Péter Szilágyi (team lead of ethereum).
From https://github.com/ethereum/go-ethereum/issues/16875:
The current default mode of sync for Geth is called fast sync. Instead
of starting from the genesis block and reprocessing all the
transactions that ever occurred (which could take weeks), fast sync
downloads the blocks, and only verifies the associated proof-of-works.
Downloading all the blocks is a straightforward and fast procedure and
will relatively quickly reassemble the entire chain.
Many people falsely assume that because they have the blocks, they are
in sync. Unfortunately this is not the case, since no transaction was
executed, so we do not have any account state available (ie. balances,
nonces, smart contract code and data). These need to be downloaded
separately and cross checked with the latest blocks. This phase is
called the state trie download and it actually runs concurrently with
the block downloads; alas it take a lot longer nowadays than
downloading the blocks.
So, what's the state trie? In the Ethereum mainnet, there are a ton of
accounts already, which track the balance, nonce, etc of each
user/contract. The accounts themselves are however insufficient to run
a node, they need to be cryptographically linked to each block so that
nodes can actually verify that the account's are not tampered with.
This cryptographic linking is done by creating a tree data structure
above the accounts, each level aggregating the layer below it into an
ever smaller layer, until you reach the single root. This gigantic
data structure containing all the accounts and the intermediate
cryptographic proofs is called the state trie.
Ok, so why does this pose a problem? This trie data structure is an
intricate interlink of hundreds of millions of tiny cryptographic
proofs (trie nodes). To truly have a synchronized node, you need to
download all the account data, as well as all the tiny cryptographic
proofs to verify that noone in the network is trying to cheat you.
This itself is already a crazy number of data items. The part where it
gets even messier is that this data is constantly morphing: at every
block (15s), about 1000 nodes are deleted from this trie and about
2000 new ones are added. This means your node needs to synchronize a
dataset that is changing 200 times per second. The worst part is that
while you are synchronizing, the network is moving forward, and state
that you begun to download might disappear while you're downloading,
so your node needs to constantly follow the network while trying to
gather all the recent data. But until you actually do gather all the
data, your local node is not usable since it cannot cryptographically
prove anything about any accounts.
If you see that you are 64 blocks behind mainnet, you aren't yet
synchronized, not even close. You are just done with the block
download phase and still running the state downloads. You can see this
yourself via the seemingly endless Imported state entries [...] stream
of logs. You'll need to wait that out too before your node comes truly
online.
1, How long does a message live in the listener queue? Until the dispatcher reads the message out of the queue in a "1 publisher 1 consumer" scenario?
Listener listener = new Listener(Queue.Default, transport, subject, new object());
listener.MessageReceived += OnMessageReceived;
Dispatcher dispatcher = new Dispatcher(listener.Queue);
2, Tibco RV is typically used in a large fanout system, with relatively loose requirements on delivery reliability, say, market data published to 20 applications in an enterprise. I've heard Tibco RV implements a "no copy" solution for fanouts - how is that even possible? I'd assume we need to at least go through all registered listeners for that queue and notify each of them, in which process the message is copied 20 times. Please enlighten me.
3, Combine question 1 and 2, it doesn't make sense for a message to live in the listener queue until ALL registered listeners have consumed the message - What happens if 1 of the 20 applications goes offline? It's going to bring down the rv daemon process due to ever increasing messages. Does Tibco RV have a lifetime limit(ttl) for each message? How do I check it and set it to new values?
There isn't much related info on Google, so please help.
Thanks.
Great questions.
Keep in mind that unless you are using RV certified messaging there will be no persistence to disk. Messages sent will remain in the memory of the sending Rendezvous daemon until they are delivered to all consumers.
That said, another thing to understand is that RV is an optimistic protocol as opposed to say, TCP which is a pessimistic protocol. Every packet is sent using TCP must be acknowledged. This round-trip protocol slows things down. Rendezvous on the other hand uses a protocol that sits on top of UDP which is session-less and does not require acks. Therefore, when a Rendezvous daemon sends a message, it is assumed to have been delivered successfully unless a retransmission request is received. So to completely answer your first question the default behavior of a Rendezvous daemon is to keep messages that it has sent in memory for 60 seconds after it has sent the message. This allows it to honor retransmission requests.
Fan out is achieved using broadcast or multicast protocols on top of UDP. The use of broadcast is discouraged and multicast is encouraged. Using multicast groups uses considerably less network resources. At the network interface level only those hosts that have joined the multicast group will receive packets associated with the Rendezvous traffic. Similarly at the network switch level multicast is a lot less resource-intensive.
Bottom line is that the sending Rendezvous daemon only sends the message out once, and the network delivers a copy of the associated packets to either each host on the subnet if broadcast is used, or the hosts that have registred interest if multicast is used.
In pub-sub, typically consumers receive messages that are sent while they are alive and consuming. Thus with pure Rendezvous, if a consumer was to go down, the subscription would be cancelled for that consumer. If we think about your example of market data, this is exactly the behavior we want. IBM trades at thousands of times a second, so if I miss a price quote it's no big deal. I'll get the next one. Furthermore, I don't want stale prices.
That said, sometimes consumers do want messages sent while they were down. This can be achieved using certified messaging and setting up persistent correspondents. For more on this see the Rendezvous Concepts Guide. Lastly, the 60-second behavior that I mentioned in point #1 can be changed using the -reliability parameter when starting the Rendezvous daemon. There are some cases where this may make sense (although the default is best for most common cases). For more details on this take a look at the Rendezvous Admin Guide.
I have some requirements for a system in need of a message queue:
The subscribers shall get individual queues.
The individual queues shall NOT be deleted when the subscriber disconnects
The subscriber shall be able to reconnect to its own queue if it looses connection
Only the subscriber shall be able to use the queue assigned to it
Nice to have: the queues survive a server restart
Can RabbitMQ be used to implement this, and in that case how?
I have only recently started using Rabbit but I believe your requirements can be addressed fairly easily.
1) I have implemented specific queues for individual subscribers by having the subscriber declare the queue (and related routing key) using its machine name as part of the queue name. The exchange takes care of routing messages appropriately by way of the binding/routing keys. In my case, all subscribers get a copy of the same message posted by the publisher and an arbitrary number of subscribers can declare their own queues and start receiving messages.
2) That's pretty much standard. If you declare a queue then it will remain in the exchange, and if it is set as durable then it will survive broker restarts. In any case, your subscriber should call queue.Declare() at startup to ensure that the queue exists but in terms of the subscriber disconnecting, the queue will remain.
3) If the queue is there and a subscriber is listening to that queue by name then there's no reason why it shouldn't be able to reconnect.
4) I haven't really delved in to the security aspects of Rabbit yet. There may be a means of securing individual queues though I'll let someone else comment on this as I'm no authority.
5) See (2). Messages will also survive a restart if set as durable as they are then written to disk. This incurs a performance penalty as there's disk I/O but that's kind of what you'd expect.
So basically, yes. Rabbit can do as you ask. In terms of 'how', there are varying degrees of 'how'. Will happily try to provide you with code-level answers should you have trouble implementing any of the above. In the meantime, and if you haven't already done so, I suggest reading through the docs:
http://www.rabbitmq.com/documentation.html
HTH. Steve