I initialized a local directory with ipfs add -r . and was able to access it through https://ipfs.io/ipfs gateway using the hash.
I was also able to get the files from another node using ipfs get -o <file-name> <hash>
Is the file served through the ipfs.io gateway or through local gateways of other decentralized participating nodes?
TLDR: No
The go/js-ipfs CLIs will not make any HTTP related requests, to public gateways or otherwise, when you perform an ipfs get
Gateways, public or local, are just a convenient way of bridging the IPFS protocol stack with the standard experience of performing an HTTP request for some data. A local gateway will let you use standard HTTP based applications (e.g. web browsers, curl, etc.) while still utilizing your locally running IPFS daemon under the hood. On the other hand, the public gateways let you use standard HTTP based applications while using someone else's (i.e. publicly run infrastructure's) IPFS daemon under the hood.
The main utility of the public gateways is making content that peers have in the public IPFS network available over HTTP to people and applications that are not able to run IPFS.
Related
I installed Kubo on my local PC, I just entered the ipfs daemon command to run it. After that, I pushed some text content, it works fine and I can access it from the IPFS public gateway. But, when I put any image or file type of content and go to access it from a public IPFS gateway, it shows "504 Gateway Time-Out (Openresty)". Also, it takes a long time to load any content, it's too slow. Is it possible to make it too fast? How can I fix the "504 Gateway Time-Out" issue?
If the gateway cannot find the data you've requested, it will time out. A gateway is just an IPFS node with an HTTP server bolted onto it (more-or-less).
If the only node containing the data is your home node, you must ensure your node is discoverable so the gateway node can find and access your data. Personally I like to put my data on web3.storage if I need the popular public gateways to find it.
For more info check out my answer on the related question Why is it so hard for web browsers to open IPFS links?.
I am trying to send transactions via web3.py interface that is connected to a local geth node. Having read some comments on why using RPC is bad, I am still wondering if using -rpc option is unsafe when port 8545 is closed. According to this article (https://www.zdnet.com/article/hackers-ramp-up-attacks-on-mining-rigs-before-ethereum-price-crashes-into-the-gutter/) the vulnerability is just on exposed ports, but since I am basically communicating with a node on local network this shouldn't be a problem right?
The article covers an attack vector, where the attackers look for machines with opened port 8545 and try to run JSON-RPC commands (on these machines) that would benefit them. This attack only works if:
The machine has the port 8545 opened to public
The port is used by an Ethereum node (and not some arbitrary app)
The node has enabled JSON-RPC
The node hasn't enabled the user/password credentials for JSON-RPC
So as long as your node is only accessible on a local network, you are pretty much safe from this attack vector (assuming there's no port forwarding etc. that would actually allow accessing your node from a public network and that there's no attacker on your local network).
Consider 3 IPFS peers A, B and C
When
peer A establishes the connection with peer B and C (using ipfs swarm connect)
Will it form a cluster with A as leader? If yes, Do we need to manually create secret key? and Who and how the key is managed?
IPFS is a decentralized system, even you establish connection using peer A, at the end they all will end up sharing each other's DHT ( Distribute Hash Table) information and come at the same level. There will not be any leader in a cluster, and all peers will have the same privileges as any other peer in the network.
And right now there is no notion of a secret key in IPFS, all the data in IPFS network is publicly available if you want you have to implement a layer on the top of it and encrypt data before putting it into IPFS.
Private IPFS is designed for a particular IPFS node to connect to other peers who have a shared secret key. With IPFS private networks, each node specifies which other nodes it will connect to. Nodes in that network don’t respond to communications from nodes outside that network.
An IPFS-Cluster is a stand-alone application and a CLI client that allocates, replicates and tracks pins across a cluster of IPFS daemons. IPFS-Cluster uses the RAFT leader-based consensus algorithm to coordinate storage of a pinset, distributing the set of data across the participating nodes.
This difference between Private IPFS and IPFS cluster is remarkable. It is worth noting that a private network is a default feature implemented within the core IPFS functionality and IPFS-Cluster is its separate app. IPFS and IPFS-Cluster applications are installed as different packages, run as separate processes, and they have different peer IDs as well as API endpoints and ports. IPFS-Cluster daemon depends on IPFS daemon and should be started afterwards.
In a private IPFS network, you should have 'Go' and IPFS installed on all the nodes. Once it is done, run the following command to install the swarm key generation utility. Swarm key allows us to create a private network and tell network peers to communicate only with those peers who share this secret key.
This command should be run only on your Node0. We generate swarm.key on the bootstrap node and then just copy it to the rest of the nodes.
go get -u github.com/Kubuxu/go-ipfs-swarm-key-gen/ipfs-swarm-key-gen
Now run this utility on your first node to generate swarm.key under .ipfs folder:
ipfs-swarm-key-gen & > ~/.ipfs/swarm.key
Copy the file generated swarm.key to the IPFS directory of each node participating in the private network. Please let me know if you need further details on this.
No.It doesn't form a cluster because there is a separate implementation of IPFS for the above mentioned problem named as IPFS Cluster where a particular node pins the data across various other nodes through which other nodes in the network can access the data.The pinning of data by the node functions through a secret-key. For more information you can go through the documentation of IPFS Cluster.
https://cluster.ipfs.io/documentation/
I am running a Ruby on Rails app (using Passenger in Nginx mode) on Google Container Engine. These pods are sitting behind a GCE network load balancer. My question is how to access the external Client IP from inside the Rails app.
The Github issue here seems to present a solution, but I ran the suggested:
for node in $(kubectl get nodes -o name | cut -f2 -d/); do
kubectl annotate node $node \
net.beta.kubernetes.io/proxy-mode=iptables;
gcloud compute ssh --zone=us-central1-b $node \
--command="sudo /etc/init.d/kube-proxy restart";
done
but I am still getting a REMOTE_ADDR header of 10.140.0.1.
On ideas on how I could get access to the real Client IP (for geolocation purposes)?
Edit: To be more clear, I am aware of the ways of accessing the client IP from inside Rails, however all of these solutions are getting me the internal Kubernetes IP, I believe the GCE network load balancer is not configured (or perhaps unable) to send the real client IP.
A Googler's answer to another version of my question verifies what I am trying to do is not currently possible with the Google Container Engine Network Load Balancer currently.
EDIT (May 31, 2017): as of Kubernetes v1.5 and up this is possible on GKE with the beta annotation service.beta.kubernetes.io/external-traffic. This was answered on SO here. Please note when I added the annotation the health checks were not created on the existing nodes. Recreating the LB and restarting the nodes solved the issue.
It seems as though this is not a rails problem at all, but one of GCE. You can try the first part of
request.env["HTTP_X_FORWARDED_FOR"]
Explanation
Getting Orgin IP From Load Balancer advises that https://cloud.google.com/compute/docs/load-balancing/http/ has the text
The proxies set HTTP request/response headers as follows:
Via: 1.1 google (requests and responses)
X-Forwarded-Proto: [http | https] (requests only)
X-Forwarded-For: <client IP(s)>, <global forwarding rule external IP> (requests only)
Can be a comma-separated list of IP addresses depending on the X-Forwarded-For entries appended by the intermediaries the client is
traveling through. The first element in the section
shows the origin address.
X-Cloud-Trace-Context: <trace-id>/<span-id>;<trace-options> (requests only)
Parameters for Stackdriver Trace.
I have deployed a hadoop cluster on google compute engine. I then run a machine learning algorithm (Cloudera's Oryx) on the master node of the hadoop cluster. The output of this algorithm is accessed via an HTTP REST API. Thus I need to access the output either by a web browser, or via REST commands. However, I cannot resolve the address for the output of the master node which takes the form http://CLUSTER_NAME-m.c.PROJECT_NAME.internal:8091.
I have allowed http traffic and allowed access to ports 80 and 8091 on the network. But I cannot resolve the address given. Note this http address is NOT the IP address of the master node instance.
I have followed along with examples for accessing IP addresses of compute instances. However, I cannot find examples of accessing a single node of a hadoop cluster on GCE, that follows this form http://CLUSTER_NAME-m.c.PROJECT_NAME.internal:8091. Any help would be appreciated. Thank you.
The reason you're seeing this is that the "HOSTNAME.c.PROJECT.internal" name is only resolvable from within the GCE network of that same instance itself; these domain names are not globally visible. So, if you were to SSH into your master node first, and then try to curl http://CLUSTER_NAME-m.c.PROJECT_NAME.internal:8091 then you should successfully retrieve the contents, whereas trying to access from your personal browser will just fail to resolve that hostname into any IP address.
So unfortunately, the quickest way for you to retrieve those contents is indeed to use the external IP address of your GCE instance. If you've already opened port 8091 on the network, simply use gcutil getinstance CLUSTER_NAME-m and look for the entry specifying external IP address; then plug that in as your URL: http://[external ip address]:8091.
If you turned up the cluster using bdutil, a more involved but nicer way to access your cluster is by running the bdutil socksproxy command. This opens a dynamic-port-forwarding SSH tunnel to your master node as a SOCKS5 proxy, so that you can then configure your browser to use localhost:1080 as your proxy server, make sure to enable remote DNS resolution, and then visit your browser using the normal http://CLUSTER_NAME-m.c.PROJECT_NAME.internal:8091 URL.