What's the difference between Mutable File System's (MFS) read:
await ipfs.files.read('/some/stuff/success.txt')
and Regular Files API's cat:
await ifps.cat(someFile)
Is it the fact that the cat method first searches your own node for the file requested, and if it can't find it there, it will attempt to find it on the broader IPFS network?
The parameter for cat is an IPFS Path, not a file path. An IPFS Path is basically all the variations on a content identifier.
cat will fetch the data from the local repo if it exists there, or ask the network if it does not.
files.read accepts an MFS path, which is a file path for a file you've already added to your IPFS repo - as well as an IPFS Path or CID object. If you pass an MFS path, a file or directory at that path must already be in the local repo already or the API will not find anything.
The documentation for files.read:
https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#filesread
The documentation for cat:
https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#cat
A good tutorial on the Regular Files API is here:
https://proto.school/#/regular-files-api/04
And one for MFS and file.read:
https://proto.school/#/mutable-file-system/10
And a tutorial on CIDs (content identifiers) is here:
https://proto.school/#/data-structures/04
Related
I figured out pinned data is cached in blocks folder. Can I just get away with copying the files I need in blocks folder to backup? Is datastore folder generated automatically by ipfs daemon?
I tried copying only the blocks folder to another ipfs deamon and it recognized the .data files as pinned files and created a different datastore folder.
There are three stores used by Kubo (formerly go-ipfs):
Key store storing private/public keys for PeerID and IPNS
Data store
Block store
These have folders in the .ipfs directory used by Kubo (with default configuration which uses leveldb for data store and flatfs for blockstore).
datastore folder: used by leveldb to store things like pins and MFS roots
blocks folder where blocks are stored. This includes non-pinned blocks that are cached by your node.
You could copy the blocks folder to another IPFS daemon with the same configuration. However, I'd be aware that it may not be the best way to do this, especially if the node is running and modifying the blocks folder.
A much more explicit way would be to use the ipfs dag export <CID> command to export .car files.
.car files are convenient because they can be imported into another IPFS node and contain inside all the blocks.
I have built a python module to access internal data files that can be accessed on multiple systems as we have mirrors of our data release. I use this config.py file to help identify all the paths. Many of the scripts include accessing this path info but I don't see a reason why readthedocs needs to build it. How can I get it to ignore these paths?
There are many other modules that do other things with the data and I have found read-the-docs to be a nice reference for new users. Unfortunately, my readthedocs builds have been failing for ages as a result of trying to find some of the local files.
https://readthedocs.org/projects/hetdex-api/builds/18207723/
FileNotFoundError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/hetdex-api/checkouts/latest/docs/hdr3/survey/amp_flag.fits'
According to https://docs.ipfs.io/guides/concepts/pinning/ , running the command ipfs add hello.txt apparently "pins" the file "hello.txt", yet why don't I see the file listed afterwards when I run the command ipfs files ls? It only lists files I added with the IPFS desktop app. Why is "hello.txt" not in the list now?
Also, I found a list of so-called "pinned" objects, by running the command ipfs pin ls, however none of the CID's that show up there correspond to "hello.txt", or even any of the previously mentioned files added using the IPFS desktop app.
How does one actually manage pinned files?
cool to see some questions about IPFS pop up here! :)
So, there are two different things:
Pins
Files/Folders (Called MFS)
They both overlap heavily, but it's best to describe that the MFS is basically a locally alterable filesystem with a mapping of 'objects' as files and folders.
You have a root ( / ) in your local IPFS client, where you can put files and folders.
For example you can add a folder recursively:
ipfs add -r --recursive /path/to/folder
You get a CID (content ID) back. This content ID represents the folder, all its files and all the file structure as a non-modifiable data structure.
This folder can be mapped to a name in your local root:
ipfs files cp /ipfs/<CID> /<foldername>
A ipfs files ls will now show this folder by name, while an ipfs pin ls --type=recursive will show the content-ID as pinned.
If you use the (Web)GUI, files will show up under the 'files' tab, while the pins show up under the 'pins' tab.
Just a side note, you don't have to pin a file or folder stored in your MFS, everything stored there will be permanently available.
If you going to change the folders, subfolders, files, etc in your MFS, the folder will get a different Content-ID and your pin will still make sure the old version is held on your client.
So if you add another file to your folder, by something like cat /path/to/file | ipfs files write --create /folder/<newfilename>, the CID of your folder will be different.
Compare ipfs files stat --hash /folder and afterwards again.
Hope I didn't fully confuse you :D
Best regards
Ruben
Answer:ipfs pin ls --type recursive
It's simple. Just run that command.
Some further notes: the type can be "direct", "recursive", "indirect", and "all". I ran these commands with these results ("Error: context canceled" means that I canceled the command with ctrl+c):
ipfs pin ls --type all - took too long, "Error: context canceled"
ipfs pin ls --type direct - took too long, "Error: context canceled"
ipfs pin ls --type indirect - took too long, "Error: context canceled"
ipfs pin ls --type recursive - worked, showed multiple, probably all, pins of mine
I don't really know what types other than recursive mean. You can read about them from the output of this command: ipfs pin ls --help.
I have hash of ipfs file, node with this file not work. But I need some how restore this file.
Can I some how restore file from hash?
You can download data from IPFS as long as there is at least one node providing it.
If the data was only at one node, and you shut it down, you won't be able to get the data until the node gets online again or someone else with the same data adds it to IPFS and announces it to DHT.
To get a list of nodes providing data for specific hash:
ipfs dht findprovs QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
If you want to download data from IPFS but can't run local node try downloading it from one of public gateways.
There are two concepts in IPFS, the connection of which is not very clear to me: IPFS pin and IPFS MFS.
As I understand it, ipfs pin allows you to keep content on your node, protecting it from being automatically removed by the garbage collector. In this case, if I add content by myself using ipfs add <file>, then it will be automatically pinned and then it can be unpinned and removed only manually.
IPFS MFS, on the other hand, allows objects to be manipulated as if they were in the file system. For example, I can copy a specific external object to MFS using ipfs files cp <id> <name>. After that, I can find out its ID using ipfs files stat <name>.
The questions are:
Are the files in MFS protected from being removed by garbage collector?
If protected, then why are they not displayed in ipfs pin ls?
Will the data be saved if I add it using ipfs add <file>, then add it to MFS using ipfs files cp <id> <name>, and then unpin it using ipfs pin rm <id>?
Is IPFS MFS a more reliable way to work with data?
these pretty good questions! Answering them separately
Are the files in MFS protected from being removed by garbage collector?
They are not by default Pinned. You will need to pin those files as well if you want them tracked by the Pinner. You can do a ipfs files stat /somePath, get the hash and then pin that hash.
The part where it gets confusing is that GC will do a "best effort" pinning, in which files that are accessed by the root of the MFS DAG will not be GC as well.
Example:
You add a file to MFS
You make a modification to that file on MFS
The previous version will get GC'ed
The latest version will be protected from GC
If you want to protect the previous, you can use the Pin API.
If protected, then why are they not displayed in ipfs pin ls?
As answered on 1., you will need to pin them manually to see it being tracked by the pinning system.
Will the data be saved if I add it using ipfs add <file>, then add it to MFS using ipfs files cp <id> <name>, and then unpin it using ipfs pin rm <id>?
Perhaps you get the gist by now. To clarify:
Pinning is a protection for garbage collection (GC). If pinned, GC won't delete it
MFS doesn't auto pin files. GC just tries to be friends with MFS and not GC files that are reachable by the root of MFS.
Is IPFS MFS a more reliable way to work with data?
It is a more familiar way as you get the regular directory structures and Unix like API to operate over files. It handles the graph manipulations for you.