How can I unzip a file asynchronously to Isolated Storage? My zip file is in Isolated Storage, and I want to wait until the unzip operation has completed.
I am using SharpzipLib and SharpGIS, but I haven't seen any support for async unzip.
Also, I have to handle unzipping of large files, so out of memory exceptions also need to be handled. Are there any examples or suitable links to do it in an async way?
Related
I figured out pinned data is cached in blocks folder. Can I just get away with copying the files I need in blocks folder to backup? Is datastore folder generated automatically by ipfs daemon?
I tried copying only the blocks folder to another ipfs deamon and it recognized the .data files as pinned files and created a different datastore folder.
There are three stores used by Kubo (formerly go-ipfs):
Key store storing private/public keys for PeerID and IPNS
Data store
Block store
These have folders in the .ipfs directory used by Kubo (with default configuration which uses leveldb for data store and flatfs for blockstore).
datastore folder: used by leveldb to store things like pins and MFS roots
blocks folder where blocks are stored. This includes non-pinned blocks that are cached by your node.
You could copy the blocks folder to another IPFS daemon with the same configuration. However, I'd be aware that it may not be the best way to do this, especially if the node is running and modifying the blocks folder.
A much more explicit way would be to use the ipfs dag export <CID> command to export .car files.
.car files are convenient because they can be imported into another IPFS node and contain inside all the blocks.
Using Google Takeout usually gives you zip or tar.gz in Google Drive.
Is there any way, by API or any programming methods, to decompress those files and put them back in Drive without downloading them locally?
You can't decompress a file in Drive, but there are some workarounds for that. Using the API you could easily develop a solution that downloads the file, then decompresses it locally and finally uploads it again. As an alternative, you could speed up that process by using a sync folder that allows you to extract the file locally. Feel free to ask any doubts about those approaches.
I am trying to extract a 14.6 GB 7z file (https://archive.org/download/stackexchange/stackoverflow.com-Posts.7z).
I have it downloaded and saved in my Google Drive. I mount my drive to Google Colab and then change the current directory to where the file is located: os.chdir('/content/drive/My Drive/.../')
When I try to unzip the file !p7zip -k -d stackoverflow.com-Posts.7z, it uses the current instance's HDD space and during this process, it runs out of all the available allocated HDD space, and hence the unzip abruptly terminates.
Is there a way to extract the file without using the instance's HDD space OR do it in chunks, such that the extract is successful.
PS: I believe, when decompressed the file size is ~100 GB
You can try to read data by blocks, using libarchive, without unzip it first.
https://github.com/dsoprea/PyEasyArchive
Here's an example notebook
I can download a file with ipfs get hash but does that seed the file too? It also downloads a copy of the file and saves it to the folder I run the command from which I don't want. I only want the file to be chopped up and seeded from .ipfs
Any files accessed through your gateway will be cached and available (seeding) to the rest of the p2p network as long as your daemon is running and publicly accessible.
You can run ipfs repo gc to clear your cache.
You may also add files to your local storage that won't be garbage collected. You can do this with ipfs pin {hash}, and you can view the pinned items with ipfs pin ls.
I uploaded a pdf file to the IPFS de-centralised network. My question here. When I have the IPFS console and I couldn't view the pdf file anymore through the https://ipfs.io/gateway. Why is that? My understanding is once the file is uploaded to the IPFS network and the file will be distributed to the nodes.
Adding a file to IPFS via ipfs add <file> does not distribute it to the network (that would be free hosting!), it only puts the file into the standard format (IPLD) and makes it possible to access over the network (IPFS) as long as someone connected to the network has the file. When you first add something, that's only you. So if you close your laptop, suddenly the file is no longer available. UNLESS someone else has downloaded it since then, because then they can distribute it while your computer is off. There are many "pinning services" which do just that, for a small fee.
Hi Your understanding is correct,But can you tell me how are you uploading files to ipfs network there are number of ways to add data to ipfs network,
if you are able to add data to ipfs you will get the hash of the data, condition is daemon is running locally so that your data can be broadcasted to other peers you are attached to, you can check it by command: ipfs swarm peers
if above conditions are fulfilled you view/get data from https://ipfs.io/ipfs/<replace with hash you will get after adding>
if daemon is not running you can able to add you file and get the hash but you files will be saved locally, you wont be able to access it from web.
please let me know if you need other information