i have a bucket in which i have multiple folders. these folders further have images stored in them. i want to make a csv with the gsutil URIs of all these images. How can I do that?
I could not find a way to get the gsutil URIs of all images at once.
You need to add -R parameter and it will go thru all folders:
gsutil ls -R gs://bucket-name
and then you can use grep -v :$ to discard folders
Related
I'm trying to figure out a way to download multiple files from a single bucket.
Assume I have a bucket with hundreds of files and I want to download 65 of those files.
I can obviously go to the console and download each file I need individually, but this is not very efficient.
One other option would be to download an entire folder using
gsutil -m cp -r gs://bucket/folder [destination folder]
However, that will download ALL files, which isn't convenient either.
Can I somehow include all the filenames I want in a CSV and have gsutil iterate that CSV file?
You can't use a CSV file directly, but you can pipe a list of urls into gsutil:
list_of_urls > gsutil -m cp -I ./download_dir
Perhaps that is good enough.
I can add recursively a bunch of files within IPFS with
$ ipfs add -r data/
How can I get a list back of all loaded file objects [in a specific directory]? Similar to aws s3 listObjects...
The ipfs file ls command does not seem to be recursive. I understand that I can call the API a thousand times but that does not seem to be very efficient.
I must be missing something here.
Thanks,
Pat.
IPFS is based on Merkle tree, so you can display all elements under your root resource. You can use:
web ui: http://localhost:8080/ipfs/<your_root_resource_hash>
graphmd: https://ipfs.io/ipfs/QmNZiPk974vDsPmQii3YbrMKfi12KTSNM7XMiYyiea4VYZ/example#/ipfs/QmRFTtbyEp3UaT67ByYW299Suw7HKKnWK6NJMdNFzDjYdX/graphmd/README.md
shell commands:
ipfs ls <your_root_resource_hash>
ipfs refs -r <your_root_resource_hash>
Docs for ipfs files ls
edit : More important, your directory name is not persisted into IPFS. You can access your resource knowing its hash, the one you get when you add it with ipfs add -r <your_dir>
You can use this command to list all objects
ipfs files ls
If you are interested in local files added with something like ipfs add -r --nocopy /files, what you want is
ipfs filestore ls
Unfortunately now this lists blocks instead of whole files, see https://github.com/ipfs/go-ipfs/issues/5293
you can use the command:
ipfs filestore ls
The format of the hg mv command is hg rename [OPTION]... SOURCE... DEST
. Path names are relative to the current directory. Thus, when you are at a command prompt at the root directory and specify hg mv -n -I * A\B Z, mercurial will create the directory Z under the root directory, and move A\B\readme.txt to Z\readme.txt.
How can you specify, under Windows, that Z is the repository root directory? I tried using '.' as destination, i.e. hg mv -n -I * A\B . but got a message that A\B\readme.txt will be copied to B\readme.txt, not to readme.txt at the root. I tried using '~' as the destination, but hg mv -n -I * A\B ~ got me a new directory named "~" below the root, obviously not what I wanted.
So my question is: How do I specify the repository root directory as the destination to the mercurial move command?
edit: I'll try to clarify the issue.
I have an OldDev repository containing two products: Product-A and Product-B. Using the '~' symbol to denote OldDev's root folder, OldDev contains two folders: ~/Product-A and ~/Product-B (in addition, of course, to ~/.hg where its metadata is stored).
Each product is composed of a few projects, and each such project is assigned a folder under the product's folder. Thus Product-A has the Project-A, Project-B and Project-C, stored in ~/Product-A/Project-A, ~/Product-A/Project-B and ~/Product-A/Project-C, correspondingly. ~/Product-A/Project-A/xxx.cs is one of (Product-A's) Project-A's files.
Now I want to extract Project-A to its own NewDev repository. As it's the single project in NewDev, it makes no sense to retain the product/project hierarchy, so I want it to be at the root of NewDev: it xxx.cs file, for example, will be #/xxx.cs, where # is the root folder of NewDev (the one contianing NewDev's .hg directory where NewDev's metadata is stored).
To extract Project-A to NewDev I used the the convert extension, as documented in "split a repository in two". I used a mapfile containing the one mapping include Product-A/Project-A.
So far, NewDev is an exact subtree of OldDev. It does not contain ~/Product-B, it does not contain ~/Product-A/Project-B nor ~/Product-A/Project-C. It only contains ~/Product-A/Project-A. The files that remained are located at exactly the same paths as before, but only those files that belong to Product-A's Project-A were retained.
So, I've achieved half of my goals: I split OldDev, with its many products and projects, and created NewDev with only one project (Project-A). However, the files of Project-A are not at # but at their old (OldDev) location #/Product-A/Project-A. I need to move them up two steps so xxx.cs, will be at #/xxx.cs and not at #/Product-A/Project-A/xxx.cs
To move the files I tried to use the hg mv command, but I can't figure how to specify the root (#) as the destination.
Solution: What worked for me, based on Marc Anton Dahmen's answer, is as follows:
convert1.txt: hg convert -s hg -d hg --filemap mapfile1.txt olddev temprepo
mapfile1.txt: include Product-A/Project-A
convert2.txt: hg convert -s hg -d hg --filemap mapfile2.txt temprepo newrepo
mapfile2.txt: rename Product-A/Project-A .
Where the text of convrert1.txt and convert2.txt, of course, shell commands.
You must use the rename directive in your filemap instead of include like so:
rename Project-A .
Moving every file in a repository and the repository data is not an hg mv operation because that cannot change where the repository meta-data is stored.
The wording of your question is still really ambiguous, but I have a decent guess as to what you want to do.
Suppose you have a repo called /some/dir/avi-repo and you really want it to be in /avi-repo. Use clone:
cd /
hg clone /some /avi-repo
Now you have two identical copies of the repo, one in /some/dir/avi-repo and one in /avi-repo. You can delete all of /some/dir/avi-repo now.
Your desire seems a little more complicated than that with a tree like:
/some
---- /.hg # the repository meta-data
---- /dir # no files in here just the sub-dir
-------- /avi-repo
------------/file.c
------------/file.dat
------------/important-file.txt
And you want to move avi-repo to /some/avi-repo. You should be able to do that with the right sequence of mercurial commands, but it is far easier to:
mkdir /temp
cd /temp
hg clone /some /temp/avi-clone
rm -r /some
mkdir /some
hg clone /temp/avi-clone /some
Or some variant of that. The point is that repatriating an entire repository is not a job for hg mv.
I need to get contents of a specific revision/node of a file in a local repository and write it to a temporary file.
I know it is possible to do through the internal Mercurial API.
Is there a built-in command or an extension?
You can use hg cat:
hg cat -r revisionid filename > tmpfile
The fastest, large and/or binary file friendly way to do this is:
hg cat -r revisionid repoRelativeFilePath -o tempFilePath
The tempFilePath, unless absolutely rooted (ex. 'C:\') will be relative to the repo's root
I've got a Mercurial repository that contains a subdirectory for design files and a sub directory for code files.
What's the best way for me to deploy the code subdirectories on my server, keeping it secure, without deploying the design files?
Ideally you should have these in separate repos as Kyle pointed out, optionally, using sub repositories.
However, with your setup as is something like this might be all you need:
hg archive -X designDocs /path/to/deploymentDir
or if you need to transfer it first:
hg archive -X designDocs --type zip /path/to/newDeploymentArchive.zip
You might try using subrepositories for the two subdirectories.