I am using r8 dexer to make classes.dex file from a bunch of *.class files. I would like to use obfuscation feature for all my classes, but the problem is that I also have AndroidManifest.xml file where class names are specified. So I have to know from which name to which name the r8 transformation took place in order to change my AndroidManifest.xml accordingly. The question is - how to prodice such a map?
Here is how the things are looking so far:
javac -cp $CLASSPATH -source 1.8 -target 1.8 com/obs/*.java
#dx --dex --output classes.dex com/obs/*.class $(cat classes.json | jq .[] | xargs -I{} echo -n "{} ")
# https://r8.googlesource.com/r8/+/refs/heads/d8-1.5.13
# https://r8.googlesource.com/r8/+archive/refs/heads/d8-1.5.13.tar.gz
#
java -jar /opt/r8/build/libs/r8.jar --version
java -jar /opt/r8/build/libs/r8.jar --release --output . --pg-conf proguard.cfg $(echo -n "$CLASSPATH" | xargs -I{} -d: echo -n " --lib {} ") com/obs/*.class $(cat classes.json | jq -r .[] | xargs -I{} echo -n "{} ")
Oh. Found it apparently.
-printmapping
https://www.guardsquare.com/en/products/proguard/manual/usage
On the R8 command line you can use the option --pg-map-output <file> instead of adding -printmapping to the configuration.
When building using the Android Gradle Plugin with minifyEnabled true the mapping file is by default generated in app/build/outputs/mapping/release.
Related
I have thousands of individually named json files in a single windows directory. I'm trying to use jq to merge them all into a single file I can then import into a jupyter notebook.
I keep getting a permissioned denied error when I try and run the following command:
jq --slurp 'map(.[])' bill
I've tried editing the directory (bill) permissions. My file path looks like this:
\downloadedJSONfiles\AK\2019-2020_31st_Legislature\bill
I downloaded jq through chocolately. I'm using cmder
You are getting permission denied because you provided a directory where a path to an ordinary file is expected.
I'm going to start with the "unix" approach because jq has unix roots. The following is the command you want to use:
jq --slurp 'map(.[])' bill/a.json bill/b.json bill/c.json ...
The shell will expand the following command into the above:
jq --slurp 'map(.[])' bill/*.json
The problem is that this can easily result in a command that's too long. So you really want one the following:
# A
(
jq '.[]' bill/a.json
jq '.[]' bill/b.json
jq '.[]' bill/c.json
jq '.[]' bill/d.json
jq '.[]' bill/e.json
jq '.[]' bill/f.json
) | jq --slurp .
# B
(
cat bill/a.json
cat bill/b.json
cat bill/c.json
cat bill/d.json
cat bill/e.json
cat bill/f.json
) | jq --slurp 'map(.[])'
# C
(
jq '.[]' bill/a.json bill/b.json bill/c.json
jq '.[]' bill/d.json bill/e.json bill/f.json
) | jq --slurp .
# D
(
cat bill/a.json bill/b.json bill/c.json
cat bill/d.json bill/e.json bill/f.json
) | jq --slurp 'map(.[])'
Something equivalent can be achieved using any of the following:
# Portable. Equivalent to A
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec jq '.[]' {} \; | jq --slurp .
# Portable. Equivalent to B
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec cat {} \; | jq --slurp 'map(.[])'
# Possibly portable with tweaks. Similar to "C"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -print0 |
xargs -r0 jq '.[]' |
jq --slurp .
# Possibly portable with tweaks. Similar to "D"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -print0 |
xargs -r0 cat |
jq --slurp .
# GNU-specific. Equivalent to "C"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec jq '.[]' {} + | jq --slurp .
# GNU-specific. Equivalent to "D"
find bill -mindepth 1 -maxdepth 1 -name '*.json' -exec cat {} + | jq --slurp 'map(.[])'
But you asked about Windows. In the Windows world it's up to programs to perform their own wildcard expansion. So you'd expect to be able to do
jq --slurp 'map(.[])' bill\*.json
However, jq wasn't properly ported.
Assertion failed!
Program: c:\bin\jq.exe
File: src/main.c, Line 256
Expression: wargc == argc
So like in unix, you have to pass all the files you want to process to jq as separate arguments.
Using cmd, you could use either of the following:
:: Not efficient
copy /y nul bill.jsonl
for %q in (bill\*.json) do jq ".[]" %q >>bill.jsonl
jq --slurp . bill.jsonl
del bill.jsonl
:: More efficient
copy /y nul+bill\*.json bill.jsonl
jq --slurp "map(.[])" bill.jsonl
del bill.jsonl
PowerShell is a far more advanced shell than cmd. With PowerShell, you could use
jq --slurp 'map(.[])' ( Get-Item bill\*.json )
But just like the simple unix version, the above can easily result in a command line that's too long. To avoid that, we can use the following:
# Not efficient
Get-Item bill\*.json | %{ jq '.[]' $_ } | jq --slurp .
# More efficient
%{ Get-Content bill\*.json } | jq --slurp 'map(.[])'
(%{...} is a shorthand for ForEach-Object {...}.)
Finally, I'm not familiar with Cmder.
How can I find all package.json files and merge them into one file using jq within Bash? The following snippet is as far as I have gotten, however it appends the files:
find ../../projects -maxdepth 4 -type f -name "package.json" \
-exec cat {} + | jq -s . > $(CURDIR)/node/.tmp/package.json
I have tried using 'add' but that seems to overwrite the target each time without merging the two.
My project structure initially looks like this:
\ projects
\ webapp
\ package.json
\ service
\ package.json
\ admin
\ package.json
\ solutions
\ killersolution
Makefile
\ node
And should look like this after make prenode (see below):
\ projects
\ webapp
\ package.json
\ service
\ package.json
\ admin
\ package.json
\ solutions
\ killersolution
Makefile
\ node
\ .tmp
\ package.json <- created
I am using a Makefile to kick this off:
prenode:
#find ./node -type d -name ".tmp" -exec rm -rf {} +;
#mkdir -p ./node/.tmp
#find ../../solutions -maxdepth 4 -type f -name "package.json" -exec cat {} + ...
Edit #1 : Example Input & Output
Let us assume 3 package.json files were found. The dependencies and devDependencies are different but must be combined:
Found file #1 ...
{
"name":"project-a",
"dependencies":{
"module-a":"1.2.3"
}
}
Found file #2 ...
{
"name":"project-b",
"dependencies":{
"module-b":"2.3.4"
}
}
Found file #3 ...
{
"name":"project-c",
"devDependencies":{
"gulp":"*"
}
}
... would all be combined to make the following file:
{
"name":"project-c",
"dependencies":{
"module-a":"1.2.3",
"module-b":"2.3.4"
},
"devDependencies":{
"gulp":"*"
}
}
*Note:
The name property, in the final output file, is irrelevant. The key here is the merge of the dependencies and devDependencies objects.
Assuming the following find command works for you (adjust if necessary)
find ../../projects -name package.json
Here is a solution which uses the jq * operator along with reduce and the -s option to merge the objects:
jq -s 'reduce .[] as $d ({}; . *= $d)' $(find ../../projects -name package.json)
If you prefer you can just as easily concatenate the files and send them to jq
find ../../projects -name package.json -exec cat {} \; | \
jq -M -s 'reduce .[] as $d ({}; . *= $d)'
As noted in my reply to your comment if you're doing this in a makefile you need to take extra steps to deal with the $ or put your filter in a file and use -f
If your jq has inputs, here is a more efficient solution than is possible using the -s ("slurp") option:
reduce inputs as $i ({};
.dependencies += ($i | .dependencies )
| .devDependencies += ($i | .devDependencies ) )
This also produces a "clean" result (i.e., no "name" field).
Remember to use the -n command-line option with inputs, e.g. along the lines of:
jq -n -f program.jq $(find ...)
reduce-free solution
In case your jq does not have inputs, here is a reduce-free solution when used with the "-s" command-line option:
map([.dependencies, .devDependencies])
| transpose
| map(add)
| {dependencies: .[0], devDependencies: .[1]}
As an FYI for anyone who needs it now, I am currently doing this using xargs and a utility called package-utils. The jq thing was more of a want than anything.
Here's the current makefile, if you'd like to use it:
prenode:
#find $(CURDIR)/node -type d -name ".tmp" -exec rm -rf {} +;
#mkdir -p $(CURDIR)/node/.tmp
#find ../../projects -type f -name "package.json" -maxdepth 4 -print0 | xargs -0 package-merge > $(CURDIR)/node/.tmp/package.json
My overall goal with this is for use within Docker and Docker Compose. I am using it to pre-cache all of the NPM modules, used within a single solution, in a base image. Basically, it creates a temporary package.json file that is used during the build process.
Within my Dockerfile is the following:
COPY ./.tmp/package.json /var/www/package.json
VOLUME /var/www
VOLUME /var/www/node_modules
WORKDIR /var/www
RUN npm install
RUN rm package.json
This allows the Node cache to be populated so any container based on this image will be able to install mostly everything from the cache.
There is a reason I'm doing this instead of a common volume for node_modules but that's a different story.
As a newbee to bash and jq, I was trying to download several urls from a json file using jq command in bash scripts.
My items.json file looks like this :
［
{"title" : [bob], "link" :[a.b.c]},
{"title" : [alice], "link" :[d.e.f]},
{"title" : [carol], "link" :[]}
]
what I was initially doing was just filter the non-empty link and put them in an array and then download the array:
#!/bin/bash
lnk=( $(jq -r '.[].link[0] | select (.!=null)' items.json) )
for element in ${lnk[#]}
do
wget $element
done
But the problem of this approach is that all the files downloaded use the link as the file names.
I wish to filter json file but still keeps the title name with the link so that i can rename the file in the wget command. But I dont have any idea on what structure should I use here. So how can i keep the title to in the filter and use it after?
You can use this:
IFS=$'\n' read -d '' -a titles < <(jq -r '.[] | select (.link[0]!=null) | .title[0]' items.json);
IFS=$'\n' read -d '' -a links < <(jq -r '.[] | select (.link[0]!=null) | .link[0]' items.json);
Then you can iterate over arrays "${title[#]}" & ${links[#]}...
for i in ${!titles[#]}; do
wget -O "${titles[i]}" "${links[#]}"
done
EDIT: Easier & safer approach:
jq -r '.[] | select (.link[0]!=null) | #sh "wget -O \(.title[0]) \(.link[0])"' items.json | bash
Here is a bash script demonstrating reading the result of a jq filter into bash variables.
#!/bin/bash
jq -M -r '
.[]
| select(.link[0]!=null)
| .title[0], .link[0]
' items.json | \
while read -r title; read -r url; do
echo "$title: $url" # replace with wget command
done
Using jq to concat json files in a directory.
The directory contains a few hundred thousand files.
jq -s '.' *.json > output.json
returns an error that the file list is too long. Is there a way to write this that uses a method that will take in more files?
If jq -s . *.json > output.json produces "argument list too long"; you could fix it using zargs in zsh:
$ zargs *.json -- cat | jq -s . > output.json
That you could emulate using find as shown in #chepner's answer:
$ find -maxdepth 1 -name \*.json -exec cat {} + | jq -s . > output.json
"Data in jq is represented as streams of JSON values ... This is a cat-friendly format - you can just join two JSON streams together and get a valid JSON stream.":
$ echo '{"a":1}{"b":2}' | jq -s .
[
{
"a": 1
},
{
"b": 2
}
]
The problem is that the length of a command line is limited, and *.json produces too many argument for one command line. One workaround is to expand the pattern in a for loop, which does not have the same limits as a command line, because bash can iterate over the result internally rather than having to construct an argument list for an external command:
for f in *.json; do
cat "$f"
done | jq -s '.' > output.json
This is rather inefficient, though, since it requires running cat once for each file. A more efficient solution is to use find to call cat with as many files as possible each time.
find . -name '*.json' -exec cat '{}' + | jq -s '.' > output.json
(You may be able to simply use
find . -name '*.json' -exec jq -s '{}' + > output.json
as well; it may depend on what is in the files and how multiple calls to jq using the -s option compares to a single call.)
[EDITED to use find]
One obvious thing to consider would be to process one file at a time, and then "slurp" them:
$ while IFS= read -r f ; cat "$f" ; done <(find . -maxdepth 1 -name "*.json") | jq -s .
This however would presumably require a lot of memory. Thus the following may be closer to what you need:
#!/bin/bash
# "slurp" a bunch of files
# Requires a version of jq with 'inputs'.
echo "["
while read f
do
jq -nr 'inputs | (., ",")' $f
done < <(find . -maxdepth 1 -name "*.json") | sed '$d'
echo "]"
How can I list normal text (.txt) filenames, that don't end with a newline?
e.g.: list (output) this filename:
$ cat a.txt
asdfasdlsad4randomcharsf
asdfasdfaasdf43randomcharssdf
$
and don't list (output) this filename:
$ cat b.txt
asdfasdlsad4randomcharsf
asdfasdfaasdf43randomcharssdf
$
Use pcregrep, a Perl Compatible Regular Expressions version of grep which supports a multiline mode using -M flag that can be used to match (or not match) if the last line had a newline:
pcregrep -LMr '\n\Z' .
In the above example we are saying to search recursively (-r) in current directory (.) listing files that don't match (-L) our multiline (-M) regex that looks for a newline at the end of a file ('\n\Z')
Changing -L to -l would list the files that do have newlines in them.
pcregrep can be installed on MacOS with the homebrew pcre package: brew install pcre
Ok it's my turn, I give it a try:
find . -type f -print0 | xargs -0 -L1 bash -c 'test "$(tail -c 1 "$0")" && echo "No new line at end of $0"'
If you have ripgrep installed:
rg -l '[^\n]\z'
That regular expression matches any character which is not a newline, and then the end of the file.
Give this a try:
find . -type f -exec sh -c '[ -z "$(sed -n "\$p" "$1")" ]' _ {} \; -print
It will print filenames of files that end with a blank line. To print files that don't end in a blank line change the -z to -n.
If you are using 'ack' (http://beyondgrep.com) as a alternative to grep, you just run this:
ack -v '\n$'
It actually searches all lines that don't match (-v) a newline at the end of the line.
The best oneliner I could come up with is this:
git grep --cached -Il '' | xargs -L1 bash -c 'if test "$(tail -c 1 "$0")"; then echo "No new line at end of $0"; exit 1; fi'
This uses git grep, because in my use-case I want to ensure files commited to a git branch have ending newlines.
If this is required outside of a git repo, you can of course just use grep instead.
grep -RIl '' . | xargs -L1 bash -c 'if test "$(tail -c 1 "$0")"; then echo "No new line at end of $0"; exit 1; fi'
Why I use grep? Because you can easily filter out binary files with -I.
Then the usual xargs/tail thingy found in other answers, with the addition to exit with 1 if a file has no newline. So this can be used in a pre-commit githook or CI.
This should do the trick:
#!/bin/bash
for file in `find $1 -type f -name "*.txt"`;
do
nlines=`tail -n 1 $file | grep '^$' | wc -l`
if [ $nlines -eq 1 ]
then echo $file
fi
done;
Call it this way: ./script dir
E.g. ./script /home/user/Documents/ -> lists all text files in /home/user/Documents ending with \n.
This is kludgy; someone surely can do better:
for f in `find . -name '*.txt' -type f`; do
if test `tail -c 1 "$f" | od -c | head -n 1 | tail -c 3` != \\n; then
echo $f;
fi
done
N.B. this answers the question in the title, which is different from the question in the body (which is looking for files that end with \n\n I think).
Most solutions on this page do not work for me (FreeBSD 10.3 amd64). Ian Will's
OSX solution does almost-always work, but is pretty difficult to follow : - (
There is an easy solution that almost-always works too : (if $f is the file) :
sed -i '' -e '$a\' "$f"
There is a major problem with the sed solution : it never gives you the
opportunity to just check (and not append a newline).
Both the above solutions fail for DOS files. I think the most
portable/scriptable solution is probably the easiest one,
which I developed myself : - )
Here is that elementary sh script which combines file/unix2dos/tail. In
production, you will likely need to use "$f" in quotes and fetch tail output
(embedded into the shell variable named last) as \"$f\"
if file $f | grep 'ASCII text' > /dev/null; then
if file $f | grep 'CRLF' > /dev/null; then
type unix2dos > /dev/null || exit 1
dos2unix $f
last="`tail -c1 $f`"
[ -n "$last" ] && echo >> $f
unix2dos $f
else
last="`tail -c1 $f`"
[ -n "$last" ] && echo >> $f
fi
fi
Hope this helps someone.
This example
Works on macOS (BSD) and GNU/Linux
Uses standard tools: find, grep, sh, file, tail, od, tr
Supports paths with spaces
Oneliner:
find . -type f -exec sh -c 'file -b "{}" | grep -q text' \; -exec sh -c '[ "$(tail -c 1 "{}" | od -An -a | tr -d "[:space:]")" != "nl" ]' \; -print
More readable version
Find under current directory
Regular files
That 'file' (brief mode) considers text
Whose last byte (tail -c 1) is not represented by od's named character "nl"
And print their paths
#!/bin/sh
find . \
-type f \
-exec sh -c 'file -b "{}" | grep -q text' \; \
-exec sh -c '[ "$(tail -c 1 "{}" | od -An -a | tr -d "[:space:]")" != "nl" ]' \; \
-print
Finally, a version with a -f flag to fix the offending files (requires bash).
#!/bin/bash
# Finds files without final newlines
# Pass "-f" to also fix those files
fix_flag="$([ "$1" == "-f" ] && echo -true || echo -false)"
find . \
-type f \
-exec sh -c 'file -b "{}" | grep -q text' \; \
-exec sh -c '[ "$(tail -c 1 "{}" | od -An -a | tr -d "[:space:]")" != "nl" ]' \; \
-print \
$fix_flag \
-exec sh -c 'echo >> "{}"' \;
Another option:
$ find . -name "*.txt" -print0 | xargs -0I {} bash -c '[ -z "$(tail -n 1 {})" ] && echo {}'
Since your question has the perl tag, I'll post an answer which uses it:
find . -type f -name '*.txt' -exec perl check.pl {} +
where check.pl is the following:
#!/bin/perl
use strict;
use warnings;
foreach (#ARGV) {
open(FILE, $_);
seek(FILE, -2, 2);
my $c;
read(FILE,$c,1);
if ( $c ne "\n" ) {
print "$_\n";
}
close(FILE);
}
This perl script just open, one per time, the files passed as parameters and read only the next-to-last character; if it is not a newline character, it just prints out the filename, else it does nothing.
This example works for me on OSX (many of the above solutions did not)
for file in `find . -name "*.java"`
do
result=`od -An -tc -j $(( $(ls -l $file | awk '{print $5}') - 1 )) $file`
last_char=`echo $result | sed 's/ *//'`
if [ "$last_char" != "\n" ]
then
#echo "Last char is .$last_char."
echo $file
fi
done
Here another example using little bash build-in commands and which:
allows you to filter for extension (e.g. | grep '\.md$' filters only the md files)
pipe more grep commands for extending the filter (like exclusions | grep -v '\.git' to exclude the files under .git
use the full power of grep parameters to for more filters or inclusions
The code basically, iterates (for) over all the files (matching your chosen criteria grep) and if the last 1 character of a file (-n "$(tail -c -1 "$file")") is not not a blank line, it will print the file name (echo "$file").
The verbose code:
for file in $(find . | grep '\.md$')
do
if [ -n "$(tail -c -1 "$file")" ]
then
echo "$file"
fi
done
A bit more compact:
for file in $(find . | grep '\.md$')
do
[ -n "$(tail -c -1 "$file")" ] && echo "$file"
done
and, of course, the 1-liner for it:
for file in $(find . | grep '\.md$'); do [ -n "$(tail -c -1 "$file")" ] && echo "$file"; done