Replace a string if it is not followed by another string - json

I would like to replace the fortawesome string (if it is not followed by the /fontawesome-common-type string) by the stephane string.
sed -e 's,"#fortawesome(/^fontawesome-common-types+),"#stephaneeybert\1,g'
sed: -e expression #1, char 65: invalid reference \1 on `s' command's RHS
An example input:
"#fortawesome/fontawesome-common-types": "^0.2.32"
"name": "#fortawesome/pro-duotone-svg-icons",
And its expected output:
"#fortawesome/fontawesome-common-types": "^0.2.32"
"name": "#stephane/pro-duotone-svg-icons",
UPDATE: I went with the simple alternative of using an intermediate variable:
EXCLUDE=fontawesome-common-types
BUFFER=EkSkLUdE
cat package/package.json \
| sed -e "s,\"#$REPO_SOURCE/$EXCLUDE,\"#$BUFFER,g" \
| sed -e "s,\"#$REPO_SOURCE,\"#$REPO_DEST,g" \
| sed -e "s,\"#$BUFFER,\"#$REPO_SOURCE/$EXCLUDE,g" \
> package/package.out.json;

sed doesn't support negative lookahead functionality. Other than the obvious perl fallback that supports lookaheads, uou may use this awk as a work-around:
awk -F 'fortawesome' -v OFS='stephane' 'NF > 1 {
s = ""
for (i=1; i<NF; ++i)
s = s $i ($(i+1) ~ /^\/fontawesome-common-type/ ? FS : OFS)
$0 = s $i
} 1' file
This awk uses fortawesome as input field separator and stephane as OFS
NF > 1 will be true when we have fortawesome in a line
we loop through fields split by fortawesome and keep track of next field
if next field starts with /fontawesome-common-type then we keep same FS otherwise use OFS

Use temporary values:
exclude='fortawesome/fontawesome-common-type';
match='fortawesome';
repl='stephane';
tmpvar='EkSkLUdE';
sed "s#$exclude#$tmpvar#g;s#$match#$repl#g;s#$tmpvar#$exclude#g" file > newfile
All cases of exclude are replaced with tmpvars, then real expected matches are replaced with repls, and then tmpvars are changed back to excludes.

Related

Unix command/s or Tcl proc to convert hex data to binary with output 1 bit per line

I have a plain text file with hex data information (32-bit word per line). Example :
cafef00d
deadbeef
That I need to convert to this :
11001010111111101111000000001101
11011110101011011011111011101111
BUT with 1 bit per line only. Starting from the LSB of the first 32-bit hex and so on. Final output file will be :
1
0
1
1
... and so on
Is there a unix command/s or can I do this in a Tcl proc ?
A tcl solution...
Assuming you've read the file into a string, the first thing is to convert the hex strings into numbers expressed in binary, LSB first. There's a few ways to do it, here's one (I like scan and format):
set binaryData [lmap hexValue $inputData {
scan $hexValue "%x" value
string reverse [format "%b" $value]
}]
For your input, that produces:
10110000000011110111111101010011 11110111011111011011010101111011
We can then convert that to be one digit per line with this:
set oneDigitPerLine [join [split [join $binaryData ""] ""] "\n"]
The innermost join gets rid of the whitespace, the split breaks it up into characters, and the outer join inserts the newline separators. (I'll not produce the result here.)
If you want to do it with linux commands, try the following:
tac: reverse text lines in a file
fold -w 1: fold a text file, column width 1
sed: replace strings
tac input_file | \
fold -w 1 | \
sed -e 's/0/0000/' | \
sed -e 's/1/0001/' | \
sed -e 's/2/0010/' | \
sed -e 's/3/0011/' | \
sed -e 's/4/0100/' | \
sed -e 's/5/0101/' | \
sed -e 's/6/0110/' | \
sed -e 's/7/0111/' | \
sed -e 's/8/1000/' | \
sed -e 's/9/1001/' | \
sed -e 's/a/1010/' | \
sed -e 's/b/1011/' | \
sed -e 's/c/1100/' | \
sed -e 's/d/1101/' | \
sed -e 's/e/1110/' | \
sed -e 's/f/1111/' | \
told -w 1 | \
tac
Another way, using a perl one-liner:
$ perl -nE 'say for split "", reverse sprintf("%032b", hex)' < input.txt
1
0
1
1
...
For each line, converts from a base-16 string to a number and turns that into a binary string, and then prints each individual character on its own line.

How can I sort a Bash output and save it to a sql db

I've been trying to get the data from this command ioreg -r -c "AppleSmartBattery" and save each one of its inputs to a sql db
$ ioreg -r -c "AppleSmartBattery"
+-o AppleSmartBattery <class AppleSmartBattery, id 0x1000222c9, registered, ma$
{
"TimeRemaining" = 179
"AvgTimeToEmpty" = 179
"AdapterDetails" = {"FamilyCode"=0}
"ChargingOverride" = 0
"AppleRawCurrentCapacity" = 2373
"InstantTimeToEmpty" = 154
"AppleRawMaxCapacity" = 3811
"ExternalChargeCapable" = No
I would need to save it to a sql table, where one column is "*" and the next one is the value after the equal
I was trying to build a "for loop", I got this far I cant figure out how to continue
batstat=$(ioreg -r -c "AppleSmartBattery")
for i in ${batstat[#]}; do
sed 's/^[^{]*{\([^{}]*\)}.*/\1/' $i
echo $i
done
I would need to accomplish the following
get one single value in quotes "" out each time the for goes by the line
assign the correct value after the equals sign to the respective quoted value
thanks :)
Not that it's impossible, but I think doing this entirely in a shell script is a bit much when there are easier solutions available.
What I'd do here is convert the output to JSON and then use a Node module like JSON-to-SQL to generate the table from the JSON schema, and JSON-SQL to convert the output to an INSERT statement, which you can then use with any Node SQL client, like sql-client.
You can also probably parse the output more cleanly and easily in Node using a module like sh to capture the ioreg command output, but here's what I came up with for converting the command output into valid JSON.
#!/bin/bash
function parseData() {
tail -n +2 $1 | \
sed -re 's/\=/\:/g' | \
sed -re 's/</\"/g' | \
sed -re 's/>/\"/g' | \
sed -re 's/No/false/g' | \
sed -re 's/Yes/true/g' | \
sed -re 's/\(/\[/g' | \
sed -re 's/\)/\]/g' | \
sed '$d' | \
sed '$d' | \
sed 's/$/,/' | \
sed '1 s/\,//' | \
sed '$ s/\,//' | \
sed '52 s/,//'
}
ioreg -r -c "AppleSmartBattery" | parseData
The only issue is if the number of lines in the output ever changes, the 52 in the last line of the parseData function would need to be updated.

Arithmetic in web scraping in a shell

so, I have the example code here:
#!/bin/bash
clear
curl -s https://www.cnbcindonesia.com/market-data/currencies/IDR=/USD-IDR |
html2text |
sed -n '/USD\/IDR/,$p' |
sed -n '/Last updated/q;p' |
tail -n-1 |
head -c+6 && printf "\n"
exit 0
this should print out some number range 14000~15000
lets start from the very basic one, what I have to do in order to print result + 1 ? so if the printout is 14000 and increment it to 1 become 14001. I suppose the result of the html2text is not calculatable since it should be something like string output not integer.
the more advance thing i want to know is how to calculate the result of 2 curl results?
What I would do, bash + xidel:
$ num=$(xidel -se '//div[#class="mark_val"]/span[1]/text()' 'https://url')
$ num=$((${num//,/}+1)) # num was 14050
$ echo $num
Output
14051
 Explanations
$((...))
is an arithmetic substitution. After doing the arithmetic, the whole thing is replaced by the value of the expression. See http://mywiki.wooledge.org/ArithmeticExpression
Command Substitution: "$(cmd "foo bar")" causes the command 'cmd' to be executed with the argument 'foo bar' and "$(..)" will be replaced by the output. See http://mywiki.wooledge.org/BashFAQ/002 and http://mywiki.wooledge.org/CommandSubstitution
Bonus
You can compute directly in xidel, thanks Reino using xquery syntax :
$ xidel -s <url> e 'replace(//div[#class="mark_val"]/span[1],",","") + 1'
And to do addition arithmetic of 2 values :
$ xidel -s <url> -e '
let $num:=replace(//div[#class="mark_val"]/span[1],",","")
return $num + $num
'

How to grep specific value from JSON file?

I have a JSON file and content like below:
[
{
"id":"54545-f919-4b0f-930c-0117d6e6c987",
"name":"Inventory_Groups",
"path":"/Groups",
"subGroups":[
{
"id":"343534-394b-429a-834e-f8774240d736",
"name":"UserGroup",
"path":"/Groups/UserGroup",
"subGroups":[
]
}
]
}
]
Now I want to grep value of key id from the subGroups area. How to achive this, if id key not duplicate then it can be achieved by:
grep -o '"id": "[^"]*' Group.json | grep -o '[^"]*$'
But in my case how can I get the value of id as it appears two times?
A valid question to ask your employer is why you're in a position to use the shell but not to use appropriate linux packages. Compare:
awk -F '[":,]+' '$2=="subGroups" {f=1} f && $2=="id" {print $3; exit}' file
(Brittle solution, will fail if the structure of your JSON changes)
To:
jq '.[].subGroups[].id' file
Which can handle compact JSON in addition to numerous other realistic complications.
Using just standard UNIX tools and assuming your sed can tolerate input without a terminating newline (otherwise we can swap out the tr for an awk command that keeps the last newline):
$ tr -d '\n' < file | sed 's/.*"subGroups":[^]}]*"id":"\([^"]*\)\".*/\1\n/'
343534-394b-429a-834e-f8774240d736
Alternatively with just a call to any awk:
$ awk '
{ rec = (NR>1 ? rec ORS : "") $0 }
END {
gsub(/.*"subGroups":[^]}]*"id":"|".*/,"",rec)
print rec
}
' file
343534-394b-429a-834e-f8774240d736

Change delimiters in files

Below I have files as they should, and further down, what I made till now. I think that in my code is the source of the problem: delimiters, but I can't get it much better.
My source file is with ; as delimiter, and the files for my database have a , as separator; also, the strings are between "":
The category file should be like this:
"1","1","testcategory","testdescription"
And the manufacturers file, like this:
"24","ASUS",NULL,NULL,NULL
"23","ASROCK",NULL,NULL,NULL
"22","ARNOVA",NULL,NULL,NULL
What I have at this moment:
- category file:
1;2;Alarmen en beveiligingen;
2;2;Apparatuur en toebehoren;
3;2;AUDIO;
- manufacturers file:
315;XTREAMER;NULL;NULL;NULL
316;XTREMEMAC;NULL;NULL;NULL
317;Y-CAM;NULL;NULL;NULL
318;ZALMAN;NULL;NULL;NULL
I tried a bit around to use sed; first, on the categories file:
cut -d ";" -f1 /home/arno/pixtmp/pixtmp.csv |sort | uniq > /home/arno/pixtmp/categories_description-in.csv
sed 's/^/;2;/g' /home/arno/pixtmp/categories_description-in.csv > /home/arno/pixtmp/categories_description-in.tmp
sed -e "s/$/;/" /home/arno/pixtmp/categories_description-in.tmp > /home/arno/pixtmp/categories_description-in.tmp2
awk 'BEGIN{n=1}{printf("%s%s\n",n++,$0)}' /home/arno/pixtmp/categories_description-in.tmp2 > /home/arno/pixtmp/categories_description$
And then on the manufacturers file:
cut -d ";" -f5 /home/arno/pixtmp/pixtmp.csv |sort | uniq > /home/arno/pixtmp/manufacturers-in
sed 's/^/;/g' /home/arno/pixtmp/manufacturers-in > /home/arno/pixtmp/manufacturers-tmp
sed -e "s/$/;NULL;NULL;NULL/" /home/arno/pixtmp/manufacturers-tmp > /home/arno/pixtmp/manufacturers-tmp2
awk 'BEGIN{n=1}{printf("%s%s\n",n++,$0)}' /home/arno/pixtmp/manufacturers-tmp2 > /home/arno/pixtmp/manufacturers.ok
You were trying to solve the problem by using cut, sed, and AWK. AWK by itself is powerful enough to solve your problem.
I wrote one AWK program that can handle both of your examples. If NULL is not a special case, and the manufacturers' file is a different format, you will need to make two AWK programs but I think it should be clear how to do it.
All we do here is tell AWK that the "field separator" is the semicolon. Then AWK splits the input lines into fields for us. We loop over the fields, printing as we go.
#!/usr/bin/awk -f
BEGIN {
FS = ";"
DQUOTE = "\""
}
function add_quotes(s) {
if (s == "NULL")
return s
else
return DQUOTE s DQUOTE
}
NF > 0 {
# if input ended with a semicolon, last field will be empty
if ($NF == "")
NF -= 1 # subtract one from NF to forget the last field
if (NF > 0)
{
for (i = 1; i <= NF - 1; ++i)
printf("%s,", add_quotes($i))
printf("%s\n", add_quotes($i))
}
}