I am trying to exploit a binary(64-bit) with gadget chaining technique. I have to pass the address
0x00007ffff7e10cf8
which holds the code for
pop rax;ret
On passing this input in little-endian format, my bash issues a warning saying :
bash: warning: command substitution: ignored null byte in input
and prints only till 7f ignoring the NULL bytes.
To make sure that I got the concept right that bash IGNORES and does not stop parsing on encountering the first NULL byte. I tried echo $(printf "\x00\x55\x44\x33\x22\x00\x34")and the output was
bash: warning: command substitution: ignored null byte in input
UD3"4
So yes, bash simply ignores the NULL bytes and keeps parsing till the end.
Is there a way to actually pass in NULL bytes in bash? I am using bash version 5.0.3
A shell script is not really the recommended way to patch binary files. However, it can be done with dd. Say, you have to write 0x00007ffff7e10cf8 at positions 0xff00-0xff07 of a file, overwriting the bytes which are already there (counting starts from 0). You can use
#!/bin/bash
input='original.bin'
output='patched.bin'
patch='\x00\x00\x7f\xff\xf7\xe1\x0c\xf8' # Keep an eye on endianness!
start="$((16#FF00))"
{
dd if="$input" bs="$start" count=1 2>/dev/null
printf "$patch"
dd if="$input" bs="$((start+${#patch}/4))" skip=1 2>/dev/null
} >"$output"
Related
I have this JSON String:
{"name":"http://someUrl/ws/someId","id":"someId"}
I just want to get value for "id" key and store it in some variable. I succesfully tried using jq. But due to some constraints, I need to achieve this just by using grep and string matching.
I tried this so far: grep -Po '"id":.*?[^\\]"'; But that is giving "id":"ws-4c906698-03a2-49c3-8b3e-dea829c7fdbe" as output. I just need the id value. Please help
With a PCRE regex, you may use lookarounds. Thus, you need to put "id":" into the positive lookbehind construct, and then match 1 or more chars other than ":
grep -Po '(?<="id":")[^"]+'
where
(?<="id":") - requires a "id":" to appear immediately to the left of the current position (but the matched text is not added to the match value) and
[^"]+ - matches and adds to the match 1 or more chars other than ".
To get the values with escaped quotes:
grep -Po '(?<="id":")[^"\\]*(?:\\.[^"\\]*)*'
Here, (?<="id":") will still match the position right after "id":" and then the following will get matched:
[^"\\]* - zero or more chars other than " and \
(?:\\.[^"\\]*)* - zero or more consequent sequences of:
\\. - a \ and any char (any escape sequence)
[^"\\]* - zero or more chars other than " and \
See Jshon, it is a command line Json parser for shell script usage.
echo '{"name":"http://someUrl/ws/someId","id":"someId"}' | jshon -e id
"someId"
Just noticed I read past the section stating you needed to use standard tools available, if your admin doesn't allow Jshon it is very likely that the system will have Python available which you could use.
echo '{"name":"http://someUrl/ws/someId","id":"someId"}' | python -c 'import sys, json; print json.load(sys.stdin)["id"]'
someId
Using grep for this is just asking for trouble, I would avoid it and opt for a proper Json parser as above.
I am running the following command:
sudo clustat | grep primary | awk 'NF{print $1",""server:"$2 ",""status:"$3}'
Results are:
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
My desired result is:
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
I can't seem to put the qoutation marks withour srewing up my output.
Use jq:
sudo clustat | grep primary |
jq -R 'split(" ")|{service:.[0], server:.[1], status:.[2]}'
The input is read as raw text, not JSON. Each line is split on a space (the argument to split may need to be adjusted depending on the actual input). jq ensures that values are properly quoted when constructing the output objects.
Don't do this: Instead, use #chepner's answer, which is guaranteed to generate valid JSON as output with all possible inputs (or fail with a nonzero exit status if no JSON representation is possible).
The below is only tested to generate valid JSON with the specific inputs shown in the question, and will quite certainly generate output that is not valid JSON with numerous possible inputs (strings with literal quotes, strings ending in literal backslashes, etc).
sudo clustat |
awk '/primary/ {
print "{\"service\":\"" $1 "\",\"server\":\"" $2 "\",\"status\":\""$3"\"}"
}'
For JSON conversion of common shell commands, a good option is jc (JSON Convert)
There is no parser for clustat yet though.
clustat output does look table-like, so you may be able to use the --asciitable parser with jc.
I'm creating a Bash script to parse the air pollution levels from the webpage:
http://aqicn.org/city/beijing/m/
There is a lot of stuff in the file, but this is the relevant bit:
"iaqi":[{"p":"pm25","v":[59,21,112],"i":"Beijing pm25 (fine
particulate matter) measured by U.S Embassy Beijing Air Quality
Monitor
(\u7f8e\u56fd\u9a7b\u5317\u4eac\u5927\u4f7f\u9986\u7a7a\u6c14\u8d28\u91cf\u76d1\u6d4b).
Values are converted from \u00b5g/m3 to AQI levels using the EPA
standard."},{"p":"pm10","v":[15,5,69],"i":"Beijing pm10
(respirable particulate matter) measured by Beijing Environmental
Protection Monitoring Center
I want the script to parse and display 2 numbers: current PM2.5 and PM10 levels (the numbers in bold in the above paragraph).
CITY="beijing"
AQIDATA=$(wget -q 0 http://aqicn.org/city/$CITY/m/ -O -)
PM25=$(awk -v FS="(\"p\":\"pm25\",\"v\":\\\[|,[0-9]+)" '{print $2}' <<< $AQIDATA)
PM100=$(awk -v FS="(\"p\":\"pm10\",\"v\":\\\[|,[0-9]+)" '{print $2}' <<< $AQIDATA)
echo $PM25 $PM100
Even though I can get PM2.5 levels to display correctly, I cannot get PM10 levels to display. I cannot understand why, because the strings are similar.
Anyone here able to explain?
The following approach is based on two steps:
(1) Extracting the relevant JSON;
(2) Extracting the relevant information from the JSON using a JSON-aware tool -- here jq.
(1) Ideally, the web service would provide a JSON API that would allow one to obtain the JSON directly, but as the URL you have is intended for viewing with a browser, some form of screen-scraping is needed. There is a certain amount of brittleness to such an approach, so here I'll just provide something that currently works:
wget -O - http://aqicn.org/city/beijing/m |
gawk 'BEGIN{RS="function"}
$1 ~/getAqiModel/ {
sub(/.*var model=/,"");
sub(/;return model;}/,"");
print}'
(gawk or an awk that supports multi-character RS can be used; if you have another awk, then first split on "function", using e.g.:
sed $'s/function/\\\n/g' # three backslashes )
The output of the above can be piped to the following jq command, which performs the filtering envisioned in (2) above.
(2)
jq -c '.iaqi | .[]
| select(.p? =="pm25" or .p? =="pm10") | [.p, .v[0]]'
The result:
["pm25",59]
["pm10",15]
I think your problem is that you have a single line HTML file that contains a script that contains a variable that contains the data you are looking for.
Your field delimiters are either "p":"pm100", "v":[ or a comma and some digits.
For pm25 this works, because it is the first, and there are no occurrences of ,21 or something similar before it.
However, for pm10, there are some that are associated with pm25 ahead of it. So the second field contains the empty string between ,21 and ,112
#karakfa has a hack that seems to work -- but he doesn't explain very well why it works.
What he does is use awk's record separator (which is usually a newline) and sets it to either of :, ,, or [. So in your case, one of the records would be "pm25", because it is preceded by a colon, which is a separator, and succeeded by a comma, also a separator.
Once it hits the matching content ("pm25") it sets a counter to 4. Then, for this and the next records, it counts this counter down. "pm25" itself, "v", the empty string between : and [, and finally reaches one when hitting the record with the number you want to output: 4 && ! 3 is false, 3 && ! 2 is false, 2 && ! 1 is false, but 1 && ! 0 is true. Since there is no execution block, awk simply prints this record, which is the value you want.
A more robust work would probably be using xpath to find the script, then use some json parser or similar to get the value.
chw21's helpful answer explains why your approach didn't work.
peak's helpful answer is the most robust, because it employs proper JSON parsing.
If you don't want to or can't use third-party utility jq for JSON parsing, I suggest using sed rather than awk, because awk is not a good fit for field-based parsing of this data.
$ sed -E 's/^.*"pm25"[^[]+\[([0-9]+).+"pm10"[^[]+\[([0-9]+).*$/\1 \2/' <<< "$AQIDATA"
59 15
The above should work with both GNU and BSD/OSX sed.
To read the result into variables:
read pm25 pm10 < \
<(sed -E 's/^.*"pm25"[^[]+\[([0-9]+).+"pm10"[^[]+\[([0-9]+).*$/\1 \2/' <<< "$AQIDATA")
Note how I've chosen lowercase variable names, because it's best to avoid all upper-case variables in shell programming, so as to avoid conflicts with special shell and environment variables.
If you can't rely on the order of the values in the source string, use two separate sed commands:
pm25=$(sed -E 's/^.*"pm25"[^[]+\[([0-9]+).*$/\1/' <<< "$AQIDATA")
pm10=$(sed -E 's/^.*"pm10"[^[]+\[([0-9]+).*$/\1/' <<< "$AQIDATA")
awk to the rescue!
If you have to, you can use this hacky way using smart counters with hand-crafted delimiters. Setting RS instead of FS transfers looping through fields to awk itself. Multi-char RS is not available for all awks (gawk supports it).
$ awk -v RS='[:,[]' '$0=="\"pm25\""{c=4} c&&!--c' file
59
$ awk -v RS='[:,[]' '$0=="\"pm10\""{c=4} c&&!--c' file
15
I am navigating a Java-based CLI menu on a remote machine with expect inside a bash script and I am trying to extract something from the output without leaving the expect session.
Expect command in my script is:
expect -c "
spawn ssh user#host
expect \"#\"
send \"java cli menu command here\r\"
expect \"java cli prompt\"
send \"java menu command\"
"
###I want to extract a specific string from the above output###
Expect output is:
Id Name
-------------------
abcd 12 John Smith
I want to extract abcd 12 from the above output into another expect variable for further use within the expect script. So that's the 3rd line, first field by using a double-space delimiter. The awk equivalent would be: awk -F ' ' 'NR==3 {$1}'
The big issue is that the environment through which I am navigating with Expect is, as I stated above, a Java CLI based menu so I can't just use awk or anything else that would be available from a bash shell.
Getting out from the Java menu, processing the output and then getting in again is not an option as the login process lasts for 15 seconds so I need to remain inside and extract what I need from the output using expect internal commands only.
You can use regexp in expect itself directly with the use of -re flag. Thanks to Donal on pointing out the single quote and double quote issues. I have given solution using both ways.
I have created a file with the content as follows,
Id Name
-------------------
abcd 12 John Smith
This is nothing but your java program's console output. I have tested this in my system with this. i.e. I just simulated your program's output with cat. You just replace the cat code with your program commands. Simple. :)
Double Quotes :
#!/bin/bash
expect -c "
spawn ssh user#domain
expect \"password\"
send \"mypassword\r\"
expect {\\\$} { puts matched_literal_dollar_sign}
send \"cat input_file\r\"; # Replace this code with your java program commands
expect -re {-\r\n(.*?)\s\s}
set output \$expect_out(1,string)
#puts \$expect_out(1,string)
puts \"Result : \$output\"
"
Single Quotes :
#!/bin/bash
expect -c '
spawn ssh user#domain
expect "password"
send "mypasswordhere\r"
expect "\\\$" { puts matched_literal_dollar_sign}
send "cat input_file\r"; # Replace this code with your java program commands
expect -re {-\r\n(.*?)\s\s}
set output $expect_out(1,string)
#puts $expect_out(1,string)
puts "Result : $output"
'
As you can see, I have used {-\r\n(.*?)\s\s}. Here the braces prevent any variable substitutions. In your output, we have a 2nd line with full of hyphens. Then a newline. Then your 3rd line content. Let's decode the regex used.
-\r\n is to match one literal hyphen and a new line together. This will match the last hyphen in the 2nd line and the newline which in turn make it to 3rd line now. So, .*? will match the required output (i.e. abcd 12) till it encounters double space which is matched by \s\s.
You might be wondering why I need parenthesis which is used to get the sub-match patterns.
In general, expect will save the expect's whole match string in expect_out(0,string) and buffer all the matched/unmatched input to expect_out(buffer). Each sub match will be saved in subsequent numbering of string such as expect_out(1,string), expect_out(2,string) and so on.
As Donal pointed out, it is better to use single quote's approach since it looks less messy. :)
It is not required to escape the \r with the backslash in case of double quotes.
Update :
I have changed the regexp from -\r\n(\w+\s+\w+)\s\s to -\r\n(.*?)\s\s.
With this way - your requirement - such as match any number of letters and single spaces until you encounter first occurrence of double spaces in the output
Now, let's come to your question. You have mentioned that you have tried -\r\n(\w+)\s\s. But, there is a problem here with \w+. Remember \w+ will not match space character. Your output has some spaces in it till double spaces.
The use of regexp will matter based on your requirements on the input string which is going to get matched. You can customize the regular expressions based on your needs.
Update version 2 :
What is the significance of .*?. If you ask separately, I am going to repeat what you commented. In regular expressions, * is a greedy operator and ? is our life saver. Let us consider the string as
Stackoverflow is already overflowing with number of users.
Now, see the effect of the regular expression .*flow as below.
* matches any number of characters. More precisely, it matches the longest string possible while still allowing the pattern itself to match. So, due to this, .* in the pattern matched the characters Stackoverflow is already over and flow in pattern matched the text flow in the string.
Now, in order to prevent the .* to match only up to the first occurrence of the string flow, we are adding the ? to it. It will help the pattern to behave as non-greedy manner.
Now, again coming back to your question. If we have used .*\s\s, then it will match the whole line since it is trying to match as much as possible. This is common behavior of regular expressions.
Update version 3:
Have your code in the following way.
x=$(expect -c "
spawn ssh user#host
expect \"password\"
send \"password\r\"
expect {\\\$} { puts matched_literal_dollar_sign}
send \"cat input\r\"
expect -re {-\r\n(.*?)\s\s}
if {![info exists expect_out(1,string)]} {
puts \"Match did not happen :(\"
exit 1
}
set output \$expect_out(1,string)
#puts \$expect_out(1,string)
puts \"Result : \$output\"
")
y=$?
# $x now contains the output from the 'expect' command, and $y contains the
# exit status
echo $x
echo $y;
If the flow happened properly, then exit code will have value as 0. Else, it will have 1. With this way, you can check the return value in bash script.
Have a look at here to know about the info exists command.
How do you process a Chrome Native Messaging API-call with a bash script?
I succeeded in doing it with python with this example
Sure I can call bash from the python code with subprocess, but is it possible to skip python and process the message in bash directly?
The problematic part is reading the JSON serialized message into a variable. The message is serialized using JSON, UTF-8 encoded and is preceded with 32-bit message length in native byte order through stdin.
echo $* only outputs:
chrome-extension://knldjmfmopnpolahpmmgbagdohdnhkik/
Also something like
read
echo $REPLY
doesn't output anything. No sign of the JSON message. Python uses struct.unpack for this. Can that be done in bash?
I suggest to not use (bash) shell scripts as a native messaging host, because bash is too limited to be useful.
read without any parameters reads a whole line before terminating, while the native messaging protocol specifies that the first four bytes specify the length of the following message (in native byte order).
Bash is a terrible tool for processing binary data. An improved version of your read command would specify the -n N parameter to stop reading after N characters (note: not bytes) and -r to remove some processing. E.g. the following would store the first four characters in a variable called var_prefix:
IFS= read -rn 4 var_prefix
Even if you assume that this stores the first four bytes in the variable (it does not!), then you have to convert the bytes to an integer. Did I already mention that bash automatically drops all NUL bytes? This characteristics makes Bash utterly worthless for being a fully capable native messaging host.
You could cope with this shortcoming by ignoring the first few bytes, and start parsing the result when you spot a { character, the beginning of the JSON-formatted request. After this, you have to read all input until the end of the input is found. You need a JSON parser that stops reading input when it encounters the end of the JSON string. Good luck with writing that.
Generating output is a easier, just use echo -n or printf.
Here is a minimal example that assumes that the input ends with a }, reads it (without processing) and replies with a result. Although this demo works, I strongly recommend to not use bash, but a richer (scripting) language such as Python or C++.
#!/bin/bash
# Loop forever, to deal with chrome.runtime.connectNative
while IFS= read -r -n1 c; do
# Read the first message
# Assuming that the message ALWAYS ends with a },
# with no }s in the string. Adopt this piece of code if needed.
if [ "$c" != '}' ] ; then
continue
fi
message='{"message": "Hello world!"}'
# Calculate the byte size of the string.
# NOTE: This assumes that byte length is identical to the string length!
# Do not use multibyte (unicode) characters, escape them instead, e.g.
# message='"Some unicode character:\u1234"'
messagelen=${#message}
# Convert to an integer in native byte order.
# If you see an error message in Chrome's stdout with
# "Native Messaging host tried sending a message that is ... bytes long.",
# then just swap the order, i.e. messagelen1 <-> messagelen4 and
# messagelen2 <-> messagelen3
messagelen1=$(( ($messagelen ) & 0xFF ))
messagelen2=$(( ($messagelen >> 8) & 0xFF ))
messagelen3=$(( ($messagelen >> 16) & 0xFF ))
messagelen4=$(( ($messagelen >> 24) & 0xFF ))
# Print the message byte length followed by the actual message.
printf "$(printf '\\x%x\\x%x\\x%x\\x%x' \
$messagelen1 $messagelen2 $messagelen3 $messagelen4)%s" "$message"
done