How To Combine Two Json Values - json

I have a text file with several hundred json entries.The first two entries look like this
{
"ip": "127.0.0.1",
"timestamp": "1565343832",
"ports": [{
"port": 80,
"proto": "tcp",
"status": "open",
"reason": "syn-ack",
"ttl": 245
}]
}
{
"ip": "127.0.0.2",
"timestamp": "1565343837",
"ports": [{
"port": 81,
"proto": "tcp",
"status": "open",
"reason": "syn-ack",
"ttl": 43
}]
}
I would like to parse this json file and combine the values for ip and port and put them in a text file.So the text file will have entries like this
127.0.0.1:80
127.0.0.2:81
How can this be done

Here is a C++ code example, use of boost::property_tree we can read json data and parse
ofstream outfile;
outfile.open ("entries.txt");
for (/*for loop for each entry*/) {
std::stringstream ss;
//assign input values to ss per entry
boost::property_tree::ptree pt;
boost::property_tree::read_json(ss, pt);
std::string IP = pt.get<std::string>("ip");
std::string PORT = pt.get<std::string>("ports.port");
outfile << IP << ":" <<PORT;
}

Related

JQ - Remove Duplicate Array Values

Edited for better clarity
I am using the following jq query to extract the AWS ARN and associated protocols. However I only need the ARN to be listed once followed by the ports and protocols
my code is jq -r '.Listeners[] | (.LoadBalancerArn), (.Protocol)' and the results are
"arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde"
"HTTP"
"arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde"
"HTTP"
"arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde"
"HTTPS"
I have tried everything including unique, first, unique_by, select, contains, etc.. and the results are always "Cannot iterate over string" or number
Desired results
"arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde"
"HTTP"
"HTTP"
"HTTPS"
Sample JSON
{
"Listeners": [
{
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde",
"Port": 9090,
"Protocol": "HTTP"
},
{
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde",
"Port": 80,
"Protocol": "HTTP"
},
{
"LoadBalancerArn": "arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde",
"Port": 443,
"Protocol": "HTTPS"
}
]
}
Group by the common field and iterate over the groups, then output the common field of the first (which is the same for the whole group), and iterate again to output other fields from the same group:
jq -r '.Listeners | group_by(.LoadBalancerArn)[]
| .[0].LoadBalancerArn, .[].Protocol'
arn:aws:elasticloadbalancing:us-xxxx-1:123456789:loadbalancer/app/msword-123456789/20b73abcde
HTTP
HTTP
HTTPS
Demo
unique works on an array, so you'll need to create one with all the LoadBalancerArn first, then call unique and get the first of the remaining array:
.Listeners | map(.LoadBalancerArn) | unique | first
Does this produce what you expected ?
jq -r '.Listeners |
group_by(.LoadBalancerArn)[] |
first |
"\(.LoadBalancerArn) \(.Protocol)"
' input.json
You say you want the ARN listed once, followed by the ports and protocols. You don't give such an example, so it's unclear whether there might be multiple different ARNs between the items in the array of listeners. Let's assume that there might well be multiple different ARNs, and for that reason I'll use slightly different test data:
{
"Listeners": [
{
"LoadBalancerArn": "arn:example:BLUE",
"Port": 9090,
"Protocol": "HTTP"
},
{
"LoadBalancerArn": "arn:example:GOLD",
"Port": 80,
"Protocol": "HTTP"
},
{
"LoadBalancerArn": "arn:example:GOLD",
"Port": 443,
"Protocol": "HTTPS"
}
]
}
group_by is the JQ function for collecting together all the items in an array that share a particular value for some expression. So you could use
.Listeners|group_by(.LoadBalancerArn)
to get an array of arrays of objects, where all of the objects in each inner array has the same value of LoadBalancerArn:
[
[
{
"LoadBalancerArn": "arn:example:BLUE",
"Port": 9090,
"Protocol": "HTTP"
}
],
[
{
"LoadBalancerArn": "arn:example:GOLD",
"Port": 80,
"Protocol": "HTTP"
},
{
"LoadBalancerArn": "arn:example:GOLD",
"Port": 443,
"Protocol": "HTTPS"
}
]
]
From there you can safely pick out the ARN from the first object in each list, knowing that the rest must have the same value:
.Listeners|group_by(.LoadBalancerArn)[](first|{LoadBalancerArn})+{Listeners:map(del(.LoadBalancerArn))}
{
"LoadBalancerArn": "arn:example:BLUE",
"Listeners": [
{
"Port": 9090,
"Protocol": "HTTP"
}
]
}
{
"LoadBalancerArn": "arn:example:GOLD",
"Listeners": [
{
"Port": 80,
"Protocol": "HTTP"
},
{
"Port": 443,
"Protocol": "HTTPS"
}
]
}

JQ if then statement scope

I'd like to use JQ to grab only the sub-records that match an if-then statement. When I use
jq 'if .services[].banner == "FQMDAAICCg==" then .services[].port else empty end
it grabs all of the ports for the record. (there are multiple services under each record and I want to restrict my then statement to only the services scope where I actually found the if condition).
How do I just get the port, banner, etc. for the specific service underneath the record which hit my condition?
example:
{
"services": [
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "PCFET0NUWVBFIEhU",
"certificate": null,
"timestamp": "2020-03-22T00:38:01.074Z",
"protocol": null,
"port": 4444
},
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "SFRUUC8xLjEgMzA",
"certificate": null,
"timestamp": "2020-03-19T01:39:45.288Z",
"protocol": null,
"port": 8080
},
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "FQMDAAICCg==",
"certificate": null,
"timestamp": "2020-03-19T01:39:45.288Z",
"protocol": null,
"port": 8085
},
{
"tls_detected": false,
"banner_is_raw": false,
"transport_protocol": "tcp",
"banner": "Q2FjaGUtQ29ud",
"certificate": null,
"timestamp": "2020-03-20T04:25:24Z",
"protocol": "http",
"port": 8080
}
],
"ip": "103.238.62.68",
"autonomous_system": {
"description": "CHAPTECH-AS-AP Chaptech Pty Ltd",
"asn": 133493,
"routed_prefix": "103.238.62.0/24",
"country_code": "AU",
"name": "CHAPTECH-AS-AP Chaptech Pty Ltd",
"path": [
11164,
3491,
63956,
7594,
7594,
7594,
7594,
133493
]
},
"location": {
"country_code": "AU",
"registered_country": "Australia",
"registered_country_code": "AU",
"continent": "Oceania",
"timezone": "Australia/Sydney",
"latitude": -33.494,
"longitude": 143.2104,
"country": "Australia"
}
}
Update:
Thanks to peak but I couldn't get the additional goals bit working below. I ended up using
jq 'select(.services[].banner == "FQMDAAICCg==") | {port: .services[].port, banner: .services[].banner, ip: .ip}' censys.json | jq 'if .banner == "FQMDAAICCg==" then .ip,.port else empty end'
which is ugly but did the trick and still allowed me to stream the data to the first filter.
Original question
How do I just get the port, banner, etc. for the specific service underneath the record which hit my condition?
To get just the "port" for the service matching the condition, you could modify your query:
.services[]
| if .banner == "FQMDAAICCg==" then .port else empty end
Equivalently:
.services[]
| select(.banner == "FQMDAAICCg==")
| .port
Additional goal
I want to end up in this example with '8085' + '103.238.62.68'
If you really want the two values in that format, you could write something along the following lines, invoking jq with the -r option:
.ip as $ip
| (.services[] | select(.banner == "FQMDAAICCg==") | .port) as $port
| "'\($port)' + '\($ip)'"
or more briefly but less readably:
"'\(.services[] | select(.banner == "FQMDAAICCg==") | .port)' + '\(.ip)'"

JSON parse Windows file path

I want to specify a windows file path in my remote-ftp Atom configuration to reference my private key but I get some parsing issues. Here's my configuration:
{
"protocol": "sftp",
"host": "somehost.com",
"port": 22,
"user": "haha",
"pass": "testpass",
"promptForPass": false,
"remote": "/",
"local": "",
"agent": "",
"privatekey": "C:\Users\haha\Documents\Keys\test_private_key.ppk",
"passphrase": "",
"hosthash": "",
"ignorehost": true,
"connTimeout": 10000,
"keepalive": 10000,
"keyboardInteractive": false,
"keyboardInteractiveForPass": false,
"remoteCommand": "",
"remoteShell": "",
"watch": [],
"watchTimeout": 500
}
With the error:
Parse Error: "privatekey": "C:\Users\haha\Docum
So any idea how i escape Windows file paths for JSON parsing?
Your JSON is invalid, since backslashes need to be escaped per spec.
A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes
Hence, the value of privateKey should be "C:\\Users\\haha\\Documents\\Keys\\test_private_key.ppk".

How we will take the keys from a json in the original order?

I have a JSON file, I want to process that JSON data as Key, value pair.
Here is my JSON file
```"users" : {
"abc": {
"ip": "-------------",
"username": "users#gmail.com",
"password": "---------",
"displayname": "-------",
"Mode": "-----",
"phonenumber": "1********1",
"pstndisplay": "+1 *******5"
},
"efg": {
"ip": "-------------",
"username": "user1#gmail.com",
"password": "---------",
"displayname": "-------",
"Mode": "-----",
"phonenumber": "1********1",
"pstndisplay": "+1 *******5"
},
"xyz": {
"ip": "-------------",
"username": "user2#gmail.com",
"password": "---------",
"displayname": "-------",
"Mode": "-----",
"phonenumber": "1********1",
"pstndisplay": "+1 *******5"```
here i tried to get json data
``` ${the file as string}= Get File ${users_json_path}
${parsed}= Evaluate json.loads("""${the file as string}""") json
${properties}= Set Variable ${parsed["users"]}
Log ${properties}
:FOR ${key} IN #{properties}
\ ${sub dict}= Get From Dictionary ${properties} ${key}
\ Log ${sub dict}
\ Signin ${sub dict}[ip] ${sub dict}[username] ${sub dict}[password] ${sub dict}[Mode]
\ Log ${key} is successfully signed in.
Expected Behavior - The keys what I am parsing should be in sequence from JSON file. For example, abc will get a sign in first then efg and xyz.
${key} = abc
${key} = efg
${key} = xyz
Below are the questions:
1) How we will take users from JSON in sequence? Right now it is taking randomly
2) What will be the best logic to achieve that?
I see you tagged the question with python 2.7 - where Bryan Oakely's comment fully holds true, the elements are in random order.
If you upgrade to python 3 though, starting from v3.6 onwards the dictionaries are guaranteed to preserve the insertion order. Thus on parsing it with the json library the result will be the same as in the source string/file.
Alternatively, in v2 you can use OrderedDict to accomplish the same - plus, specifying the object_pairs_hook argument to JSONDecoder - with it you'll basically specify the result will be OrderedDict:
${parsed}= Evaluate json.loads("""${the file as string}""", object_pairs_hook=collections.OrderedDict) json, collections
#!/usr/bin/env python3
import json
def main():
json_str = """
{
"users" : {
"abc": {
"ip": "-------------",
"username": "users#gmail.com",
"password": "---------",
"displayname": "-------",
"Mode": "-----",
"phonenumber": "1********1",
"pstndisplay": "+1 *******5"
},
"efg": {
"ip": "-------------",
"username": "user1#gmail.com",
"password": "---------",
"displayname": "-------",
"Mode": "-----",
"phonenumber": "1********1",
"pstndisplay": "+1 *******5"
}
}
}
"""
json_object = json.loads(json_str)
for line in json_str.split('\n'):
if '"' in line and ':' in line and '{' in line and '"users"' not in line:
key = line.split('"')[1]
print(key, json_object['users'][key])

Reading JSON file content in LINUX

Dears,
Can someone help me out reading the content of JSON file in LINUX machine without using JQ, Python & Ruby. Looking for purely SHELL scripting. We need to iterate the values if multiple records found. In the below case we have 2 set of records, which needs to iterated.
{
"version": [
"sessionrestore",1
],
"windows": [
{
"tabs": [
{
"entries": [
{
"url": "http://orf.at/#/stories/2.../" (http://orf.at/#/stories/2.../%27) ,
"title": "news.ORF.at",
"charset": "UTF-8",
"ID": 9588,
"docshellID": 298,
"docIdentifier": 10062,
"persist": true
},
{
"url": "http://oracle.at/#/stories/2.../" (http://oracle.at/#/stories/2.../%27) ,
"title": "news.at",
"charset": "UTF-8",
"ID": 9589,
"docshellID": 288,
"docIdentifier": 00062,
"persist": false
}
]
}
}
}