mongoexport - issue with JSON query (extended JSON - Invalid JSON input) - json

I have started learning MongoDB recently. Today the instructor taught us the mongoexport command. While practicing the same, I face a typical issue which none of the other batchmates including the instructor faced. I use MongoDB version 4.2.0 on my Windows 10 machine.
If I use mongoexport for my collection without any -q parameter to specify any filtering condition, it works fine.
mongoexport -d trainingdb -c employee -f empId,name,designation -o \mongoexport\all-employees.json
2019-09-17T18:00:30.300+0530 connected to: mongodb://localhost/
2019-09-17T18:00:30.314+0530 exported 3 records
However, whenever I specify the JSON query as -q (or --query) it gives an error as follows.
mongoexport -d trainingdb -c employee -f empId,name,designation -q {'designation':'Developer'} -o \mongoexport\developers.json
2019-09-17T18:01:45.381+0530 connected to: mongodb://localhost/
2019-09-17T18:01:45.390+0530 Failed: error parsing query as Extended JSON: invalid JSON input
The same error persists in all the different flavors I had attempted with for the query.
-q {'designation':'Developer'}
--query {'designation':'Developer'}
-q "{'designation':'Developer'}"
I had even attempted with a different query condition on the 'empId' as -q {'empId':'1001'} But no luck. I keep getting the same error.
As per one of the suggestions given in the StackOverflow website, I tried with the following option but getting a different error.
-q '{"designation":"Developer"}'
The error is : 'query '[39 123 101 109 112 73 100 58 49 48 48 49 125 39]' is not valid JSON: json: cannot unmarshal string into Go value of type map[string]interface {}'.
2019-09-17T20:24:58.878+0530 query '[39 123 101 109 112 73 100 58 49 48 48 49 125 39]' is not valid JSON: json: cannot unmarshal string into Go value of type map[string]interface {}
2019-09-17T20:24:58.882+0530 try 'mongoexport --help' for more information
I am really not sure what is missing here ? Tried with a bit of Googling and also gone through the official MongoDB documentation of the mongoexport - but no luck.
The employee collection in my system looks like the follows with 3 documents.
> db.employee.find().pretty()
{
"_id" : ObjectId("5d80d1ae0d4d526a42fd95ad"),
"empId" : 1001,
"name" : "Raghavan",
"designation" : "Developer"
}
{
"_id" : ObjectId("5d80d1b20d4d526a42fd95ae"),
"empId" : 1002,
"name" : "Kannan",
"designation" : "Architect"
}
{
"_id" : ObjectId("5d80d1b40d4d526a42fd95af"),
"empId" : 1003,
"name" : "Sathish",
"designation" : "Developer"
}
>
Update
As suggested by #NikosM, I have saved the query in a .json file (query.json) and tried the same mongoexport command with the new approach. Still, no luck. Same Marshal error.
cat query.json
{"designation":"Developer"}
mongoexport -d trainingdb -c employee -f empId,name,designation -q 'query.json' -o \mongoexport\developers.json
2019-09-17T21:16:32.849+0530 query '[39 113 117 101 114 121 46 106 115 111 110 39]' is not valid JSON: json: cannot unmarshal string into Go value of type map[string]interface {}
2019-09-17T21:16:32.852+0530 try 'mongoexport --help' for more information
Any help on this will be highly appreciated.

The following different approach made it work at last - where I had specified the JSON query with the double quotes escaped with the backslash : -q "{\"designation\":\"Developer\"}".
mongoexport -d trainingdb -c employee -f empId,name,designation -q "{\"designation\":\"Developer\"}" -o \mongoexport\developers.json
2019-09-17T21:33:01.642+0530 connected to: mongodb://localhost/
2019-09-17T21:33:01.658+0530 exported 2 records
cat developers.json
{"_id":{"$oid":"5d80d1ae0d4d526a42fd95ad"},"empId":1001.0,"name":"Raghavan","designation":"Developer"}
{"_id":{"$oid":"5d80d1b40d4d526a42fd95af"},"empId":1003.0,"name":"Sathish","designation":"Developer"}
Thank you very much #Caconde. Your suggestion helped.
But I am really not sure why this does not work in my machine alone and the reason for this tweak in the format of the query.

There is another approaches that I found out to work which were using the triple double-quote (""") for outside encasing.
mongoexport -d trainingdb -c employee -f empId,name,designation -q """ {"designation":"Developer"} """ -o \mongoexport\developers.json

The following different approach made it work at last - where I had specified the JSON query with the double quotes escaped with the backslash : -q "{"designation":"Developer"}".
for me it was
"{\"sensor_name\":\"Heat Recovery System Header Mass Flow\"}"
THIS ANSWER SOLVED MY ISSUE TYSM

Related

Unable to parse JSON using jq tool due to node name containing unexpected character [duplicate]

This question already has answers here:
How to use jq when the variable has reserved characters?
(3 answers)
Closed 1 year ago.
I tried with jq to parse some JSON output inside my GitLab CI pipeline so I can extract needed information. I've tried many different ways, but I can't get the desired information out of the target node, because it has special characters and when I get to that node, the pipeline fails in each case. This is the current state of my pipeline.
This is the problematic job:
get results (dev branch):
stage: Results of scanning image
variables:
RESULTS: ""
STATUS: ""
SEVERITY: ""
image: alpine
only:
refs:
- dev
allow_failure: true
before_script:
- apk update && apk upgrade
- apk --no-cache add curl
- apk add jq
script:
- 'RESULTS=$(curl -H "Authorization: Basic `echo -n ${HARBOR_USER}:${HARBOR_PASSWORD} | base64`" -X GET "https://url.to.registry/api/v2.0/projects/project/repositories/repo-name/artifacts/latest?page=1&page_size=10&with_tag=true&with_label=true&with_scan_overview=true&with_signature=true&with_immutable_status=true")'
- echo $RESULTS
- RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
- echo $RESULTS
- "STATUS=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .scan_status'"
- "SEVERITY=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .severity'"
- echo "Printing the results of the image scanning process on Harbor registry:"
- echo "status of scan:$STATUS"
- echo "severity of scan:$SEVERITY"
- echo "For more information of scan results please visit Harbor registry!"
tags:
- dev
- docker
This is the JSON output that I get from the curl command:
{
"addition_links":{
"build_history":{
"absolute":false,
"href":"..."
},
"vulnerabilities":{
"absolute":false,
"href":"...."
}
},
"digest":"sha256:bcd665be2b7c6725b410029db385d7c6c71a9ce557427cbd0f54d01a9",
"extra_attrs":{
"architecture":"amd64",
"author":null,
"created":"2021-10-22T10:28:46.058276455Z",
"os":"linux"
},
"icon":"sha256:0048162a053ee7518615bef084403614f8bca43b40ae2e762e11e06",
"id":362,
"labels":null,
"manifest_media_type":"application/vnd.docker.distribution.manifest.v2+json",
"media_type":"application/vnd.docker.container.image.v1+json",
"project_id":3,
"pull_time":"2021-10-22T10:28:55.305Z",
"push_time":"2021-10-22T10:28:49.341Z",
"references":null,
"repository_id":12,
"scan_overview":{
"application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0":{
"complete_percent":100,
"duration":8,
"end_time":"2021-10-22T10:28:57.356Z",
"report_id":"e83854eb-2304-4c58-85c9-a3e0fd9067a8",
"scan_status":"Success",
"severity":"Critical",
"start_time":"2021-10-22T10:28:49.827Z",
"summary":{
"summary":{
"Critical":7,
"High":47,
"Low":18,
"Medium":47
},
"total":119
}
}
}
}
My initial idea was to using jq to extract scan_status and severity with this command:
RESULTS=$RESULTS | jq '.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status'
after running that command I got this error:
jq: error: syntax error, unexpected ';', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status
jq: 1 compile error
Now I am trying text replacement, but that doesn't work either.
How should I proceed in this case?
Use square brackets and double quotes around the problematic key:
jq '.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status'
Also, this doesn't do what you think:
RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
First of all, tr doesn't replace strings, it replaces characters. sed can replace strings.
Moreover, the pipe | is used when the left hand side produces output. Variable assignment doesn't produce any output.
Finally, to assign the output of a command to a variable, you need to use Command Substitution:
var=$(command)
~~ ~
Wrap the key in question inside barckets and quotes like so:
.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status
Demo
From a shell syntax perspective, this looks wrong:
RESULTS=$RESULTS | jq '...'
Assigning a variable produces no output, so jq on the other side of the pipe has no input.
If it is okay for you to ignore the name of the problematic key, then you can substitute it with empty brackets:
RESULTS=$RESULTS | jq '.scan_overview [] .scan_status'
This is a bit sloppy and may match more than you would like, but in your narrow example, it will successfully pull out the value of .scan_status.
The reason this works, in this case, is because the application/vnd... property is the only property in the scan_overview object.
A more confident match would be achievable by using more quotes:
RESULTS=$RESULTS | jq '.scan_overview ."application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0" .scan_status'

How to properly chain multiple jq statements together when processing json in the shell such as with curl?

I am new to jq so if this is not a jq question or a json question please point me in the right direction. I am not sure of the correct terminology so it is making it hard for me to properly articulate the problem.
I am using to curl to pull some json that I want to filter out keys with specific values. Here is some of the sample json:
{
"id": "593f468c81aaa30001960e16",
"name": "Name 1",
"channels": [
"593f38398481bc00019632e5"
],
"geofenceProfileId": null
}
{
"id": "58e464585180ac000a748b57",
"name": "Name 2",
"channels": [
"58b480097f04f20007f3cdca",
"580ea26616de060006000001"
],
"geofenceProfileId": null
}
{
"id": "58b4d6db7f04f20007f3cdd2",
"name": "Name 3",
"channels": [
"58b8a25cf9f6e19cf671872f"
],
"geofenceProfileId": "57f53018271c810006000001"
}
When I run the following command:
curl -X GET -H 'authorization: Basic somestring=' "https://myserver/myjson" |
jq '.[] | {id: .id, name: .name, channels: .channels, geofenceProfileId: .geofenceProfileId}' |
jq '.[] | select(.channels == 58b8a25cf9f6e19cf671872f)'
I get the following error:
jq: error: syntax error, unexpected IDENT, expecting ';' or ')' (Unix shell quoting issues?) at , line 1:
.[] | select(.channels == 58b8a25cf9f6e19cf671872f)
jq: 1 compile error
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 351k 0 351k 0 0 1109k 0 --:--:-- --:--:-- --:--:-- 1110k
Is this error because jq pretty prints the output of the first statement and the second statement is expecting it to be in one code block? If so, how do I convert it back to non pretty print format or how can I use jq to run a new filter on the output?
Basically I am trying to parse hundreds of records and filter out all of the records that are in a specific channel number or have a specific geofenceProfileId.
I'd suggest you start with:
jq 'select(.channels | index("58b8a25cf9f6e19cf671872f"))'
In fact, this might even be exactly the filter you want. If you want to remove the "channels" once you've made the selection, you could augment the filter above as follows:
select(.channels | index("58b8a25cf9f6e19cf671872f")) | del(.channels)
The main thing to note is that one can create a pipeline WITHIN a single invocation of jq. So most likely you'll end up with: curl ... | jq ...
Btw
The jq expression {"id": .id} can be abbreviated to {id}, so instead of:
{id.id, name: .name, channels: .channels, geofenceProfileId: .geofenceProfileId}
you could write:
{id, name, channels, geofenceProfileId}
Probably not related to your case but I managed to transform my command
npm pkg get version -ws | jq "select(to_entries | min_by(.value) | .value)"
to
npm pkg get version -ws | jq "to_entries | min_by(.value) | .value"
and result is same. May be it helps. SO the idea is to pipe inside jq statement

How do I get field from HTTP GET JSON result to file?

I am trying to make a HTTP GET request to an API service and push one of the returned fields in the JSON result to a txt file.
Based on this previously asked question: (Getting JSON value from cURL in Linux Bash)
...I have a bash script as follows...
TOKEN_FILE="/myhome/project/resources/auto_token.txt"
AUTH_RESULT=$(curl -i -H "Content-Type: application/json" "https://access.mywebservice.com/access/oauth/token?grant_type=client_credentials&client_id=123456&client_secret=MySecretPassword");
RESULT_FIELDS=$( cat <<EOF | json_reformat | \
sed -rne '/:/s#^\s+"(\w+)":\s+"([^"]+)",?#json_\1="\2"#gp'
[$AUTH_RESULT]
EOF
)
if [ -f "$TOKEN_FILE" ]
then
echo "$RESULT_FIELDS" > "$TOKEN_FILE"
fi
The expected JSON result looks like this (copied from Postman):
{
"access_token": "eyJ5bGciOiJSUzI1NiJ6.eyJzY29wZSI6WyJDUl7iLCJNQVAiLCJQVFkiLCJ8R1QiLCJTVFMiLCJUVEwiXSwiaXNzIjoiaHR0cHM6Ly9hY2Nlc3MtdWF0LWFwaS5jb3JlbG9naWMuYXNpYSIsImVudl9hx2Nlc3NfcmVzdHJpY3QiOmZhbHNlLCJleHAiOjE0NjcyODMwODcsImNsaWVudF9pZCI6IjhhOTY4OGJjIn0.F2iQfVsi9zntOxKYrNRukSIwuQ_LGSi_WMIXKII2A3GOEaqs-WmFTi7az9rvvfDsOl9rHy_s_66A6PiCpPftyw21Fl0aZZRoFcKv2H_zDUHuxOEs8V36jHeLghV7pjHwYI_nG68CIGvfuRWFNzQuiMFWc_i8oB3n5noSd8fQqa4",
"token_type": "bearer",
"expires_in": 43199,
"scope": "PROD1 PROD2 PROD3",
"iss": "https://access.mywebservice.com",
"env_access_restrict": false
}
I get the following errors returned...
bash-4.1$ ./token_renewal_test_05.sh
: command not foundt_05.sh: line 2:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
115 576 0 576 0 0 2266 0 --:--:-- --:--:-- --:--:-- 30315
: command not foundt_05.sh: line 3:
: command not foundt_05.sh: line 4:
./token_renewal_test_05.sh: line 14: warning: here-document at line 10 delimited by end-of-file (wanted `EOF')
./token_renewal_test_05.sh: line 13: warning: here-document at line 9 delimited by end-of-file (wanted `EOF')
: command not foundt_05.sh: line 13:
lexical error: invalid char in json text.
sed -rne '/:/s#^\s+"(\w+)":\s+"
(right here) ------^
: command not foundt_05.sh: line 10:
./token_renewal_test_05.sh: line 16: syntax error: unexpected end of file
I'm a bit new to bash and despite what appears to be a direct pointer to the issue am having problems resolving this one (note this is version 5)!
Can anyone offer any assistance with this one?
PS: I do not have jq either.
Thanks!
Regards,
Chris
Caveat emptor as per this comment on Parsing JSON with UNIX tools.
A working solution for your format:
eval $(cat <<EOF | \
sed -re 's/(,|\{|\})//g' | \
sed -re 's/"(\w+)":\s*"?([^"]*)"?$/json_\1='\''\2'\''/'
$JSON
EOF
)
set | grep '^json_'
json_access_token=eyJ5bGciOiJSUzI1NiJ6.eyJzY29wZSI6WyJDUl7iLCJNQVAiLCJQVFkiLCJ8R1QiLCJTVFMiLCJUVEwiXSwiaXNzIjoiaHR0cHM6Ly9hY2Nlc3MtdWF0LWFwaS5jb3JlbG9naWMuYXNpYSIsImVudl9hx2Nlc3NfcmVzdHJpY3QiOmZhbHNlLCJleHAiOjE0NjcyODMwODcsImNsaWVudF9pZCI6IjhhOTY4OGJjIn0.F2iQfVsi9zntOxKYrNRukSIwuQ_LGSi_WMIXKII2A3GOEaqs-WmFTi7az9rvvfDsOl9rHy_s_66A6PiCpPftyw21Fl0aZZRoFcKv2H_zDUHuxOEs8V36jHeLghV7pjHwYI_nG68CIGvfuRWFNzQuiMFWc_i8oB3n5noSd8fQqa4
json_env_access_restrict=false
json_expires_in=43199
json_iss=https://access.mywebservice.com
json_scope='PROD1 PROD2 PROD3'
json_token_type=bearer
Thanks again Chepner and Drew
I was having too many issues with Sed (probably due to my lack of exprience). As it turns out, I tried using a lookbehind. Sed doesn't have this but grep does so knowing the strcuture of my JSON response will never chance, I was able to get my token extracted using the following with grep instead...
grep -o -P '(?<="access_token":").*(?=","token_type")'

SNMP OUTPUT OPTIONS - How do I get the OID response value only?

I have to go through and collect a few OIDs from some SNMP enabled network printers with a BASH script I have been working on.
My Request:
snmpget -v2c -c public 192.168.0.77
.1.3.6.1.2.1.1.1
.1.3.6.1.2.1.1.2
My Actual Response:
.1.3.6.1.2.1.1.1 = Counter32: 1974
.1.3.6.1.2.1.1.2 = Counter32: 633940
The Desired Response:
1974
633940314
(just the oid values only)
I looked and tested several options using the resource from the site below:
http://www.netsnmp.org/docs/man/snmpcmd.html#lbAF
-Oq removes '=' so running
snmpget -v2c -c public -Oq 10.15.105.133
.1.3.6.1.2.1.1.1
.1.3.6.1.2.1.1.2
returns
.1.3.6.1.2.1.1.1 Counter32: 1974
.1.3.6.1.2.1.1.2 Counter 32: 633940314
so I know I am phrasing my request properly.
I am taking the values and writing them to a MYSQL dB, I set the data types in my tale schema, the request is consistent so I know the definition of the OID, so I do not need all the information I am getting back, just the value of the OID itself, so I can write it to my dB without manipulating the the response. I probably can manipulate the response pulling the information to the right of ":" and writing the value of the OID.
I am relatively new to SNMP (http://www.net-snmp.org/), but I can not see why this is not a more commonly asked question because I have been searching everywhere for an answer and this post is my last recourse...
You can tune the output with the -O argument:
snmpgetnext -Oqv -v 2c -c public 192.168.0.77 .1
2
See the --help:
q: quick print for easier parsing
v: print values only (not OID = value)
You can postprocess the output with a simple Awk or sed script, or even just grep (provided you have grep -P).
snmpget -v2c -c public 192.168.0.77 <<'____HERE' | awk '{ print $4 }'
.1.3.6.1.2.1.1.1
.1.3.6.1.2.1.1.2
____HERE
or
.... | sed 's/.*: //'
or
.... | grep -oP ':\K[0-9]+'

mongoexport JSON assertion: 10340 Failure parsing JSON string

I'm trying to export CSV file list from mongoDB and save the output file to my directory, which is /home/asaj/. The output file should have the following columns: name, file_name, d_start and d_end.
The query should filter data with status equal to "FU" or "FD", and d_end > Dec. 10, 2012.
In mongoDB, the query is working properly. The query below is limited to 1 data output. See query below:
> db.Samples.find({ $or : [ { status : 'FU' }, { status : 'FD'} ], d_end : { $gte : ISODate("2012-12-10T00:00:00.000Z") } }, {_id: 0, name: 1, file_name: 1, d_start: 1, d_end: 1}).limit(1).toArray();
[
{
"name" : "sample"
"file_name" : "sample.jpg",
"d_end" : ISODate("2012-12-10T05:1:57.879Z"),
"d_start" : ISODate("2012-12-10T02:31:34.560Z"),
}
]
>
In CLI, mongoexport command looks like this:
mongoexport -d maindb -c Samples -f "name, file_name, d_start, d_end" -q "{'\$or' : [ { 'status' : 'FU' }, { 'status' : 'FD'} ] , 'd_end' : { '\$gte' : ISODate("2012-12-10T00:00:00.000Z") } }" --csv -o "/home/asaj/currentlist.csv"
But i always ended up with this error:
connected to: 127.0.0.1
Wed Dec 19 16:58:17 Assertion: 10340:Failure parsing JSON string near: , 'd_end
0x5858b2 0x528cb4 0x52902e 0xa9a631 0xa93e4d 0xa97de2 0x31b441ecdd 0x4fd289
mongoexport(_ZN5mongo11msgassertedEiPKc+0x112) [0x5858b2]
mongoexport(_ZN5mongo8fromjsonEPKcPi+0x444) [0x528cb4]
mongoexport(_ZN5mongo8fromjsonERKSs+0xe) [0x52902e]
mongoexport(_ZN6Export3runEv+0x7b1) [0xa9a631]
mongoexport(_ZN5mongo4Tool4mainEiPPc+0x169d) [0xa93e4d]
mongoexport(main+0x32) [0xa97de2]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x31b441ecdd]
mongoexport(__gxx_personality_v0+0x3c9) [0x4fd289]
assertion: 10340 Failure parsing JSON string near: , 'd_end
I'm having error in ", 'd_end' " in mongoexport CLI. I'm not so sure if it is a JSON syntax error because query works on MongoDB.
Please help.
After asking someone knows MongoDB better than me, we found out that the problem is the
ISODate("2012-12-10T00:00:00.000Z")
We found the answer on this question: mongoexport JSON parsing error
To resolve this error, first, we convert it to strtotime:
php > echo strtotime("12/10/2012");
1355126400
Next, multiple strtotime result by 1000. This date will looks like this:
1355126400000
Lastly, change ISODate("2012-12-10T00:00:00.000Z") to new Date(1355126400000) in the mongoexport command.
Now, the CLI mongoexport looks like this and it works:
mongoexport -d maindb -c Samples -f "id,file_name,d_start,d_end" -q "{'\$or' : [ { 'status' : 'FU' }, { 'status' : 'FD'} ] , 'd_end' : { '\$gte' : new Date(1355126400000) } }" --csv -o "/home/asaj/listupdate.csv"
Note: remove space between each field names in -f or --fields option.
I know it has little to do with this question, but the title of this post brought it up in Google so since I was getting the exact same error I'll add an answer. Hopefully it helps someone.
My issue was adding a MongoId query for _id to a mongoexport console command on Windows. Here's the error:
Assertion: 10340:Failure parsing JSON string near: _id
The problem ended up being that I needed to wrap the JSON query in double quotes, and the ObjectId had to be in double quotes (not single!), so I had to escape those quotes. Here's the final query that worked, for future reference:
mongoexport -u USERNAME -pPASSWORD -d DATABASE -c COLLECTION
--query "{_id : ObjectId(\"5148894d98981be01e000011\")}"